Large model reasoning system and reasoning request processing method and device

Abstract

The invention discloses a large model reasoning system and a processing method and device of a reasoning request, and relates to the technical field of artificial intelligence, in the large model reasoning system provided by the invention, clients and reasoning nodes can cooperatively realize a large model reasoning function through interaction, the reasoning request is sent to the reasoning node, and the reasoning node sends the intermediate data of one or more first reasoning units in the reasoning units corresponding to the reasoning request to the first client, so that the first client does not need to consume local computing resources to generate the intermediate data of the first reasoning units; the intermediate data of the reasoning unit is reused in the large model reasoning process of the client, so that the large model reasoning efficiency is effectively improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a large-scale model reasoning system, a method and apparatus for processing reasoning requests. Background Technology

[0002] Currently, there is an increasing demand for large model inference on the client side. During the execution of large model inference requests on the client side, due to the limited storage capacity of the client, after generating and using the key matrix and value matrix (KVcache) of the inference unit, the client will discard the KVcache of the inference unit in time to free up storage space. As a result, the client needs to consume a lot of computing resources to regenerate the KVcache every time it executes an inference request, which seriously affects the inference efficiency.

[0003] Therefore, how to improve the inference efficiency of large model inference on the client side is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a large model inference system, a method and apparatus for processing inference requests, which enables the reuse of intermediate data of inference units during the large model inference process on the client side, thereby effectively improving the efficiency of large model inference.

[0005] Firstly, a large-model inference system is provided, comprising a first client and an inference node; wherein, the inference node is configured to: receive an inference request sent by the first client; determine the first inference unit in the inference unit corresponding to the inference request, wherein the inference unit corresponding to the inference request refers to the basic data unit when performing inference using the large model; and send the intermediate data of the first inference unit to the first client; the first client is configured to execute the inference request based on the intermediate data of the first inference unit and the large model.

[0006] In the large model inference system provided in this application, inference nodes can be implemented in various ways, such as cloud nodes on a cloud platform, terminal devices, edge nodes, or public network nodes, etc., and this application is not limited to these. The client and the inference node can interact to collaboratively implement the large model inference function. Specifically, when the first client executes an inference request, it sends the inference request to the inference node. The inference node then sends intermediate data from one or more first inference units corresponding to the inference request to the first client. This eliminates the need for the first client to expend local computing resources to generate the intermediate data for these inference units, enabling the reuse of intermediate data from inference units during the client's large model inference process, thereby effectively improving the efficiency of large model inference.

[0007] Furthermore, the inference unit corresponding to the inference request refers to the basic data unit used when performing inference using a large model. That is, the large model uses the inference unit as the basic data processing unit to perform inference. For example, in a text inference scenario, the inference unit is the word segment obtained by segmenting the text; in an image inference scenario, the inference unit is the image block obtained by segmenting the image, with each image block being a inference unit; in an audio inference scenario, the inference unit is the audio segment obtained by segmenting the audio, with each audio segment being a inference unit; in a video inference scenario, the inference unit is the image block obtained by segmenting each video frame, with each image block being a inference unit, and so on.

[0008] In some embodiments, the inference node is configured to: retrieve data from a storage system based on the inference unit corresponding to the inference request, the storage system storing intermediate data of the inference unit generated using a large model; if a second inference unit and its intermediate data are found in the storage system, determine a first inference unit based on the second inference unit and the inference unit corresponding to the inference request, wherein the second inference unit matches any inference unit corresponding to the inference request, and the first inference unit includes the second inference unit; if no inference unit matching the inference unit corresponding to the inference request is found in the storage system, use the inference unit corresponding to the inference request as the first inference unit.

[0009] In this way, the inference node searches the storage system based on the inference unit corresponding to the inference request, and then determines the first inference unit based on the search results. If the inference node finds an inference unit in the storage system that matches the inference unit corresponding to the inference request, it can use the found inference unit as the first inference unit. If the inference node does not find an inference unit in the storage system that matches the inference unit corresponding to the inference request, it can use all the inference units corresponding to the inference request as the first inference unit. This eliminates the need to consume the local computing resources of the first client to generate intermediate data for these inference units, reduces the computing pressure on the first client, and improves the inference efficiency of large models on the first client side.

[0010] In some embodiments, the inference node is configured to: determine reference information based on the second inference unit and the inference unit corresponding to the inference request, wherein the reference information indicates the number of inference units, in addition to the second inference unit, that need to generate intermediate data to execute the inference request; if the reference information meets a condition, designate the second inference unit as the first inference unit; or, if the reference information does not meet a condition, determine a third inference unit based on the inference unit corresponding to the inference request and the second inference unit, wherein the first inference unit includes the second inference unit and the third inference unit, and the third inference unit refers to the inference unit other than the second inference unit among the inference units corresponding to the inference request. In some embodiments, the condition is that the number of inference units indicated by the reference information is less than a threshold.

[0011] In this way, the first inference unit is determined based on the computing power of the first client. For the first client to execute the inference request, if the first client still needs to generate a large number of inference units with intermediate data after the inference node sends the intermediate data of the retrieved inference units to the first client, the inference node can determine one or more additional first inference units based on the retrieved inference units, thereby reducing the computing pressure on the first client and improving the efficiency of large model inference on the client side.

[0012] In some embodiments, the inference unit corresponding to the inference request further includes a fourth inference unit, which is different from the first inference unit; the first client is further configured to: generate intermediate data of the fourth inference unit using a large model; and execute the inference request based on the intermediate data of the first inference unit and the intermediate data of the fourth inference unit.

[0013] In some embodiments, the inference node is a cloud node, and the large model inference system further includes an edge node. The inference node is used for any of the following: sending intermediate data of the first inference unit to the first client through the edge node; or controlling the first client to obtain intermediate data of the first inference unit from the edge node.

[0014] In large-scale model inference systems that also include edge nodes, these edge nodes, acting as an intermediate layer between the cloud and the edge, can receive instructions and data from the cloud and provide local computing and storage services to the edge devices. Therefore, edge nodes can store intermediate data for inference units, such as the intermediate data of inference units that have reached a certain access threshold. Edge nodes can also generate intermediate data for inference units using the large model. Transmitting the intermediate data of the first inference unit through edge nodes can shorten the data transmission path and reduce communication latency.

[0015] In some embodiments, the inference node is an edge node, and the inference node is further used to: generate intermediate data for the first inference unit using the large model; or, if the large model inference system also includes cloud nodes, obtain the intermediate data for the first inference unit from the cloud nodes. In this way, the client can leverage the storage, computing, and other capabilities of the edge nodes to improve the efficiency of large model inference.

[0016] In some embodiments, the inference node is a terminal device, and the inference node is further configured to: generate intermediate data for the first inference unit using the large model; or, obtain the intermediate data for the first inference unit through a network node. In this way, the client can leverage the storage, computing, and other capabilities of other terminal devices to improve the inference efficiency of the large model.

[0017] In some embodiments, the inference node is further configured to: retrieve a storage system based on the received fifth inference unit, the storage system being used to store intermediate data of the inference unit generated using the large model; if the intermediate data of the fifth inference unit is not found in the storage system, send a data transmission instruction to a second client in the large model inference system, the data transmission instruction instructing the intermediate data of the fifth inference unit to be sent to the inference node, the second client storing the intermediate data of the fifth inference unit.

[0018] In the large model inference system provided in this application, the client can send the intermediate data of the locally generated inference unit to the inference node, so that the inference node can update the intermediate data stored in the storage system in a timely manner, providing technical support for realizing cross-client intermediate data reuse. It should be understood that the inference unit has a smaller data volume compared to its intermediate data, and the inference node's storage system may already contain the intermediate data of that inference unit. Based on this, allowing the inference node to decide whether the client needs to send the intermediate data of the inference unit can avoid redundant data transmission and processing, saving communication and computing resources.

[0019] In some embodiments, if the inference node receives a fifth inference unit sent by multiple third clients in a large model inference system, the inference node is further configured to: determine a second client from the multiple third clients based on one or more of the network transmission status, data transmission latency, and the location of the third clients.

[0020] This process means that if the intermediate data of the same inference unit is stored on multiple clients, the inference node can select one or more clients to send data transmission instructions based on one or more of the network transmission status, data transmission latency, and the location of these clients, thereby saving communication resources.

[0021] In some embodiments, the inference node is further configured to: receive intermediate data of the fifth inference unit sent by the second client; and store the intermediate data of the fifth inference unit in a storage system.

[0022] In some embodiments, the inference node is a cloud node, and the large model inference system also includes edge nodes; the second client is used to send the intermediate data of the fifth inference unit to the edge node; the edge node is used to merge the received intermediate data of the inference unit and send it to the inference node.

[0023] In the case of a large-scale model inference system including edge nodes, the edge nodes, acting as an intermediary layer between the cloud and the edge, can receive instructions and data from the cloud and provide local computing and storage services to the edge devices. Therefore, the second client can send intermediate data from the fifth inference unit to the edge node, which then merges the received intermediate data and sends it to the inference node. In other words, the edge node can receive intermediate data from multiple clients' inference units, merge the received intermediate data, and send it to the inference node, avoiding the inference node receiving a large amount of intermediate data from multiple clients in a dispersed manner.

[0024] Secondly, a method for processing inference requests is provided, executed by an inference node in a large model inference system. The large model inference system also includes a first client. The method includes: receiving an inference request sent by the first client; determining a first inference unit in the inference unit corresponding to the inference request, wherein the inference unit corresponding to the inference request refers to the basic data unit when performing inference using the large model; and sending intermediate data of the first inference unit to the first client, wherein the first client is used to execute the inference request based on the intermediate data of the first inference unit and the large model.

[0025] In some embodiments, determining a first inference unit among the inference units corresponding to an inference request includes: retrieving data from a storage system based on the inference unit corresponding to the inference request, the storage system storing intermediate data of inference units generated using a large model; if a second inference unit and its intermediate data are found in the storage system, determining a first inference unit based on the second inference unit and the inference unit corresponding to the inference request, wherein the second inference unit matches any one of the inference units corresponding to the inference request, and the first inference unit includes the second inference unit; if no inference unit matching the inference unit corresponding to the inference request is found in the storage system, the inference unit corresponding to the inference request is taken as the first inference unit.

[0026] In some embodiments, determining a first inference unit based on a second inference unit and the inference unit corresponding to the inference request includes: determining reference information based on the second inference unit and the inference unit corresponding to the inference request, wherein the reference information indicates the number of inference units other than the second inference unit that need to generate intermediate data in order to execute the inference request; if the reference information meets the conditions, the second inference unit is used as the first inference unit; or, if the reference information does not meet the conditions, determining a third inference unit based on the inference unit corresponding to the inference request and the second inference unit, wherein the first inference unit includes the second inference unit and the third inference unit, and the third inference unit refers to the inference unit other than the second inference unit among the inference units corresponding to the inference request.

[0027] In some embodiments, the inference node is a cloud node, and the large model inference system further includes an edge node, which sends intermediate data of the first inference unit to the first client, including any one of the following: sending intermediate data of the first inference unit to the first client through the edge node; or controlling the first client to obtain intermediate data of the first inference unit from the edge node.

[0028] In some embodiments, the inference node is an edge node, and the method further includes: generating intermediate data for the first inference unit using a large model; or, if the large model inference system also includes a cloud node, obtaining the intermediate data for the first inference unit from the cloud node.

[0029] In some embodiments, the inference node is a terminal device, and the method further includes: generating intermediate data of the first inference unit using a large model; or, obtaining the intermediate data of the first inference unit through a network node.

[0030] In some embodiments, the method further includes: retrieving a storage system based on the received fifth inference unit, the storage system being used to store intermediate data of inference units generated using a large model; if the intermediate data of the fifth inference unit is not found in the storage system, sending a data transmission instruction to a second client in the large model inference system, the data transmission instruction instructing the intermediate data of the fifth inference unit to be sent to the inference node, the second client storing the intermediate data of the fifth inference unit.

[0031] In some embodiments, if the inference node receives a fifth inference unit sent by multiple third clients in a large model inference system, the method further includes: determining a second client from the multiple third clients based on one or more of the following: network transmission status, data transmission latency, and the location of the third clients.

[0032] In some embodiments, the method further includes: receiving intermediate data of the fifth inference unit sent by the second client; and storing the intermediate data of the fifth inference unit in a storage system.

[0033] Thirdly, a method for processing inference requests is provided, executed by a client of a large model inference system, which also includes inference nodes. The method includes: sending an inference request to the inference node; receiving intermediate data of the first inference unit in the inference unit corresponding to the inference request, wherein the inference unit corresponding to the inference request refers to the basic data unit when performing inference using the large model; and executing the inference request using the large model based on the intermediate data of the first inference unit.

[0034] In some embodiments, the inference unit corresponding to the inference request further includes a fourth inference unit, which is different from the first inference unit; the method further includes: generating intermediate data of the fourth inference unit using a large model; and executing the inference request using the large model based on the intermediate data of the first inference unit, including: executing the inference request using the large model based on the intermediate data of the first inference unit and the intermediate data of the fourth inference unit.

[0035] In some embodiments, the method further includes one or more of the following: sending the inference unit corresponding to the inference request to the inference node; sending the inference unit corresponding to the inference result of the inference request to the inference node.

[0036] In some embodiments, the method further includes: receiving a data transmission instruction sent by an inference node; and sending intermediate data of the inference unit indicated by the data transmission instruction to the inference node.

[0037] Fourthly, an apparatus for processing inference requests is provided, the apparatus including at least one functional module for performing the inference request processing method provided as described in the second aspect above or any alternative method of the second aspect above.

[0038] Fifthly, another inference request processing apparatus is provided, the apparatus comprising at least one functional module for performing the inference request processing method provided as described in the third aspect above or any alternative method of the third aspect above.

[0039] In a sixth aspect, a computing device is provided, the computing device including a processor and a memory, the processor being configured to execute instructions stored in the memory such that the computing device performs the functions of an inference node in a large model inference system as provided in the first aspect above or any alternative embodiment of the first aspect above.

[0040] In a seventh aspect, a computing device cluster is provided, comprising a plurality of computing devices, each computing device including a processor and a memory; the processor of the plurality of computing devices is configured to execute instructions stored in the memory of the plurality of computing devices, such that the computing device cluster performs the functions of a large model inference system as provided in the first aspect above or any alternative embodiment of the first aspect above.

[0041] Eighthly, a computer program product containing instructions is provided, which, when executed by a cluster of computing devices, causes the cluster of computing devices to perform the functions of a large model inference system as provided in the first aspect above or any alternative method of the first aspect above.

[0042] A ninth aspect provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, enable the cluster of computing devices to perform the functions of a large model inference system as provided in the first aspect or any alternative embodiment of the first aspect.

[0043] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0044] Figure 1 This is a schematic diagram of the reasoning process in a large language model;

[0045] Figure 2 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0046] Figure 3 This is a schematic diagram of the architecture of a large model inference system provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram of the architecture of another large model inference system provided in the embodiments of this application;

[0048] Figure 5 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of this application;

[0049] Figure 6 This is a flowchart of a method for processing inference requests provided in an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of an inference unit corresponding to an image provided in an embodiment of this application;

[0051] Figure 8 This is a schematic diagram illustrating how a large-scale model inference system processes inference requests, as provided in an embodiment of this application.

[0052] Figure 9 This is a schematic diagram illustrating another large-model inference system for processing inference requests provided in an embodiment of this application;

[0053] Figure 10 This is a schematic diagram illustrating another large-model inference system processing inference requests provided in an embodiment of this application;

[0054] Figure 11This is a schematic diagram of the structure of a reasoning request processing device provided in an embodiment of this application;

[0055] Figure 12 This is a schematic diagram of another inference request processing device provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings. It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the inference units and large models involved in this application were obtained under fully authorized conditions.

[0057] To facilitate understanding, the key terms and concepts involved in this application will be explained below.

[0058] Large models refer to artificial intelligence (AI) models with a large number of parameters and complex computational structures. These models are typically built from deep neural networks and have billions or even hundreds of billions of parameters. Large models have wide applications in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems, but are not limited to these.

[0059] Large language models (LLMs) are AI language processing models trained on large-scale text data. Examples include academic papers and social media posts.

[0060] Generative large models are used to perform inference based on a given inference request and generate inference results. The inference request may include text, images, audio, or video, and this application is not limited to these. Generative large models typically include LLMs, multimodal models, etc. Multimodal models are used to generate inference results in another modality based on input information in one modality provided by the user. Here, modality refers to the form of data, and multimodal data refers to data in multiple forms. For example, multimodal data may include text, images, video, audio, etc. In some embodiments, the generative large model is a model based on the transformer architecture; however, it can also be a model based on other architectures, such as a model based on the Mamba architecture.

[0061] In this application, intermediate data refers to the intermediate computation results generated by a large model (such as a generative large model) during the inference process. The intermediate data may differ for large models with different architectures. For example, for a large model based on the transformer architecture, which typically employs a self-attention mechanism, the computation process usually involves a key matrix and a value matrix, abbreviated as KVcache. Accordingly, the intermediate data includes the KVcache. Of course, intermediate data can also include other information, such as hidden state. Similarly, if a generative model based on the Mamba architecture also uses a self-attention mechanism, then the intermediate data can also include KVcache, hidden state, etc.

[0062] In this application, an inference unit refers to the basic data unit used in large-scale model reasoning. That is, the large-scale model uses inference units as basic data processing units to perform reasoning. For example, in a text reasoning scenario, the inference unit is a word segment obtained by word segmentation of the text. A word segment can be a character, word, phrase, etc. A sequence of word segments can be used as input to the large-scale model. For instance, assuming the reasoning request includes the text "What color is an apple?", the large-scale model segments the text to obtain multiple word segments: "apple", "is", "what", and "color". Each word segment should be understood as a reasoning unit. Similarly, in an image reasoning scenario, the inference unit is an image block obtained by segmenting an image; each image block is a reasoning unit. In an audio reasoning scenario, the inference unit is an audio segment obtained by segmenting audio; each audio segment is a reasoning unit. In a video reasoning scenario, the inference unit is an image block obtained by segmenting each video frame; each image block is a reasoning unit. The above method of determining the inference unit is an example; in other embodiments, other methods can also be used to determine the inference unit. Furthermore, this application does not limit the method of text segmentation, image segmentation, or video frame segmentation in its embodiments. In some scenarios, the inference unit is also referred to as a token, and this application does not limit this.

[0063] A data center (DC) is a complex set of facilities used for the centralized processing, storage, transmission, exchange, and management of information. This complex set of facilities typically includes servers, storage devices, switches, routers, firewalls, and other equipment. For example, if this complex set of facilities is configured in each of several different regions, each of these regions can be called a data center. These regions could be different cities, different districts within the same city, or different buildings. Similarly, if this complex set of facilities is configured in each of several different server rooms, each of these server rooms can be called a data center. Even a single rack within a server room can be considered a data center, and different racks within the same server room are called different data centers.

[0064] Acceleration chips, also known as acceleration cards, acceleration devices, or accelerators, are a type of specialized hardware device or computer system designed to accelerate computation in AI scenarios. In the embodiments of this application, acceleration chips may be, for example, graphics processing units (GPUs), neural network processing units (NPUs), intelligent processing units (IPUs), tensor processing units (TPUs), domain-specific architecture (DSA) chips, and so on, but are not limited to these.

[0065] The prefill and decode phases are two inference phases that divide the inference process of a generative large model. The prefill phase is also called the full inference phase, and the decode phase is also called the incremental inference phase. The prefill phase is used to perform calculations on the inference units corresponding to the inference requests to obtain intermediate data for each inference unit. The decode phase is used to generate the inference result based on the intermediate data of the inference units. See below for reference. Figure 1 Taking a large language model as an example, this paper introduces the prefill and decoding stages.

[0066] Figure 1 This is a schematic diagram of the reasoning process in a large language model. For example... Figure 1 As shown, taking dialogue reasoning through a large language model as an example, Figure 1In this context, T0-T2 are inference units obtained by dividing the user-proposed question, also known as input tokens. During the pre-filling phase, intermediate data corresponding to each of the multiple input tokens T0-T2 is determined, with the intermediate data corresponding to input token T0 being t0, input token T1 being t1, and input token T2 being t2. During the decoding phase, the first decoding block determines the output token T3 based on the intermediate data t0-t2 corresponding to the multiple input tokens T0-T2, with the intermediate data corresponding to output token T3 being t3. At this point, the determined intermediate data includes t0-t3. The second decoding block determines the output token T4 based on the intermediate data t0-t2 corresponding to the multiple input tokens T0-T2 and the intermediate data t3 corresponding to output token T3, with the intermediate data corresponding to output token T4 being t4. At this point, the determined intermediate data includes t0-t4. The third decoding block, based on the intermediate data t0-t2 corresponding to the multiple input tokens T0-T2 and the intermediate data t3-t4 corresponding to the output tokens T3-T4, determines the output token T5, and the intermediate data corresponding to the output token T5 is t5. The determined intermediate data at this point includes t0-t5. The fourth decoding block, based on the intermediate data t0-t2 corresponding to the multiple input tokens T0-T2 and the intermediate data t3-t5 corresponding to the output tokens T3-T5, determines the output token T6, and the intermediate data corresponding to the output token T6 is t6. The determined intermediate data at this point includes t0-t6. The result composed of multiple output tokens T3-T6 is determined as the question-and-answer result.

[0067] For example, if a user asks the question "What color is an apple?", the question can be divided into multiple input tokens: "apple", "is", "what", and "color". In the pre-filling stage, the intermediate data corresponding to each of these input tokens is determined. In the decoding stage, the first decoding block determines the output token "apple" based on the intermediate data corresponding to these input tokens; the second decoding block determines the output token "of" based on the intermediate data corresponding to these input tokens and the intermediate data corresponding to the output token "apple"; the third decoding block determines the output token "color" based on the intermediate data corresponding to these input tokens and the intermediate data corresponding to the output tokens "apple", "of", and "color"; the fourth decoding block determines the output token "is" based on the intermediate data corresponding to these input tokens and the intermediate data corresponding to the output tokens "apple", "of", "color", and "is"; and the fifth decoding block determines the output token "red" based on the intermediate data corresponding to these input tokens and the intermediate data corresponding to the output tokens "apple", "of", "color", and "is". Combining these output tokens yields the question-answering result "The color of an apple is red".

[0068] In the above Figure 1 In the inference process shown, the intermediate data corresponding to each input token determined in the pre-filling phase is used by each decoding block in the decoding phase to generate an output token. Simultaneously, the intermediate data corresponding to each output token is also used by subsequent decoding blocks to generate subsequent output tokens. Therefore, a series of intermediate data for inference units are generated during the inference process, and this intermediate data is used multiple times during inference.

[0069] The application scenarios and implementation environment of this application are described below.

[0070] This application provides a large model inference system applicable to scenarios involving large models for inference. As described above, large model inference generates a series of intermediate data for inference units, which are used multiple times during the inference process. If the large model is deployed on servers or cloud platforms with strong storage and computing capabilities, the intermediate data of inference units can be cached during the inference process to avoid redundant computation, or in other words, inference efficiency can be improved by reusing the intermediate data of inference units. However, with the increasing demand for large model inference on the client side, due to the limited storage capacity of the client, the client discards the intermediate data of the inference units promptly after generating and using it to free up storage space when executing inference requests. This results in the client needing to consume a large amount of computing resources to regenerate the intermediate data each time it executes an inference request, severely impacting inference efficiency. Therefore, this application provides a large model inference system that can also reuse the intermediate data of inference units to improve inference efficiency during the client-side implementation of large model inference.

[0071] The following is for reference. Figures 2 to 4 This paper will further introduce the implementation environment and application scenarios of this application. Figure 2 This is a schematic diagram of an implementation environment provided in an embodiment of this application. For example... Figure 2 As shown, the implementation environment of this application includes a large model inference system 100, which includes a client 101 and an inference node 102. The client 101 and the inference node 102 are connected via a wired network or a wireless network.

[0072] Client 101 is used to implement large model inference functionality. Indicatively, the user's terminal device runs client 101, and the terminal device is a device with large model inference capabilities. In some embodiments, the terminal device is equipped with an acceleration chip to accelerate the large model inference process of client 101. The acceleration chip may be, for example, a GPU, NPU, IPU, TPU, DSA chip, etc. In other embodiments, a processor such as a central processing unit (CPU) deployed on the terminal device is used to provide large model inference capabilities to client 101. This application does not limit the manner in which the terminal device implements large model inference capabilities. For example, the user triggers client 101 to execute an inference request using a large model by operating client 101. Taking an LLM (Limited Linear Model) as an example, client 101 generates an inference request containing the question based on the user's input question, executes the inference request using the LLM, and outputs the inference result, i.e., obtains the answer to the question. This application does not limit the type of terminal device; for example, the terminal device may be a smartphone, game console, desktop computer, augmented reality terminal, tablet computer, e-book reader, laptop computer, or IoT device, etc., and is not limited thereto. It should be noted that the number of clients 101 shown in the figure is only for illustrative purposes and does not constitute a limitation on this application. The number of clients 101 in the large model inference system 100 can be dozens or hundreds or more, or it can be fewer.

[0073] Inference node 102 is a node used to assist client 101 in implementing large model inference functionality. Schematic, inference node 102 possesses one or more of the following: storage functionality and large model inference functionality. Storage functionality refers to inference node 102's ability to store intermediate data of inference units generated using the large model. Large model inference functionality refers to inference node 102's ability to execute inference requests using the large model, generate intermediate data of the inference units corresponding to the inference requests, and output inference results. It should be understood that client 101 has the ability to independently implement large model inference. However, in some scenarios, such as when the input sequence of the large model corresponding to the inference request is a long sequence, client 101 can send an inference request to inference node 102 to utilize the functionality of inference node 102. For example, upon receiving an inference request, inference node 102 sends the intermediate data of the inference units corresponding to the inference request to client 101, thereby eliminating the need for client 101 to expend local computing resources to generate the intermediate data of these inference units and improving inference efficiency. The intermediate data of the inference unit sent by inference node 102 to client 101 can be intermediate data already stored on inference node 102 (e.g., inference node 102 receives intermediate data from inference unit of client 101 and stores the received intermediate data), or it can be intermediate data generated by inference node 102 using the large model. This application does not limit this. This process can also be understood as client 101 and inference node 102 interacting to collaboratively realize the large model inference function. Inference node 102 assists client 101 in completing the pre-filling stage in the large model inference process, and client 101 performs the subsequent decoding stage. It should be noted that the number of inference nodes 102 shown in the figure is only for illustrative purposes and does not constitute a limitation of this application. The number of inference nodes 102 in the large model inference system 100 can be dozens, hundreds, or more, or it can be fewer.

[0074] In this application embodiment, the inference node 102 can be implemented in various ways. Schematic, the inference node 102 can be a cloud node on a cloud platform, a terminal device, an edge node, or a public network node, etc., but this application is not limited to these. See below for reference. Figure 3 and Figure 4 This paper introduces the application scenarios of the large model inference system 100 under different implementation methods of inference node 102.

[0075] Figure 3 This is a schematic diagram of the architecture of a large model inference system provided in an embodiment of this application. For example... Figure 3As shown, the large-scale model inference system 100 includes a client 101 and an inference node 102. The inference node 102 is a cloud node on a cloud platform. This application does not limit the type of cloud platform; the cloud platform can be a public cloud, a private cloud (e.g., a data center deployed within an enterprise), a hybrid cloud (e.g., a combination of public and private clouds), or an industry cloud (e.g., jointly built and operated by enterprises within a specific industry), etc. The cloud node, for example, is a scheduling node of the cloud platform. Figure 3 In the architecture shown, client 101 can be referred to as the edge, and inference node 102 can be referred to as the cloud. The interaction between client 101 and inference node 102 to collaboratively achieve large-model inference can also be understood as a cloud-assisted edge mechanism. That is, client 101 can leverage the storage and computing capabilities of the cloud platform to improve the efficiency of large-model inference. Here, cloud platform is short for cloud computing platform, referring to services based on hardware and software resources that provide computing, networking, and storage capabilities. Through the network "cloud," massive amounts of data are processed and analyzed remotely before being returned to the user, featuring large scale, distributed architecture, virtualization, high availability, scalability, on-demand service, and security. Cloud platforms can achieve rapid deployment and release of configurable computing resources with relatively low management costs or low interaction complexity between users and service providers. Indicatively, a cloud node is a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. In some embodiments, a cloud node may also be a virtual machine instance, a container instance, etc., but this application is not limited thereto.

[0076] exist Figure 3In the illustrated architecture, in some scenarios, the large model inference system 100 also includes an edge node 103. The edge node 103 is located at the edge of the network environment, close to the client 101. The edge node 103 can be, for example, an independent physical server or a server cluster consisting of multiple physical servers. Schematic, the edge node 103 connects the cloud side and the edge side; that is, the edge node 103 acts as an intermediate layer between the cloud side and the edge side, receiving instructions and data from the cloud side and providing local computing and storage services to the edge devices. Schematic, based on the foregoing description, the client 101 and the inference node 102 can collaborate to implement the large model inference function through interaction. When the large model inference system 100 also includes the edge node 103, the interaction between the client 101 and the inference node 102 can pass through the edge node 103. For example, the inference node 102 sends intermediate data from the inference unit to the edge node 103, and the edge node 103 then sends the intermediate data from the inference unit to the client 101. For example, client 101 sends intermediate data of the inference unit to edge node 103, and edge node 103 sends intermediate data of the inference unit to inference node 102.

[0077] Figure 4 This is a schematic diagram of the architecture of another large-model inference system provided in an embodiment of this application. For example... Figure 4 As shown, the large-scale model inference system 100 includes a client 101 and an inference node 102, wherein the inference node 102 is a terminal device. This application does not limit the type of terminal device corresponding to the inference node 102. For example, the large-scale model inference system 100 is deployed in a home network. A home network refers to a network system that connects multiple electronic devices within a home to achieve resource sharing, device interconnection, and information exchange. The client 101 can run on terminal devices such as mobile phones, laptops, and smart home devices used by the user, and the inference node 102 can be a desktop computer or remote office computer in the home network. Figure 4 In the architecture shown, both client 101 and inference node 102 are referred to as end-side devices. The interaction between client 101 and inference node 102 to collaboratively implement large model inference can be understood as an end-to-end collaborative mechanism. That is, client 101 can leverage the storage and computing capabilities of other terminal devices in the home network to improve the efficiency of large model inference. Furthermore, in Figure 4 In the architecture shown, client 101 and inference node 102 can establish a communication connection through near field communication (NFC), and this application does not limit this.

[0078] The above Figure 3 and Figure 4The illustrations shown are several schematic descriptions of the large model inference system provided in this application and do not constitute a limitation on this application. In practical applications, the large model inference system provided in this application can be flexibly deployed according to the needs of the client. For example, the implementation method and number of inference nodes can be flexibly selected according to the geographical location of the terminal device running the client, etc. This application does not limit this.

[0079] In some embodiments, the aforementioned wireless or wired networks use standard communication technologies and / or protocols. Networks include, but are not limited to, data center networks, storage area networks (SANs), local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), mobile, wired or wireless networks, private networks, or any combination of virtual private networks. In some embodiments, the network is a Transmission Control Protocol / Internet Protocol (TCP / IP) network and an RDMA network, such as RoCE networks, InfiniBand (IB) networks, etc., within a data center network; this is not limited. In some implementations, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or part of the link. In other embodiments, custom and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0080] Additionally, this application provides a computing device capable of being configured as a terminal device or inference node (such as a cloud node, edge node, or public network node) involved in the aforementioned large-model inference system 100. (See reference) Figure 5 , Figure 5 This is a schematic diagram of the hardware structure of a computing device provided in an embodiment of this application. Figure 5As shown, the computing device 500 includes a memory 501, a processor 502, a communication interface 503, and a bus 504. The memory 501, processor 502, and communication interface 503 are interconnected via the bus 504.

[0081] Memory 501 refers to a device for storing data, which may be a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having an instruction or data structure form and accessible by a computer, but not limited thereto. Illustratively, memory 501 is used to store at least a piece of program code. When the program code stored in memory 501 is executed by processor 502, processor 502 performs the steps executed by the client or inference node in the following method embodiments.

[0082] The processor 502 can be a network processor (NP), a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or an integrated circuit used to control the execution of the program in this application. The processor 502 can be a single-core processor or a multi-core processor. There can be one or more processors 502. The memory 501 and the processor 502 can be separate or integrated.

[0083] The communication interface 503 uses a transceiver module, such as a transceiver, to enable communication between the computing device 500 and other devices or communication networks. For example, data can be acquired through the communication interface 503.

[0084] Bus 504 may include a pathway for transmitting information between various components of computing device 500 (e.g., memory 501, processor 502, communication interface 503).

[0085] In some embodiments, the computing device 500 may further include an acceleration chip, such as a GPU, NPU, IPU, TPU, DSA chip, etc., which are not limited in this application.

[0086] In some embodiments, the computing device 500 may further include an output device and an input device. The output device communicates with the processor 502 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 502 and can receive user input in various ways. For example, the input device may be a mouse, a keyboard, a touchscreen device, or a sensing device, etc.

[0087] This application also provides a computing device cluster, which includes multiple computing devices that are interconnected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0088] The following section describes the process of implementing large-model inference in a large-model inference system.

[0089] Figure 6 This is a flowchart illustrating a method for processing inference requests provided in an embodiment of this application. For example... Figure 6 As shown, this method is applied to the above Figures 2 to 4 In the large model inference system shown, taking the interaction between the first client and the inference node in the large model inference system as an example, the method includes the following steps 601 to 606.

[0090] 601. The first client sends an inference request to the inference node.

[0091] In this embodiment, the first client is any client in the large model inference system. Indicatively, the user's terminal device runs the first client. The user operates the first client, inputting information related to large model inference, triggering the first client to generate an inference request. The inference request includes the information input by the user, hereinafter referred to as input information. The user-input information can be text, voice, images, video, etc., and this application does not limit this. In some embodiments, the inference request also includes other information, such as the version of the large model, user information, request type (e.g., image recognition or text processing), etc., and is not limited thereto.

[0092] In some embodiments, the first client sends an inference request to the inference node when the input information of the inference request meets certain conditions. The conditions for meeting the input information of the inference request can be set according to requirements. For example, if the input information is text, the condition is that the text has more than 500 characters; if the input information is speech, the condition is that the speech duration is more than 60 seconds; if the input information is an image, the condition is that the image size is more than 1MB, and so on. These are merely illustrative examples and do not constitute a limitation of this application. In other words, the first client can determine whether it needs an inference node to assist in large model inference based on whether the input information of the inference request meets certain conditions, or whether it needs an inference node to assist in large model inference based on whether the input information is a long sequence. In this way, inference requests can be sent to the inference node in a targeted manner. These inference requests typically involve a large amount of input information processing. Sending these inference requests to the inference node to assist in large model inference can effectively save the local computing resources of the first client and improve the efficiency of large model inference. Moreover, for inference requests where the input information does not meet the conditions, the first client can process them locally. Since the amount of input information to be processed in these inference requests is usually small, the first client can execute the inference requests efficiently without interacting with the inference node, thus saving communication resources and reducing latency.

[0093] Furthermore, the input information in the inference request can be either the original input information provided by the user or multiple inference units obtained by dividing the original input information; this application does not limit this. Based on the foregoing description of inference units, in this application, an inference unit refers to the basic data unit used in inference with a large model. That is, the large model uses inference units as basic data processing units to implement inference. For example, an inference unit is a word segment obtained by segmenting text; an image block obtained by segmenting an image; an audio segment obtained by segmenting audio; an image block obtained by segmenting each video frame, etc. This application does not limit the type of inference unit.

[0094] 602. The inference node receives the inference request sent by the first client.

[0095] 603. The inference node determines the first inference unit in the inference unit corresponding to the inference request.

[0096] In this embodiment, the number of inference units corresponding to the inference request is multiple. The first inference unit refers to the inference unit determined by the inference node that is to send corresponding intermediate data to the first client. The number of first inference units can be one or more, and this application does not limit this. Based on the aforementioned step 601, the form of the input information in the inference request can be the original input information provided by the user, or it can be multiple inference units obtained by dividing the original input information. Accordingly, in this step, if the form of the input information in the inference request received by the inference node is the original input information provided by the user, then the inference node divides the original input information to obtain the inference unit corresponding to the inference request and determines the first inference unit. If the form of the input information in the inference request received by the inference node is multiple inference units obtained by dividing the original input information, then the inference node determines the first inference unit from the multiple inference units.

[0097] Schematic illustration: The inference node can access a storage system used to store intermediate data of inference units generated using a large model. In the storage system, each inference unit corresponds to one set of intermediate data. This application does not limit the implementation of the storage system. For example, when the inference node is a cloud node, the storage system can be deployed in the cloud platform's data center; or, when the inference node is a terminal device, the storage system is deployed locally on the terminal device. Furthermore, the intermediate data of the inference units stored in the storage system is historical information, including one or more of the following: intermediate data of the inference unit corresponding to a historical inference request, and intermediate data of the inference unit corresponding to the inference result. This application does not limit the source of the intermediate data of the inference units in the storage system. For example, it can be intermediate data from a client (here, the client refers to any client in the large model inference system), or intermediate data generated by the inference node using the large model during a historical time period, etc.

[0098] In some embodiments, the storage system stores intermediate data of inference units based on an index structure. The index structure can be, for example, a tree structure (in which case the storage system can be understood as a storage system built on a knowledge tree), or other forms of index structure, such as an information set. The intermediate data of the inference units can be directly stored in this information set, and the correspondence between inference units and intermediate data, as well as the dependencies between each inference unit, can be established within this information set. This application does not limit the form of the index structure. It should be understood that in large-model inference scenarios, for any inference request, the inference units corresponding to that request are usually related, and for any inference result, the inference units corresponding to that result are usually also related. Accordingly, when storing the intermediate data of these inference units, the storage system can store it according to the relationships between different inference units. Illustratively, for text, among the multiple inference units corresponding to the text, non-first inference units depend on each inference unit preceding it. For images, among the multiple inference units corresponding to the image, non-first inference units depend on each inference unit preceding it. That is, non-first inference units depend on each inference unit preceding them. For example, for text, suppose the text is a user's question "What color is an apple?" The multiple inference units corresponding to this text are "apple," "is," "what," and "color." The inference unit "is" depends on the inference unit "apple," the inference unit "what" depends on the inference units "apple" and "is," and the inference unit "color" depends on the inference units "apple," "is," and "what." Similarly, for images, such as... Figure 7 As shown, Figure 7 This is a schematic diagram of an inference unit corresponding to an image provided in an embodiment of this application. Figure 7The image shown includes 6 reasoning units, according to Figure 7 The order of the various reasoning units is as follows: Reasoning unit 2 depends on reasoning unit 1, reasoning unit 3 depends on reasoning units 1 and 2, reasoning unit 4 depends on reasoning units 1, 2 and 3, reasoning unit 5 depends on reasoning units 1, 2, 3 and 4, and reasoning unit 6 depends on reasoning units 1, 2, 3, 4 and 5.

[0099] In this step, the inference node searches the storage system based on the inference unit corresponding to the inference request, and then determines the first inference unit based on the search results. The search results indicate whether a matching inference unit exists in the storage system. In some embodiments, the index structure of the storage system includes multiple inference units, each corresponding to intermediate data. Thus, the inference node's search of the storage system based on the inference unit corresponding to the inference request includes: searching the storage system's index structure based on the dependencies of the inference unit in the inference request to determine whether a matching inference unit exists in the storage system. To improve search accuracy, the storage system's index structure can also be searched using prefix matching; this application does not limit this approach.

[0100] The following describes how the inference node determines the first inference unit under different search results, illustrating the following two cases:

[0101] Case 1: If the second inference unit and its intermediate data are retrieved in the storage system, the first inference unit is determined based on the second inference unit and the inference unit corresponding to the inference request.

[0102] In this context, the second inference unit matches any one of the inference units corresponding to the inference request, and the first inference unit includes the second inference unit. That is, if the inference node retrieves inference units in the storage system that match the inference unit corresponding to the inference request, then these inference units are considered the first inference unit. For example, if the number of inference units corresponding to the inference request is 10, and the inference node retrieves 3 second inference units and their intermediate data in the storage system, then these 3 second inference units are considered the first inference unit.

[0103] In some embodiments, when an inference node retrieves a second inference unit and its intermediate data, it determines reference information based on the second inference unit and the inference unit corresponding to the inference request. Based on whether the reference information meets certain conditions, it further determines a first inference unit. The reference information indicates the number of inference units that, in addition to the second inference unit, need to generate intermediate data to execute the inference request. For example, if the number of inference units corresponding to the inference request is 10, and the inference node retrieves 3 second inference units and their intermediate data from the storage system, then the number of inference units indicated by the reference information is 7. The conditions that the reference information meets can be set according to requirements. For example, the condition may be that the number of inference units indicated by the reference information is less than a threshold. This threshold can be set according to actual needs and is not limited here.

[0104] The following describes how inference nodes determine the first inference unit based on whether the reference information meets the conditions, illustrating the following scenarios:

[0105] Case 1-A: If the reference information meets the conditions, the second inference unit is used as the first inference unit. That is, if the number of inference units indicated by the reference information is less than the threshold, it means that there are fewer inference units that need to generate intermediate data. The first client can efficiently generate the intermediate data of these inference units locally. Based on this, the retrieved second inference units can be used as all the first inference units. In this case, the number of retrieved second inference units is equal to the number of first inference units.

[0106] Scenario 1-B: If the reference information does not meet the conditions, a third inference unit is determined based on the inference unit corresponding to the inference request and the second inference unit. The first inference unit includes the second and third inference units, and the third inference unit refers to the inference unit other than the second inference unit in the inference unit corresponding to the inference request. The number of third inference units can be one or more, and this application does not limit this. Furthermore, the inference units other than the second inference unit in the inference unit corresponding to the inference request can be all inference units other than the second inference unit, or it can be some of the inference units other than the second inference unit. In other words, if the number of inference units indicated by the reference information is greater than or equal to a threshold, it indicates that there are many inference units that still need to generate intermediate data. The first client may need to consume a lot of computing resources to generate the intermediate data for these inference units. Based on this, the retrieved second inference unit and one or more third inference units from the inference unit corresponding to the inference request can be used as the first inference unit. For example, if the number of inference units corresponding to the inference request is 10, and the condition that the reference information meets is that the number of inference units indicated by the reference information is less than the threshold of 3, and the inference node retrieves 3 second inference units in the storage system, then determines that the number of inference units indicated by the reference information is 7, which does not meet the condition. Based on this, one or more inference units other than the second inference units are determined from the inference units corresponding to the inference request. That is, the inference node can use all 7 remaining inference units other than the second inference units as third inference units, or it can select some inference units from these remaining 7 inference units as third inference units. Accordingly, the number of third inference units is less than or equal to 7, and the number of first inference units is less than or equal to 10.

[0107] Based on the above scenarios 1-A and 1-B, the method of determining the first inference unit based on whether the reference information meets the conditions is introduced. In this way, the first inference unit is determined based on the computing power of the first client. For the first client to execute the inference request, if the number of inference units that the first client still needs to generate intermediate data after the inference node sends the intermediate data of the retrieved inference units to the first client is large, the inference node can determine one or more additional first inference units based on the retrieved inference units, thereby reducing the computing pressure on the first client and improving the inference efficiency of large models on the first client side.

[0108] Scenario 2: If no matching inference unit is found in the storage system, the inference unit corresponding to the inference request is used as the first inference unit. In this case, the number of inference units corresponding to the inference request is equal to the number of first inference units. This process means that if the inference node does not find a matching inference unit in the storage system, all inference units corresponding to the inference request are used as the first inference units. This eliminates the need for the first client's local computing resources to generate intermediate data for these inference units, reducing the computational burden on the first client and improving the efficiency of large model inference on the client side.

[0109] 604. The inference node sends the intermediate data of the first inference unit to the first client.

[0110] In this embodiment, the inference node sends the intermediate data of the first inference unit to the first client based on the communication method between the inference node and the first client. The communication method between the inference node and the first client is described in the foregoing description of the implementation environment of this application and will not be repeated here.

[0111] Furthermore, based on the aforementioned step 603, the number of first inference units can be one or more. Moreover, when there are multiple first inference units, for any one first inference unit, it can be an inference unit that has stored intermediate data in the storage system or an inference unit that has not stored intermediate data in the storage system. Accordingly, if the first inference unit is an inference unit that has stored intermediate data in the storage system, the inference node reads the intermediate data of the first inference unit from the storage system and sends the intermediate data of the first inference unit to the first client; if the first inference unit is an inference unit that has not stored intermediate data in the storage system, the inference node generates the intermediate data of the first inference unit using the large model and sends the intermediate data of the first inference unit to the first client.

[0112] 605. The first client receives intermediate data from the first inference unit.

[0113] 606. The first client executes inference requests based on the intermediate data and large model of the first inference unit.

[0114] In this embodiment, based on the aforementioned step 603, the number of first inference units can be one or more. Furthermore, when there are multiple first inference units, these units can be all the inference units corresponding to the inference request, or only a portion of the inference units corresponding to the inference request. Accordingly, if the intermediate data of the first inference units received by the first client is the intermediate data of all the inference units corresponding to the inference request, the first client executes the inference request based on the received intermediate data of the first inference units and the large model, and outputs the inference result. If the intermediate data of the first inference units received by the first client is the intermediate data of only a portion of the inference units corresponding to the inference request, the remaining inference units are referred to as fourth inference units. That is, the inference units corresponding to the inference request include not only the first inference units but also the fourth inference units. The fourth inference units differ from the first inference units; the first client generates the intermediate data of the fourth inference units using the large model; based on the intermediate data of the first and fourth inference units, the inference request is executed, and the inference result is output.

[0115] In some embodiments, based on the inference request processing method shown in steps 601 to 606 above, in the large model inference system provided by this application, the client can send the intermediate data of the locally generated inference unit to the inference node, so that the inference node can update the intermediate data stored in the storage system in a timely manner, providing technical support for realizing cross-client intermediate data reuse. Indicatively, the large model inference system includes one or more clients. The inference node can receive intermediate data of the inference unit from any client. In some instances, the client first sends the inference unit to the inference node. If the inference node's storage system does not store the intermediate data of the inference unit, it sends a data transmission instruction to the client that stores the intermediate data of the inference unit, so that the client receiving the data transmission instruction sends the intermediate data of the inference unit to the inference node. It should be understood that the inference unit has a smaller data volume than the intermediate data of the inference unit, and the inference node's storage system may already store the intermediate data of the inference unit. Based on this, having the inference node decide whether the client needs to send the intermediate data of the inference unit can avoid the transmission and processing of redundant data, saving communication and computing resources.

[0116] The following section describes the implementation of the above process, taking the inference unit received by the inference node as the fifth inference unit as an example.

[0117] Schematic, the inference node searches the storage system based on the received fifth inference unit. The storage system stores intermediate data of the inference units generated using the large model. If the intermediate data of the fifth inference unit is not found in the storage system, a data transmission instruction is sent to the second client in the large model inference system. This data transmission instruction instructs the intermediate data of the fifth inference unit to be sent to the inference node. The second client stores the intermediate data of the fifth inference unit. It should be noted that the second client can be the same as the aforementioned first client, or it can be different; this application does not limit this. Furthermore, the inference node receives the intermediate data of the fifth inference unit sent by the second client and stores the intermediate data of the fifth inference unit in the storage system.

[0118] In some embodiments, if an inference node receives a fifth inference unit sent by multiple third clients in a large model inference system, the inference node determines a second client from among the multiple third clients based on one or more of the following: network transmission status, data transmission latency, and the location of the third clients. Each third client stores intermediate data of the fifth inference unit; this application does not limit the number of third clients. That is, if the intermediate data of the same inference unit is stored on multiple clients, the inference node can select one client to send a data transmission command based on one or more of the following: network transmission status, data transmission latency, and the location of these clients, thereby saving communication resources. Of course, multiple clients can also be selected to improve the success rate of data transmission; this application does not limit this.

[0119] Schematic representation: Network transmission status includes, for example, network bandwidth, packet loss rate, jitter, etc. Network transmission status reflects the basic quality of the network link. When intermediate data of the same inference unit is stored on multiple clients, the inference node can select the client with the acceptable network link quality based on the network transmission status. Data transmission latency reflects the time taken to transmit intermediate data of the inference unit between the client and the inference node. When intermediate data of the same inference unit is stored on multiple clients, the inference node can select the client with the acceptable latency based on the data transmission latency. Client location reflects the communication distance between the client and the inference node. When intermediate data of the same inference unit is stored on multiple clients, the inference node can select the client with the acceptable communication distance based on the locations of the multiple clients. In some embodiments, the inference node may also consider one or more of the above factors to determine the client that needs to upload intermediate data. In practical applications, this can be configured according to requirements, and this application does not impose any limitations on this.

[0120] In addition, in some scenarios, if the inference node receives multiple different inference units, and the intermediate data of each inference unit is stored on multiple clients, the inference node can determine the clients that need to upload intermediate data based on one or more of the network transmission status, data transmission latency, and the location of these clients. This can distribute the data transmission pressure from the clients to the inference node and improve the utilization of communication resources.

[0121] For example, the inference node receives inference unit 1 and inference unit 2. The intermediate data of inference unit 1 is stored in clients 1-3, and the intermediate data of inference unit 2 is stored in clients 4-6. Clients 1-3 are located in regions A, B, and C, respectively, and clients 4-6 are located in regions A, B, and C, respectively. Since the communication distance between region A and the inference node is the shortest, a data transmission instruction is sent to client 1 to make client 1 send the intermediate data of inference unit 1 to the inference node, and a data transmission instruction is sent to client 4 to make client 4 send the intermediate data of inference unit 2 to the inference node, thereby reducing communication latency.

[0122] For example, an inference node receives 100 different inference units, each corresponding to one of 100 clients. 40 clients corresponding to inference units are located in region A, and 60 clients correspond to inference units in region B. Considering the network transmission status of the inference node, 20 clients are selected from region A, and 20 clients from region B. These 40 clients are instructed to send the intermediate data of the inference units within the first time period (e.g., one hour) to avoid the inference node receiving too much intermediate data from the same region, which could cause network congestion. Alternatively, the remaining 60 clients can be instructed to send the intermediate data of the inference units in the second time period (e.g., one hour later), thereby achieving load balancing and preventing the inference node from simultaneously receiving too much intermediate data within a certain time period.

[0123] The above-described method for determining the client that needs to upload intermediate data by the inference node is merely an illustrative example and does not constitute a limitation of this application. The inference node may also randomly select the client or adopt other strategies to select the client.

[0124] In summary, in the large model inference system provided in this application, the client and inference nodes can collaboratively implement large model inference functionality through interaction. Specifically, when the first client executes an inference request, it sends the request to the inference node. The inference node then sends intermediate data from one or more first inference units corresponding to the inference request to the first client. This eliminates the need for the first client to expend local computing resources to generate the intermediate data for these inference units, enabling the reuse of intermediate data during the client's large model inference process and effectively improving the efficiency of large model inference. Furthermore, the client can send locally generated intermediate data of inference units to the inference node, allowing the inference node to update the intermediate data stored in the storage system in a timely manner, providing technical support for cross-client intermediate data reuse.

[0125] Based on the above Figure 6 The illustrated embodiment, taking the interaction between the client and the inference node as an example, introduces the way a large-model inference system handles inference requests. On one hand, the client and the inference node in the large-model inference system can collaborate to achieve large-model inference functionality through interaction; this process can also be understood as the download stage (or usage stage) of intermediate data for the inference unit. On the other hand, the client in the large-model inference system can send the intermediate data of the locally generated inference unit to the inference node, providing technical support for realizing cross-client intermediate data reuse; this process can also be understood as the upload stage of intermediate data for the inference unit. Based on the foregoing introduction to the large-model inference system, it can be seen that there are multiple implementation methods for the inference node; the following refers to... Figures 8 to 10 Taking different implementation methods of inference nodes as examples, this paper provides an exemplary description of how a large model inference system processes inference requests.

[0126] Figure 8 This is a schematic diagram illustrating how a large-model inference system processes inference requests, as provided in an embodiment of this application. Figure 8 As shown, the large-scale model inference system includes a client and inference nodes, where the inference nodes are cloud nodes. In some embodiments, the large-scale model inference system also includes edge nodes. Figure 8 In the scenario shown, there is no limitation on whether the large model inference system includes edge nodes.

[0127] like Figure 8As shown in Figure (a), multiple terminal devices run clients. Taking any one client (such as the first client 801) as an example, the intermediate data download stage of the inference unit is described. Schematic, the first client generates an inference request containing the input information based on the user-provided input information. When the input information of the inference request is a long sequence, the client sends the inference request to the inference node. In this case, the inference node is, for example, the scheduling node of the cloud platform. The inference node receives the inference request, searches the storage system based on the inference unit corresponding to the inference request, and then determines the first inference unit based on the search results. The first inference unit refers to the inference unit determined by the inference node that is to send corresponding intermediate data to the first client. Specifically, for any first inference unit, if the first inference unit is an inference unit whose intermediate data has been stored in the storage system, the inference node sends the intermediate data of the first inference unit in the storage system to the first client; if the first inference unit is an inference unit whose intermediate data has not been stored in the storage system, the inference node generates the intermediate data of the first inference unit using a large model and sends the intermediate data of the first inference unit to the first client. When the inference node is the scheduling node of the cloud platform, the inference node can control the computing nodes on the cloud platform that have large model inference capabilities to generate intermediate data for the first inference unit.

[0128] In some embodiments, where the large model inference system further includes edge nodes, since the edge nodes act as an intermediate layer between the cloud and the edge, they can both receive instructions and data from the cloud and provide local computing and storage services to the edge devices. Therefore, edge nodes can store intermediate data of inference units, such as intermediate data of inference units whose access counts have reached a threshold. Edge nodes can also generate intermediate data for inference units using the large model. Transmitting the intermediate data of the first inference unit through edge nodes can shorten the data transmission path. Illustratively, after the inference node determines the first inference unit, for any given first inference unit, the inference node sends the intermediate data of the first inference unit to the first client, including any of the following:

[0129] (1) The intermediate data of the first inference unit is sent to the first client via an edge node. In some embodiments, if the inference node determines that the intermediate data of the first inference unit is stored on an edge node, it sends the intermediate data of the first inference unit to the first client via the edge node, for example, by sending a data transmission request to the edge node to cause the edge node to send the intermediate data of the first inference unit to the client. In other embodiments, if the intermediate data of the first inference unit is not stored on the edge node, the inference node sends an intermediate data generation request to the edge node to cause the edge node to generate the intermediate data of the first inference unit using a large model and then send the intermediate data of the first inference unit to the first client.

[0130] (2) Control the first client to obtain intermediate data of the first inference unit from the edge node. That is, the inference node can control the first client to send a data transmission request to the edge node to obtain the intermediate data of the first inference unit. For the edge node, if the edge node stores the intermediate data of the first inference unit, it sends the intermediate data of the first inference unit to the first client. If the edge node does not store the intermediate data of the first inference unit, it generates the intermediate data of the first inference unit locally using the large model and then sends the intermediate data of the first inference unit to the first client.

[0131] It should be noted that the above-described method of sending intermediate data of the first inference unit from the inference node to the first client is merely illustrative and does not constitute a limitation of this application. In some embodiments, the configuration can be flexibly adjusted according to the deployment method of the large model inference system. For example, if multiple edge nodes store intermediate data of multiple first inference units, the first client can obtain intermediate data of multiple first inference units from these edge nodes based on the communication distance between different edge nodes and the first client to accelerate the acquisition process. Alternatively, intermediate data of multiple first inference units can be obtained from these edge nodes according to the priority of intermediate data of different inference units. For example, low-priority intermediate data can be obtained from edge nodes farther away from the first client. This application does not limit this approach.

[0132] like Figure 8 As shown in Figure (b), multiple terminal devices run clients. The following describes the intermediate data upload stage of the inference unit. Illustratively, for any client, the client can send the intermediate data of the locally generated inference unit to the inference node. During the intermediate data upload stage, for the inference node, it searches its storage system based on the received fifth inference unit. If the inference node's storage system does not store the intermediate data of that inference unit, it sends a data transmission instruction to the client (such as the second client 802) that stores the intermediate data of the fifth inference unit, so that the client receiving the data transmission instruction sends the intermediate data of the fifth inference unit to the inference node. If multiple fifth inference units are received from multiple third clients, the second client is determined from among the multiple third clients based on one or more of the network transmission status, data transmission latency, and the location of the third clients. The implementation of this process is described above. Figure 6 The embodiments shown are not described in detail here.

[0133] In some embodiments, where the large model inference system also includes edge nodes, the edge nodes, acting as an intermediate layer between the cloud and the edge, can both receive instructions and data from the cloud and provide local computing and storage services to the edge devices. Therefore, the second client can send intermediate data from the fifth inference unit to the edge node, which then merges the received intermediate data from the inference units and sends it to the inference node. That is, the edge node can receive intermediate data from inference units from multiple clients, merge the received intermediate data, and send it to the inference node, avoiding the inference node receiving a large amount of intermediate data transmitted from numerous clients in a dispersed manner.

[0134] After the above Figure 8 The illustrated embodiment describes the process by which a large-model inference system handles inference requests when the inference node is a cloud node. Through this method, the client can leverage the storage and computing capabilities of the cloud platform to improve the efficiency of large-model inference.

[0135] Figure 9 This is a schematic diagram illustrating another large-model inference system processing inference requests provided in an embodiment of this application. For example... Figure 9 As shown, the large-model inference system includes a client and inference nodes, with the inference nodes being edge nodes. In some embodiments, the large-model inference system also includes cloud nodes. Figure 9 In the scenario shown, there is no limitation on whether the large model inference system includes cloud nodes.

[0136] like Figure 9 As shown in Figure (a), multiple terminal devices run clients. Taking any one client (such as the first client 901) as an example, the intermediate data download stage of the inference unit is described. Schematic, the first client generates an inference request containing the input information based on the user-provided input information. If the input information of the inference request is a long sequence, the client sends the inference request to the inference node. In this case, the inference node is, for example, the edge node closest to the first client. The process by which the inference node determines the first inference unit and sends the intermediate data of the first inference unit to the first client is the same as described above. Figure 8 The embodiments shown are similar and will not be described again.

[0137] In some embodiments, where the large model inference system also includes cloud nodes, since the edge nodes act as an intermediate layer between the cloud and the device side, they can both receive instructions and data from the cloud side and provide local computing and storage services to the device side. Therefore, if the cloud node stores the intermediate data of the first inference unit, the edge node can obtain the intermediate data of the first inference unit from the cloud node.

[0138] like Figure 9As shown in Figure (b), multiple terminal devices run clients. The following describes the intermediate data upload stage of the inference unit. Illustratively, for any client, the client can send the intermediate data of the locally generated inference unit to the inference node. During the intermediate data upload stage, for the inference node, it searches its storage system based on the received fifth inference unit. If the inference node's storage system does not store the intermediate data of that inference unit, it sends a data transmission instruction to the client (such as the second client 902) that stores the intermediate data of the fifth inference unit, so that the client receiving the data transmission instruction sends the intermediate data of the fifth inference unit to the inference node. If multiple fifth inference units are received from third clients, the second client is determined from among the multiple third clients based on one or more of the network transmission status, data transmission latency, and the location of the third clients. The implementation of this process is described above. Figure 6 The embodiments shown are not described in detail here.

[0139] After the above Figure 9 The illustrated embodiment describes the process by which a large model inference system handles inference requests when the inference node is an edge node. Through this method, the client can leverage the storage and computing capabilities of the edge node to improve the efficiency of large model inference.

[0140] Figure 10 This is a schematic diagram illustrating how a large-model inference system processes inference requests, as provided in an embodiment of this application. Figure 10 As shown, the large-scale model inference system includes a client and inference nodes, with the inference nodes being terminal devices. In this scenario, the terminal devices corresponding to the client and the inference nodes can be deployed on a home network.

[0141] like Figure 10 As shown in Figure (a), multiple terminal devices run clients. Taking any one client (such as the first client 1001) as an example, the intermediate data download stage of the inference unit is described. Schematic, the first client generates an inference request containing the input information based on the user-provided input information. If the input information of the inference request is a long sequence, the client sends the inference request to the inference node. In this case, the first client is, for example, a mobile phone used by the user, and the inference node is, for example, a desktop computer. The process by which the inference node determines the first inference unit and sends the intermediate data of the first inference unit to the first client is the same as described above. Figure 8 and Figure 9 The illustrated embodiments are similar and will not be described again. In some embodiments, the inference node is communicatively connected to a network node, and the inference node obtains intermediate data of the first inference unit through the network node, which may be a server or device accessed through a public network.

[0142] like Figure 10 As shown in Figure (b), multiple terminal devices run clients. The following describes the intermediate data upload stage of the inference unit. Illustratively, for any client, the client can send the intermediate data of the locally generated inference unit to the inference node. During the intermediate data upload stage, for the inference node, it searches its storage system based on the received fifth inference unit. If the inference node's storage system does not store the intermediate data of that inference unit, it sends a data transmission instruction to the client (such as the second client 1002) that stores the intermediate data of the fifth inference unit, so that the client receiving the data transmission instruction sends the intermediate data of the fifth inference unit to the inference node. If multiple fifth inference units are received from multiple third clients, the second client 1002 is determined from among the multiple third clients based on one or more of the network transmission status, data transmission latency, and the location of the third clients. The implementation of this process is described above. Figure 6 The embodiments shown are not described in detail here.

[0143] After the above Figure 10 The illustrated embodiment describes the process by which a large model inference system handles inference requests when the inference node is a terminal device. Through this method, the client can leverage the storage and computing capabilities of other terminal devices to improve the efficiency of large model inference.

[0144] The methods of the embodiments of this application have been described above, and the apparatus of the embodiments of this application will be described below. It should be understood that the technical features described in the method embodiments are also applicable to the apparatus embodiments below.

[0145] refer to Figure 11 , Figure 11 This is a schematic diagram of the structure of a reasoning request processing device provided in an embodiment of this application. Figure 11 As shown, the device 1100 includes a receiving module 1101, a determining module 1102, and a sending module 1103.

[0146] The first receiving module 1101 is used to receive the inference request sent by the first client;

[0147] The first determining module 1102 is used to determine the first reasoning unit in the reasoning unit corresponding to the reasoning request. The reasoning unit corresponding to the reasoning request refers to the basic data unit when using a large model for reasoning.

[0148] The first sending module 1103 is used to send intermediate data of the first inference unit to the first client, and the first client is used to execute inference requests based on the intermediate data of the first inference unit and the large model.

[0149] In some embodiments, the determining module 1102 includes:

[0150] The retrieval unit is used to retrieve data from the storage system based on the inference unit corresponding to the inference request. The storage system is used to store intermediate data of the inference units generated using the large model.

[0151] The determining unit is configured to: if a second inference unit and intermediate data of the second inference unit are retrieved in the storage system, determine a first inference unit based on the second inference unit and the inference unit corresponding to the inference request, wherein the second inference unit matches any inference unit corresponding to the inference request, and the first inference unit includes the second inference unit; if no inference unit matching the inference unit corresponding to the inference request is retrieved in the storage system, the inference unit corresponding to the inference request is taken as the first inference unit.

[0152] In some embodiments, the determining unit is configured to:

[0153] Based on the second inference unit and the inference unit corresponding to the inference request, reference information is determined. The reference information indicates the number of inference units that need to generate intermediate data in addition to the second inference unit when executing the inference request.

[0154] If the reference information meets the conditions, the second reasoning unit will be used as the first reasoning unit; or,

[0155] If the reference information does not meet the conditions, a third reasoning unit is determined based on the reasoning unit corresponding to the reasoning request and the second reasoning unit. The first reasoning unit includes the second reasoning unit and the third reasoning unit. The third reasoning unit refers to the reasoning unit other than the second reasoning unit in the reasoning unit corresponding to the reasoning request.

[0156] In some embodiments, the inference node is a cloud node, and the large model inference system further includes edge nodes. The sending module 1103 is used for any of the following:

[0157] The intermediate data of the first inference unit is sent to the first client via edge nodes; or,

[0158] Control the first client to obtain intermediate data of the first inference unit from the edge node.

[0159] In some embodiments, the inference node is an edge node, and the apparatus further includes: a first generation module for generating intermediate data for the first inference unit using a large model; or, a first acquisition module for acquiring intermediate data for the first inference unit from a cloud node if the large model inference system also includes a cloud node.

[0160] In some embodiments, the inference node is a terminal device, and the device further includes: a second generation module for generating intermediate data of the first inference unit using a large model; or a second acquisition module for acquiring intermediate data of the first inference unit through a network node.

[0161] In some embodiments, the device further includes:

[0162] The retrieval module is used to retrieve the storage system based on the received fifth inference unit. The storage system is used to store intermediate data of inference units generated using a large model.

[0163] The second sending module is also used to send a data transmission instruction to the second client in the large model inference system if the intermediate data of the fifth inference unit is not found in the storage system. The data transmission instruction instructs the intermediate data of the fifth inference unit to be sent to the inference node. The second client stores the intermediate data of the fifth inference unit.

[0164] In some embodiments, if the inference node receives a fifth inference unit sent by multiple third clients in a large model inference system, the apparatus further includes:

[0165] The second determining module is used to determine the second client from multiple third clients based on one or more of the following: network transmission status, data transmission delay, and the location of the third client.

[0166] In some embodiments, the device further includes:

[0167] The second receiving module is used to receive intermediate data from the fifth inference unit sent by the second client;

[0168] The storage module is used to store the intermediate data of the fifth inference unit to the storage system.

[0169] Through the aforementioned device, the client and the inference node can collaborate to achieve large model inference functionality through interaction. When the first client executes an inference request, it sends the inference request to the inference node. The inference node then sends the intermediate data of one or more first inference units corresponding to the inference request to the first client. In this way, the first client does not need to expend local computing resources to generate the intermediate data of these inference units, thus enabling the reuse of intermediate data of inference units during the client's large model inference process, thereby effectively improving the efficiency of large model inference.

[0170] The device 1100 described above is configured in the inference node of the above method embodiment. The modules in the device 1100 and the other operations and / or functions described above are respectively for implementing various steps and methods implemented by the inference node in the method embodiment. For specific details, please refer to the above method embodiment. For the sake of brevity, they will not be repeated here.

[0171] refer to Figure 12 , Figure 12 This is a schematic diagram of another inference request processing device provided in an embodiment of this application. Figure 12 As shown, the device 1200 includes a transmitting module 1201, a receiving module 1202, and an execution module 1203.

[0172] Sending module 1201 is used to send inference requests to inference nodes;

[0173] The receiving module 1202 is used to receive intermediate data of the first inference unit in the inference unit corresponding to the inference request. The inference unit corresponding to the inference request refers to the basic data unit when performing inference using a large model.

[0174] Execution module 1203 is used to execute inference requests using a large model based on intermediate data from the first inference unit.

[0175] In some embodiments, the reasoning unit corresponding to the reasoning request further includes a fourth reasoning unit, which is different from the first reasoning unit; the device further includes:

[0176] The generation module is used to generate intermediate data for the fourth inference unit using the large model;

[0177] The execution module 1203 is used to: execute inference requests using a large model based on the intermediate data of the first inference unit and the intermediate data of the fourth inference unit.

[0178] In some embodiments, the sending module 1201 is further configured to perform one or more of the following:

[0179] Send the inference request corresponding to the inference unit to the inference node;

[0180] The inference unit corresponding to the inference result of the inference request sent to the inference node.

[0181] In some embodiments, the receiving module 1202 is further configured to receive a data transmission instruction sent by the inference node; the sending module 1201 is further configured to send the intermediate data of the inference unit indicated by the data transmission instruction to the inference node.

[0182] Through the aforementioned device, the client and the inference node can collaborate to achieve large model inference functionality through interaction. When the first client executes an inference request, it sends the inference request to the inference node. The inference node then sends the intermediate data of one or more first inference units corresponding to the inference request to the first client. In this way, the first client does not need to expend local computing resources to generate the intermediate data of these inference units, thus enabling the reuse of intermediate data of inference units during the client's large model inference process, thereby effectively improving the efficiency of large model inference.

[0183] The device 1200 described above is configured in the client of the above method embodiment. The modules in the device 1200 and the other operations and / or functions described above are respectively for implementing various steps and methods implemented by the client in the method embodiment. For specific details, please refer to the above method embodiment. For the sake of brevity, they will not be repeated here.

[0184] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when processing inference requests. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the inference request processing apparatus and the inference request processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0185] In this embodiment, the apparatus, device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0186] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0187] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0188] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0189] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0190] In the description of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences.

[0191] In this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0192] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0193] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A large-scale model reasoning system, characterized in that, The large-scale model inference system includes a first client and inference nodes; The inference node is used for: Receive the inference request sent by the first client; The first reasoning unit in the reasoning unit corresponding to the reasoning request is determined, and the reasoning unit corresponding to the reasoning request refers to the basic data unit when using a large model for reasoning; Send the intermediate data of the first inference unit to the first client; The first client is used to execute the inference request based on the intermediate data of the first inference unit and the large model.

2. The system according to claim 1, characterized in that, The inference node is used for: Based on the reasoning unit corresponding to the reasoning request, a retrieval is performed in the storage system, which is used to store intermediate data of the reasoning unit generated using the large model; If a second inference unit and its intermediate data are retrieved in the storage system, the first inference unit is determined based on the second inference unit and the inference unit corresponding to the inference request. The second inference unit matches any one of the inference units corresponding to the inference request. The first inference unit includes the second inference unit. If no matching inference unit is found in the storage system, the inference unit corresponding to the inference request is used as the first inference unit.

3. The system according to claim 2, characterized in that, The inference node is used for: Based on the second inference unit and the inference unit corresponding to the inference request, reference information is determined, wherein the reference information indicates the number of inference units that need to generate intermediate data in addition to the second inference unit when executing the inference request; If the reference information meets the conditions, the second reasoning unit is used as the first reasoning unit; or... If the reference information does not meet the conditions, a third reasoning unit is determined based on the reasoning unit corresponding to the reasoning request and the second reasoning unit. The first reasoning unit includes the second reasoning unit and the third reasoning unit. The third reasoning unit refers to the reasoning unit other than the second reasoning unit among the reasoning units corresponding to the reasoning request.

4. The system according to claim 3, characterized in that, The condition is that the number of inference units indicated by the reference information is less than a threshold.

5. The system according to any one of claims 1 to 4, characterized in that, The reasoning unit corresponding to the reasoning request further includes a fourth reasoning unit, which is different from the first reasoning unit; the first client is also used for: The intermediate data for the fourth inference unit is generated using the large model. The inference request is executed based on the intermediate data of the first inference unit and the intermediate data of the fourth inference unit.

6. The system according to any one of claims 1 to 5, characterized in that, The inference node is a cloud node, and the large model inference system also includes edge nodes. The inference node is used for any of the following: The intermediate data of the first inference unit is sent to the first client via the edge node; or, The first client is controlled to obtain intermediate data of the first inference unit from the edge node.

7. The system according to any one of claims 1 to 5, characterized in that, The inference node is an edge node, and the inference node is also used for: The large model is used to generate intermediate data for the first inference unit; or, In the case where the large model inference system also includes cloud nodes, intermediate data of the first inference unit is obtained from the cloud nodes.

8. The system according to any one of claims 1 to 5, characterized in that, The inference node is a terminal device, and the inference node is further used for: The large model is used to generate intermediate data for the first inference unit; or, The intermediate data of the first inference unit is obtained through network nodes.

9. The system according to any one of claims 1 to 8, characterized in that, The inference node is also used for: The storage system is retrieved based on the received fifth inference unit, the storage system being used to store intermediate data of the inference units generated using the large model; If the intermediate data of the fifth inference unit is not found in the storage system, a data transmission instruction is sent to the second client in the large model inference system. The data transmission instruction instructs the intermediate data of the fifth inference unit to be sent to the inference node. The second client stores the intermediate data of the fifth inference unit.

10. The system according to claim 9, characterized in that, If the inference node receives the fifth inference unit sent by multiple third clients in the large model inference system, the inference node is further configured to: The second client is determined from the plurality of third clients based on one or more of the following: network transmission status, data transmission latency, and the location of the third client.

11. The system according to claim 9 or 10, characterized in that, The inference node is also used for: Receive intermediate data from the fifth inference unit sent by the second client; The intermediate data of the fifth inference unit is stored in the storage system.

12. The system according to any one of claims 9 to 11, characterized in that, The inference node is a cloud node, and the large model inference system also includes edge nodes; The second client is used to send the intermediate data of the fifth inference unit to the edge node; The edge node is used to merge the intermediate data received from the inference unit and send it to the inference node.

13. A method for processing inference requests, characterized in that, Executed by an inference node in a large-scale model inference system, the large-scale model inference system further comprising a first client, the method includes: Receive the inference request sent by the first client; The first reasoning unit in the reasoning unit corresponding to the reasoning request is determined, and the reasoning unit corresponding to the reasoning request refers to the basic data unit when using a large model for reasoning; The intermediate data of the first inference unit is sent to the first client, and the first client is used to execute the inference request based on the intermediate data of the first inference unit and the large model.

14. The method according to claim 13, characterized in that, The step of determining the first reasoning unit among the reasoning units corresponding to the reasoning request includes: Based on the reasoning unit corresponding to the reasoning request, a retrieval is performed in the storage system, which is used to store intermediate data of the reasoning unit generated using the large model; If a second inference unit and its intermediate data are retrieved in the storage system, the first inference unit is determined based on the second inference unit and the inference unit corresponding to the inference request. The second inference unit matches any one of the inference units corresponding to the inference request. The first inference unit includes the second inference unit. If no matching inference unit is found in the storage system, the inference unit corresponding to the inference request is used as the first inference unit.

15. The method according to claim 14, characterized in that, Determining the first reasoning unit based on the second reasoning unit and the reasoning unit corresponding to the reasoning request includes: Based on the second inference unit and the inference unit corresponding to the inference request, reference information is determined, wherein the reference information indicates the number of inference units that need to generate intermediate data in addition to the second inference unit when executing the inference request; If the reference information meets the conditions, the second reasoning unit is used as the first reasoning unit; or... If the reference information does not meet the conditions, a third reasoning unit is determined based on the reasoning unit corresponding to the reasoning request and the second reasoning unit. The first reasoning unit includes the second reasoning unit and the third reasoning unit. The third reasoning unit refers to the reasoning unit other than the second reasoning unit among the reasoning units corresponding to the reasoning request.

16. The method according to any one of claims 13 to 15, characterized in that, The inference node is a cloud node, and the large model inference system also includes edge nodes. Sending the intermediate data of the first inference unit to the first client includes any one of the following: The intermediate data of the first inference unit is sent to the first client via the edge node; or, The first client is controlled to obtain intermediate data of the first inference unit from the edge node.

17. The method according to any one of claims 13 to 15, characterized in that, The inference node is an edge node, and the method further includes: The large model is used to generate intermediate data for the first inference unit; or, In the case where the large model inference system also includes cloud nodes, intermediate data of the first inference unit is obtained from the cloud nodes.

18. The method according to any one of claims 13 to 15, characterized in that, The inference node is a terminal device, and the method further includes: generating intermediate data of the first inference unit using the large model; or, obtaining intermediate data of the first inference unit through a network node.

19. The method according to any one of claims 13 to 18, characterized in that, The method further includes: The storage system is retrieved based on the received fifth inference unit, the storage system being used to store intermediate data of the inference units generated using the large model; If the intermediate data of the fifth inference unit is not found in the storage system, a data transmission instruction is sent to the second client in the large model inference system. The data transmission instruction instructs the intermediate data of the fifth inference unit to be sent to the inference node. The second client stores the intermediate data of the fifth inference unit.

20. The method according to claim 19, characterized in that, If the inference node receives the fifth inference unit sent by multiple third clients in the large model inference system, the method further includes: The second client is determined from the plurality of third clients based on one or more of the following: network transmission status, data transmission latency, and the location of the third client.

21. The method according to claim 19 or 20, characterized in that, The method further includes: Receive intermediate data from the fifth inference unit sent by the second client; The intermediate data of the fifth inference unit is stored in the storage system.

22. A method for processing inference requests, characterized in that, The method, executed by a client of a large-model inference system, which also includes inference nodes, comprises: Send an inference request to the inference node; Receive intermediate data from the first inference unit in the inference unit corresponding to the inference request, wherein the inference unit corresponding to the inference request refers to the basic data unit when performing inference using a large model; Based on the intermediate data from the first inference unit, the inference request is executed using the large model.

23. The method according to claim 22, characterized in that, The reasoning unit corresponding to the reasoning request further includes a fourth reasoning unit, which is different from the first reasoning unit; the method further includes: The intermediate data for the fourth inference unit is generated using the large model. The step of executing the inference request using the large model based on the intermediate data of the first inference unit includes: executing the inference request using the large model based on the intermediate data of the first inference unit and the intermediate data of the fourth inference unit.

24. The method according to claim 22 or 23, characterized in that, The method further includes one or more of the following: Send the inference request corresponding to the inference node; The inference unit corresponding to the inference result of the inference request is sent to the inference node.

25. The method according to claim 23 or 24, characterized in that, The method further includes: Receive data transmission instructions sent by the inference node; The intermediate data of the inference unit indicated by the data transmission instruction is sent to the inference node.

26. A processing apparatus for reasoning requests, characterized in that, The apparatus is used to perform the processing method for the reasoning request as described in any one of claims 13 to 21.

27. A processing apparatus for reasoning requests, characterized in that, The apparatus is used to perform the processing method for the reasoning request as described in any one of claims 22 to 25.

28. A computing device, characterized in that, The computing device includes a processor and a memory, the processor being configured to execute instructions stored in the memory to enable the computing device to perform the functions of an inference node in a large model inference system as described in any one of claims 1 to 12.

29. A computing device cluster, characterized in that, It includes multiple computing devices, each of which includes a processor and memory; The processors of the plurality of computing devices are used to execute instructions stored in the memory of the plurality of computing devices, so that the cluster of computing devices can realize the function of the large model inference system as described in any one of claims 1 to 12.

30. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device cluster, the computing device cluster enables the large model inference system as described in any one of claims 1 to 12.

31. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a cluster of computing devices, enable the large model inference system as described in any one of claims 1 to 12.

Images