A data reasoning method, system, device and storage medium

Abstract

The application discloses a data reasoning method, system, device and storage medium. The method comprises the following steps: a master node generates a query vector, a key vector and a value vector based on input tokens of a current layer of a Transformer model, and adds the key vector and the value vector to a local key-value cache of the master node; the master node broadcasts the query vector to multiple slave nodes through a network device; the multiple nodes perform attention calculation based on the query vector and local key-value caches of the multiple nodes respectively, to obtain initial attention vectors corresponding to the multiple nodes respectively; the multiple slave nodes send the initial attention vectors to the master node through the network device; the network device aggregates the initial attention vectors corresponding to the multiple slave nodes respectively, to obtain a target attention vector; and the master node performs model reasoning based on the target attention vector through the Transformer model, to obtain output tokens. In this way, the reasoning time can be reduced, and the network bandwidth requirement can be reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a data reasoning method, system, device and storage medium. BACKGROUND

[0002] Large language model reasoning refers to a process of reasoning by using a trained large language model, in which a user inputs prompt information to the model, and the model outputs answer information corresponding to the prompt information. The input and output of the model are collectively referred to as context.

[0003] In actual application, as the context of the model grows, the computation amount of large language model reasoning increases linearly. Based on this, currently, Ring Attention (a ring attention mechanism using a distributed architecture) can be used to run the reasoning process of the large language model among multiple distributed nodes, so as to decompose the entire reasoning process in a distributed memory manner. However, in a long context reasoning scenario, the length of the text sequence that needs to be processed and understood by the model can reach an infinite length, and the above scheme still has defects, the reasoning task takes a long time, and the demand for network bandwidth is huge, resulting in poor reasoning effect for long context. SUMMARY

[0004] Embodiments of the present application provide a data reasoning method, system, device and storage medium to reduce the time taken by the reasoning task and reduce the demand for network bandwidth.

[0005] In a first aspect, embodiments of the present application provide a data reasoning method applied to a data reasoning system, wherein the data reasoning system uses a Transformer model as a reasoning model, the data reasoning system includes a network device and multiple nodes, the multiple nodes include a master node and multiple slave nodes, and the data reasoning method includes the following steps.

[0006] The master node generates a query vector, a key vector and a value vector based on the input token of the current layer of the Transformer model, and adds the key vector and the value vector to the local key-value cache of the master node.

[0007] The master node broadcasts the query vector to the multiple slave nodes through the network device.

[0008] The multiple nodes perform attention calculation based on the query vector and the local key-value cache of each node to obtain an initial attention vector corresponding to each node.

[0009] The multiple slave nodes send the initial attention vector of each node to the master node through the network device.

[0010] The network device aggregates the initial attention vectors corresponding to the plurality of slave nodes respectively to obtain a target attention vector.

[0011] The master node performs model inference based on the target attention vector through the Transformer model to obtain an output token of the Transformer model.

[0012] Optionally, the network device is a network card, and the plurality of nodes are in communication connection with the network card.

[0013] The master node broadcasts the query vector to the plurality of slave nodes through the network device, including:

[0014] The master node sends a first data packet to the plurality of slave nodes through the network card, and the first data packet includes the query vector, a layer identifier of the current layer, an inference task identifier, and an identifier of the attention cache of the master node.

[0015] The plurality of nodes perform attention calculation based on the query vector and local key-value caches of the plurality of nodes respectively to obtain initial attention vectors corresponding to the plurality of nodes respectively, including:

[0016] The master node performs attention calculation based on the query vector and a local key-value cache of the master node to obtain an initial attention vector corresponding to the master node, and writes the initial attention vector corresponding to the master node into the attention cache of the master node.

[0017] The plurality of slave nodes obtain the first data packet, and perform indexing based on the layer identifier of the current layer and the inference task identifier to determine local key-value caches of the plurality of slave nodes respectively, and perform attention calculation based on the query vector and the local key-value caches of the plurality of slave nodes respectively to obtain initial attention vectors corresponding to the plurality of slave nodes respectively.

[0018] Optionally, the plurality of slave nodes send the initial attention vectors to the master node through the network device, including:

[0019] The plurality of slave nodes respectively call the network card to send a second data packet to the master node, and the second data packets sent by the plurality of slave nodes respectively include the initial attention vectors of the plurality of slave nodes respectively, a scaling factor, and an identifier of the attention cache of the master node.

[0020] The network device aggregates the initial attention vectors corresponding to the plurality of slave nodes respectively to obtain a target attention vector, including:

[0021] In a case where the network card receives a second data packet sent by a first slave node of the plurality of slave nodes, the network card aggregates an initial attention vector of the first slave node and a locally stored attention vector based on a locally stored scaling factor and a scaling factor of the first slave node, to obtain an aggregated attention vector and an aggregated scaling factor, and locally stores the aggregated attention vector and the aggregated scaling factor;

[0022] In a case where the network card receives a second data packet sent by a second slave node of the plurality of slave nodes, the network card continues to aggregate an initial attention vector of the second slave node and the aggregated attention vector based on the aggregated scaling factor and a scaling factor of the second slave node, until the initial attention vectors corresponding to the plurality of slave nodes are all aggregated to obtain the target attention vector.

[0023] The network card writes the target attention vector into the attention cache of the master node based on an identifier of the attention cache of the master node.

[0024] Optionally, the network device includes a plurality of switches, and the plurality of nodes and the plurality of switches form a tree structure, the plurality of nodes respectively serving as leaf nodes of the tree structure, and the plurality of switches respectively serving as non-leaf nodes of the tree structure.

[0025] The master node broadcasts the query vector to the plurality of slave nodes through the network device, including:

[0026] The master node sends the query vector to a root node in the tree structure, and the master node is connected to the root node.

[0027] The root node sends the query vector to a first child node in the tree structure, the first child node is connected to the root node and is located at a next layer of the root node.

[0028] In a case where the first child node is a non-leaf node, the first child node continues to send the query vector to a second child node in the tree structure until the query vector is sent to leaf nodes other than the master node, the second child node is connected to the first child node and is located at a next layer of the first child node.

[0029] Optionally, for each slave node of the plurality of slave nodes, the plurality of slave nodes send respective initial attention vectors to the master node through the network device, including:

[0030] Each of the slave nodes sends a third data packet to a first parent node in the tree structure, each of the slave nodes is connected to a respective first parent node and is located at a next layer of the respective first parent node, and the third data packet sent by each of the slave nodes comprises an initial attention vector of the slave node and a scaling factor;

[0031] The network device aggregates the initial attention vectors of the slave nodes to obtain a target attention vector, including:

[0032] Upon receiving the third data packet from the corresponding slave node, the first parent node of each of the slave nodes aggregates the initial attention vector of the corresponding slave node and a locally stored attention vector based on the locally stored scaling factor and the scaling factor of the corresponding slave node to obtain an aggregated attention vector and an aggregated scaling factor, and locally stores the aggregated attention vector and the aggregated scaling factor;

[0033] In the absence of a root node in the first parent node of each of the slave nodes, the first parent node of each of the slave nodes sends the aggregated attention vector to a respective second parent node in the tree structure until the root node receives the aggregated intermediate attention vectors of the slave nodes, the second parent node of each of the first parent nodes is connected to the respective first parent node and is located at a previous layer of the respective first parent node;

[0034] The root node aggregates the aggregated attention vectors of the slave nodes to obtain the target attention vector.

[0035] Optionally, before the master node generates the query vector, the key vector and the value vector based on the input token of the current layer of the Transformer model, the method further comprises:

[0036] The master node encodes the input text to obtain an encoding result, and allocates the encoding result to the plurality of nodes;

[0037] Each of the plurality of nodes performs model inference based on the allocated encoding result through the Transformer model to obtain a query vector, a local key-value cache and a first attention vector of each node, and broadcasts the local key-value cache of each node to other nodes in the plurality of nodes;

[0038] Each of the nodes generates a second attention vector with other nodes based on the respective query vector and the local key-value cache broadcasted by the other nodes;

[0039] Each node aggregates the first attention vector and the second attention vector between the other nodes to obtain a third attention vector of each node;

[0040] Each node generates an output vector based on the third attention vector through the Transformer model;

[0041] The master node receives the output vectors respectively sent by the plurality of slave nodes, and generates an input token of the current layer based on the output vector of the master node and the output vectors respectively sent by the plurality of slave nodes.

[0042] Optionally, the data reasoning method further comprises:

[0043] In a case where the local key-value cache of the master node is occupied to the maximum capacity, a new node is allocated for the data reasoning system;

[0044] The local key-value cache of the master node is moved to the new node, or the master node is changed to a slave node, and the new node is taken as a new master node.

[0045] In a second aspect, the embodiments of the present application provide a data reasoning system, which adopts a Transformer model as a reasoning model, and comprises a network device and a plurality of nodes, wherein the plurality of nodes comprise a master node and a plurality of slave nodes;

[0046] The master node is configured to generate a query vector, a key vector and a value vector based on an input token of a current layer of the Transformer model, add the key vector and the value vector to a local key-value cache of the master node, and broadcast the query vector to the plurality of slave nodes through the network device;

[0047] The plurality of nodes are configured to perform attention calculation based on the query vector and respective local key-value caches of the plurality of nodes to obtain initial attention vectors respectively corresponding to the plurality of nodes;

[0048] The plurality of slave nodes are configured to send respective initial attention vectors to the master node through the network device;

[0049] The network device is configured to aggregate the initial attention vectors respectively corresponding to the plurality of slave nodes to obtain a target attention vector;

[0050] The master node is further configured to perform model reasoning based on the target attention vector through the Transformer model to obtain an output token of the Transformer model.

[0051] In a third aspect, an electronic device is provided, and the device includes a processor, a memory, and a system bus;

[0052] The processor and the memory are connected through the system bus;

[0053] The memory is configured to store a program, and the program includes instructions that, when executed by the processor, cause the processor to perform any of the implementation steps of the data inference method based on the weak measurement technology.

[0054] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores instructions, which, when executed on an electronic device, cause the electronic device to perform any of the implementation steps of the data inference method based on the weak measurement technology.

[0055] From the above technical solutions, the embodiments of the present application have the following advantages:

[0056] In the embodiments of the present application, the data inference system can use a Transformer model as an inference model, and accordingly, the data inference system includes a network device and multiple nodes, and the multiple nodes include a master node and multiple slave nodes. Based on this, the master node can first generate a query vector, a key vector, and a value vector based on the input tokens of the current layer of the Transformer model, and add the key vector and the value vector to the local key-value cache of the master node. Then, the master node can broadcast the query vector to the multiple slave nodes through the network device, whereby the multiple nodes can perform attention calculation based on the query vector and their respective local key-value caches to obtain initial attention vectors corresponding to the multiple nodes respectively. Then, the multiple slave nodes can send their respective initial attention vectors to the master node through the network device, so that the network device can aggregate the initial attention vectors corresponding to the multiple slave nodes respectively to obtain a target attention vector. Finally, the master node performs model inference based on the target attention vector through the Transformer model to obtain the output tokens of the Transformer model.

[0057] In this way, the master node can only broadcast the query vector, and since the query vector occupies less cache than the local key-value cache, this can help reduce the bandwidth demand of the network and reduce the time spent on the inference task. For example, in a long context inference scenario, this can significantly improve the inference efficiency of the long context and improve the inference effect. Moreover, aggregating the initial attention vectors corresponding to the multiple slave nodes using the network device can also make full use of the computing power of the network device, save the computing resources of the graphics processing units of the multiple nodes respectively, and thus further reduce the time spent on the inference task. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A structural schematic diagram of a data reasoning system provided for an embodiment of the present application;

[0059] Figure 2 A flowchart of a data reasoning method provided for an embodiment of the present application;

[0060] Figure 3 A schematic diagram of an overall framework of a data reasoning method provided for an embodiment of the present application;

[0061] Figure 4 A schematic diagram of an overall framework of another data reasoning method provided for an embodiment of the present application. DETAILED DESCRIPTION

[0062] As described above, the computation of large language model reasoning increases linearly with the growing context of the model. Based on this, Ring Attention (a ring-shaped attention mechanism using a distributed architecture) can be used to run the reasoning process of a large language model among multiple distributed nodes.

[0063] Specifically, for a reasoning task, Ring Attention can determine a fixed master node to perform complete transformer model reasoning, and each node saves its own local key-value cache in the form of a ring topology and transmits it in a counterclockwise or clockwise direction. Accordingly, the master node will continuously receive the local key-value cache from the neighbor nodes to iteratively calculate the attention vector.

[0064] However, since the master node of Ring Attention is fixed and the local key-value cache of each node is transmitted in the form of a ring topology, the entire reasoning task needs to be transmitted and calculated nxl times, where n represents the number of nodes and l represents the total number of layers of the model. In this way, a single reasoning task takes a long time, and the longer a single reasoning task takes, the longer it stays in the Graphics Processing Unit (GPU) memory, which greatly affects the number of concurrent tasks of the system and results in a large overall task delay.

[0065] Moreover, each node in Ring Attention not only needs to keep its own local key-value cache, but also needs to manually transmit the local key-value cache from other nodes in the ring topology, so it needs twice the memory space, resulting in a large memory overhead.

[0066] In addition, in the process of transmitting the local key-value cache between the nodes, it is equivalent to receiving and sending a complete local key-value cache for each node for each inference task, and therefore, the demand for network bandwidth is huge.

[0067] Therefore, in the Ring Attention scheme, although the entire inference process can be decomposed in a distributed memory manner, in a long context inference scenario, the length of the text sequence that the model needs to process and understand can reach an infinite length, and the above scheme still has defects, the inference task takes a long time, the memory overhead is large, and the demand for network bandwidth is huge, resulting in poor inference effect on long context.

[0068] Therefore, in order to solve the above problems, the embodiments of the present application provide a data inference method, and are applied to a data processing system. Wherein the data inference system can use a transformer model as an inference model, and accordingly, the data inference system includes a network device and a plurality of nodes, and the plurality of nodes include a master node and a plurality of slave nodes. Based on this, the master node can first generate a query vector, a key vector and a value vector based on the input token of the current layer of the transformer model, and add the key vector and the value vector to the local key-value cache of the master node. Then, the master node can broadcast the query vector to the plurality of slave nodes through the network device, so that the plurality of nodes can perform attention calculation based on the query vector and the local key-value cache of each node to obtain an initial attention vector corresponding to each node. Then, the plurality of slave nodes can send the initial attention vector to the master node through the network device, so that the network device can aggregate the initial attention vectors corresponding to the plurality of slave nodes to obtain a target attention vector. Finally, the master node performs model inference based on the target attention vector through the transformer model to obtain the output token of the transformer model.

[0069] In this way, the master node can only broadcast the query vector, and since the query vector occupies less cache than the local key-value cache, this can help to reduce the bandwidth demand of the network and reduce the time spent on the inference task. For example, in a long context inference scenario, this can significantly improve the inference efficiency of long context and improve the inference effect. Moreover, aggregating the initial attention vectors corresponding to the plurality of slave nodes using the network device can also make full use of the computing power of the network device, save the computing resources of the graphics processing unit of each node, and further reduce the time spent on the inference task.

[0070] In addition, compared with Ring Attention which needs n-1 calculation periods to obtain an attention vector, in the embodiment of the present application, the master node and the plurality of slave nodes can simultaneously perform the calculation of the attention vector, and the aggregation is performed by the network device, which has less impact on the performance of the GPU. Therefore, when the node scale n is large, the execution time of a single inference task can be further reduced, and a faster inference speed is achieved. Further, the faster inference speed also means that the local key-value cache stays in the GPU memory for a shorter time, and therefore, the data inference system can have more resources allocated to concurrent other inference tasks.

[0071] It should be noted that the embodiment of the present application can not limit the execution subject of the data inference method. For example, the data inference method of the embodiment of the present application can be applied to a terminal device or a server and the like data processing device. The terminal device can be a smart phone, a computer, a personal digital assistant (PDA), a tablet computer and the like electronic device. The server can be a standalone server, a cluster server or a cloud server.

[0072] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0073] Figure 1 A structural schematic diagram of a data inference system provided by the embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, the data inference system provided by the embodiment of the present application adopts a Transformer model as an inference model. Correspondingly, the data inference system includes a network device and a plurality of nodes, and the plurality of nodes include a master node and a plurality of slave nodes. Figure 1

[0074] Specifically, the data inference model can be divided into a prefill stage and a generation stage, and the two stages will be described exemplarily below.

[0075] ​For the prefill stage of large model inference, the model can generate query vector q, key vector k and value vector v according to the input text, and then calculate the first output token. Since the main purpose of the prefill stage is to calculate, using the transmission KV cache instead of Q can make the calculation more local, thereby ensuring efficiency.

[0076] Based on this, in the specific implementation, first, the master node can encode the input text, and then distribute the encoding result to different nodes. Next, when calculating each layer of the transformer model, each node in the multiple nodes can generate query vector q, key vector k and value vector v according to the input text, wherein the key vector k and the value vector v can constitute the local key-value cache KV cache of each node. At this time, each node can further combine the respective query vector q, key vector k and value vector v to perform attention calculation to obtain the first attention vector of each node. Then, all nodes can broadcast the local key-value cache KV cache to other nodes. When each node receives the local key-value cache KV cache sent by other nodes, it can combine its own query vector q to perform attention calculation to generate the second attention vector between each node and other nodes. Then, each node can aggregate the first attention vector and all second attention vectors by an aggregation operation to generate the third attention vector of each node. Finally, each node can continue to use the third attention vector to perform model inference to obtain the output vector of the Transformer model, and send the output vector to the master node to generate the first output token token.

[0077] For the generation stage of large model inference, the model can generate the next output token according to each new input token and the historical key-value cache KV cache.

[0078] In a specific implementation, first, the master node can obtain a query vector q, a key vector k and a value vector v from the input tokens of the current layer when performing computation for each layer of the transformer model. The query vector q is broadcast to multiple slave nodes, while the key vector k and the value vector v are added to the local key-value cache KV cache of the master node. Then, the multiple nodes can perform attention computation based on the query vector q and the local key-value cache KV cache of each node to obtain initial attention vectors corresponding to the multiple nodes respectively. Then, the multiple slave nodes among the multiple nodes can send the initial attention vectors to the master node through the network device, and the network device can aggregate the initial attention vectors corresponding to the multiple slave nodes respectively through online aggregation to obtain a target attention vector. Then, the master node can continue to use the target attention vector for model inference to obtain the next output token of the transformer model.

[0079] It should be noted that in the above process, if the local cache key-value KV cache of the master node is full, a new node can be allocated for the system, and the local cache key-value KV cache of the master node can be moved to the new node. Alternatively, the master node is changed to a slave node, and the new node is used as a new master node. Therefore, the master node is not fixed. In addition, for different inference tasks, the master node can also be different, and is not fixed.

[0080] In summary, in combination with the overall architecture of the data inference system described above, in the embodiment of the present application, in the generation phase of large model inference, the master node can only broadcast the query vector. Since the query vector occupies less cache than the local key-value cache, this can help to reduce the bandwidth demand of the network and reduce the time spent on the inference task. In addition, aggregating the initial attention vectors corresponding to the multiple slave nodes using the network device can also make full use of the computing power of the network device, save the computing resources of the graphics processing unit of each node, and further reduce the time spent on the inference task. In addition, compared with Ring Attention, which requires n-1 computation cycles to obtain an attention vector, the master node and the multiple slave nodes can simultaneously perform attention vector computation, and the aggregation by the network device has less impact on the performance of the GPU. Therefore, when the node scale n is large, the execution time of a single inference task can be further reduced, and faster inference speed can be achieved. Further, faster inference speed also means that the local key-value cache stays in the GPU memory for a shorter time, so the data inference system can allocate more resources to concurrent other inference tasks.

[0081] Based on the data reasoning system provided in the above embodiments, the embodiments of the present application can also provide a data reasoning method for the generation process described above. The data reasoning method will be described below in combination with embodiments and drawings.

[0082] Figure 2 A flowchart of a data reasoning method provided by the embodiments of the present application. The data reasoning method provided by the embodiments of the present application describes the overall implementation of the scheme based on the interaction process between each node in the data reasoning system and the interaction process between each node and the network device. Based on this, the data reasoning method includes Figure 2

[0083] S101: The master node generates a query vector, a key vector and a value vector based on the input token of the current layer of the Transformer model, and adds the key vector and the value vector to the local key-value cache of the master node.

[0084] In actual application, for the generation process of the query vector, the key vector and the value vector, the embodiments of the present application can not be specifically limited, and any algorithm for generating the query vector, the key vector and the value vector that exists or may exist in the future can be used to implement. For example, the master node can perform matrix multiplication operation on the input token of the current layer of the Transformer model and the parameter matrix W Q ,W K ,W V to obtain the query vector, the key vector and the value vector corresponding to the input token.

[0085] Further, when the master node adds the key vector and the value vector to the local key-value cache of the master node, if the local key-value cache of the master node is full, that is, in the case that the local key-value cache of the master node is occupied to the maximum capacity, the data reasoning system can be allocated a new node first, and the local key-value cache of the master node can be moved to the new node, or the master node can be changed to a slave node, and the new node can be used as a new master node.

[0086] S102: The master node broadcasts the query vector to a plurality of slave nodes through a network device.

[0087] In actual application, the network device can be a network card or a switch. The network card can be implemented by using a smart network interface card (Smart Network Interface Card, SmartNIC) or a data processing unit (Data Processing Unit, DPU) that supports online aggregation operation. In order to facilitate understanding, the above broadcast process will be exemplarily described below in combination with two different network devices, i.e., the network card and the switch.

[0088] ​As an example, in combination with Figure 3 As shown in the figure, the network device takes a network card as an example, and correspondingly, the plurality of nodes are in communication connection with the network card to realize end-to-end communication transmission.

[0089] Based on this, for step S102, in the specific implementation, the master node can send the first data packet to the plurality of slave nodes through the network card. In actual application, the first data packet includes a query vector, a layer identifier of a current layer, an inference task identifier, and an identifier of the attention cache of the master node.

[0090] The layer identifier of the current layer can uniquely represent the current layer of the Transformer model, and the inference task identifier can uniquely represent the currently executed inference task. Therefore, through the layer identifier of the current layer and the inference task identifier, the slave node can be indexed to the local key-value cache corresponding to the current layer and the currently executed inference task. Moreover, the identifier of the attention cache of the master node can uniquely represent the attention cache of the master node. Therefore, through the identifier of the attention cache of the master node, the slave node can quickly and accurately determine the address of the attention cache of the master node, facilitating writing back of the initial attention vector obtained through calculation.

[0091] In this way, while the master node broadcasts the query vector to the plurality of slave nodes, it can also send other information to the plurality of slave nodes, so that the plurality of slave nodes can not only calculate the initial attention of each slave node based on the query vector, but also index to the local key-value cache of each slave node through the layer identifier and the inference task identifier, and write back the initial attention vector obtained through calculation to the attention cache of the master node.

[0092] In addition, in the embodiment of the present application, in order to improve the model inference effect, the offset information between input features can also be considered when performing attention calculation, so as to capture and understand the features and the relationship between the features. Based on this, in the embodiment of the present application, the first data packet can also include offset information corresponding to the payload. The payload refers to the data information contained in the first data packet. In this way, in the subsequent attention calculation, the offset corresponding to the payload can be further considered. By considering the offset between features, the model can more accurately identify and interpret the data, thereby improving the inference performance of the model.

[0093] As another example, in combination with Figure 4 As shown in the figure, the network device takes a switch supporting online aggregation operation as an example. Since the switch is naturally located at the convergence position of the network, it is more convenient for aggregation processing. Correspondingly, the number of switches can be a plurality, and the plurality of nodes are interconnected through the forwarding of the plurality of switches.

[0094] Further, a tree structure can be constructed based on the plurality of nodes and the plurality of switches, i.e., the plurality of nodes and the plurality of switches form the tree structure. In actual applications, the tree structure can include leaf nodes and non-leaf nodes, and correspondingly, the plurality of nodes are respectively leaf nodes of the tree structure, and the plurality of switches are respectively non-leaf nodes of the tree structure. The non-leaf nodes further include a root node of the tree structure, and intermediate nodes between the root node and the leaf nodes.

[0095] In the tree structure, two layers of nodes connected to each other are parent and child nodes, i.e., a node in an upper layer is a parent node of a node in a lower layer connected to the node in the upper layer, and the node in the lower layer is a child node of the node in the upper layer connected to the node in the lower layer. Correspondingly, the plurality of nodes in the data reasoning system can maintain respective parent node information and child node information.

[0096] Based on this, for step S102, in specific implementation, first, the master node can send a query vector to a root node in the tree structure, wherein the master node is connected to the root node. That is, the master node unicasts the query vector to the switch serving as the root node. In actual applications, the master node can also unicast other data in the first data packet, such as the identifiers of the nodes in the tree structure and the identifier of the attention cache of the master node, to the root node.

[0097] Then, the root node can send the query vector to a first child node in the tree structure. The first child node is connected to the root node and is in a next layer of the root node. Here, the root node can query its child node, i.e., the first child node, according to the identifiers of the nodes in the tree structure sent by the master node, and send the query vector to the first child node.

[0098] In this way, in the case that the first child node is a leaf node and is not the master node, the first leaf node can represent a slave node in the data reasoning system, and therefore, the query vector has been sent to the slave node, and the entire sending process can be ended.

[0099] In the case that the first child node is a non-leaf node, the first child node can continue to send the query vector to a second child node in the tree structure until the query vector is sent to a leaf node other than the master node, the second child node being connected to the first child node and being in a next layer of the first child node. That is, when the first child node is a non-leaf node but an intermediate node, the switch serving as the intermediate node can continue to send the query vector to a node in the next layer, and so on until the query vector is sent to the other leaf node, i.e., the plurality of slave nodes, and the entire sending process can be ended.

[0100] S103: The plurality of nodes perform attention calculation based on the query vector and respective local key-value caches of the plurality of nodes to obtain initial attention vectors respectively corresponding to the plurality of nodes.

[0101] In the embodiment of the present application, for step S103, in the specific implementation, first, the master node can perform attention calculation based on the query vector and the local key-value cache of the master node to obtain an initial attention vector corresponding to the master node, and write the initial attention vector corresponding to the master node into the attention cache of the master node.

[0102] Here, after the master node completes the calculation, the data reasoning system can first put the current reasoning task into a waiting queue, switch to other reasoning tasks for scheduling, to prevent the GPU of the master node from idling.

[0103] Next, the plurality of slave nodes can obtain the first data packet sent by the master node through the network card, and index based on the layer identifier of the current layer and the reasoning task identifier to determine the local key-value cache of each of the plurality of slave nodes. In actual application, the plurality of slave nodes can index based on the reasoning task identifier and the layer identifier of the current layer to determine the local key-value cache required by each of the plurality of slave nodes.

[0104] Then, the plurality of slave nodes can perform attention calculation based on the query vector and the local key-value cache of each of the plurality of slave nodes to obtain an initial attention vector corresponding to each of the plurality of slave nodes.

[0105] In actual application, after receiving the first data packet, the plurality of slave nodes can each generate a task with the highest priority to quickly start the attention calculation process. Alternatively, the plurality of slave nodes can generate a scheduling operation supporting task terminal and preemption to reduce the waiting time of the attention calculation process.

[0106] In addition, it should be noted that the process of attention calculation can be implemented by using any existing or future attention algorithm, and the embodiment of the present application does not specifically limit this.

[0107] S104: The plurality of slave nodes sends the initial attention vector of each to the master node through the network device.

[0108] As mentioned above, the network device can be a network card or a switch. Based on this, in order to facilitate understanding, the implementation process of step S104 will be described below in combination with two different network devices.

[0109] As an example, in combination with Figure 3As shown, the network device takes a network card as an example, and the step S104 can include that the plurality of slave nodes respectively invoke the network card to send the second data packet to the master node, wherein the second data packets sent by the plurality of slave nodes respectively include the initial attention vector of each of the plurality of slave nodes, the scaling factor and the identifier of the attention cache of the master node. In this way, the network card can perform attention aggregation on each initial attention vector based on the second data packets sent by the plurality of slave nodes respectively.

[0110] In addition, the second data packet sent by each slave node can further include a slave node bitmap. The slave node bitmap refers to a data structure for indicating the state of the slave node by using bits. For example, taking 32 slave nodes as an example, a 32-bit bitmap can be used as the slave node bitmap, and each bit (1 bit) in the slave node bitmap corresponds to a node, for example, the 0th bit indicates the state of node 1, the 1st bit indicates the state of node 2, and so on, and the 31st bit indicates the state of node 32. Accordingly, when the value of a certain bit is 0, it can indicate that the initial attention vector corresponding to the slave node has not been aggregated, and when the value is 1, it can indicate that the initial attention vector corresponding to the slave node has been aggregated. Therefore, when the values of all bits in the slave node bitmap are 1, it can indicate that the entire aggregation process is completed.

[0111] As another example, in combination with Figure 4 As shown, the network device takes a switch as an example, and for each slave node in the plurality of slave nodes, the step S104 can include that each slave node sends a third data packet to a first parent node in a tree structure, wherein each slave node is connected to a respective first parent node and is located at a next layer of the respective first parent node, and the third data packet sent by each slave node includes the initial attention vector and the scaling factor of each slave node. Here, since the plurality of slave nodes are all leaf nodes in the tree structure, each slave node has a respective first parent node, and the first parent node of each slave node can be consistent, and then each slave node uploads the initial attention vector to the switch as the first parent node, so that the switch as the first parent node aggregates the received initial attention vector.

[0112] In addition, the third data packet sent by each slave node also includes a slave node bitmap. The slave node bitmap refers to a data structure for indicating the state of the slave node by using bits. For example, taking 32 slave nodes as an example, a 32-bit bitmap can be used as the slave node bitmap, and each bit (1 bit) in the slave node bitmap corresponds to a node, for example, the 0th bit indicates the state of node 1, the 1st bit indicates the state of node 2, and so on, and the 31st bit indicates the state of node 32. Accordingly, when the value of a certain bit is 0, it can indicate that the initial attention vector corresponding to the slave node has not been aggregated, and when the value is 1, it can indicate that the initial attention vector corresponding to the slave node has been aggregated. Therefore, when the values of all bits in the slave node bitmap are 1, it can indicate that the entire aggregation process is completed.

[0113] It should be noted that the second data packet and the third data packet described above are both data packets sent by the slave node, and the data contents included in the two data packets are the same. In order to facilitate understanding, the second data packet is used to describe the data packet sent by the slave node to the master node in the scenario where the network device is a network card, and the third data packet is used to describe the data packet sent by the slave node to the corresponding parent node in the tree structure in the scenario where the network device is a switch.

[0114] S105: The network device aggregates the initial attention vectors corresponding to the plurality of slave nodes respectively to obtain a target attention vector.

[0115] As a possible implementation, in combination with Figure 3 As shown in FIG. 10, based on the implementation process of step S104 for the network card described above, when the network card receives the second data packet sent by the first slave node in the plurality of slave nodes, the network card aggregates the initial attention vector of the first slave node and the locally stored attention vector based on the locally stored scaling factor and the scaling factor of the first slave node, to obtain an aggregated attention vector and an aggregated scaling factor, and locally stores the aggregated attention vector and the aggregated scaling factor.

[0116] For example, for the aggregation process, a SOFTMAX function is used to implement, and the log value of the denominator of the SOFTMAX function is based on 2. Based on this, the above aggregation process can be shown in the following formula (1):

[0117]

[0118] Where a0 is the aggregated attention vector, a is the locally stored attention vector of the network card, l is the locally stored scaling factor of the network card, l0 is the aggregated scaling factor, a ′ is the initial attention vector of the first slave node, l ′a scaling factor of the first slave node.

[0119] It should be noted that the above formula (1) specifically describes the process of one aggregation processing, and the subsequent aggregation processing can be implemented by using the above formula (1), which will not be described below.

[0120] In addition, in actual application, when the network card performs the above aggregation operation, a software lock or a hardware lock can be used to ensure the atomicity of the operation, so as to avoid errors when the initial attention vectors of multiple slave nodes are aggregated at the same time.

[0121] Then, in the case that the network card receives a second data packet sent by a second slave node in the multiple slave nodes, the network card continues to aggregate the initial attention vector of the second slave node and the aggregated attention vector based on the aggregated scaling factor and the scaling factor of the second slave node, until the initial attention vectors corresponding to the multiple slave nodes are all aggregated to obtain target attention vectors. That is, the network card can start the aggregation process after receiving the second data packet corresponding to one slave node, that is, the network card can directly aggregate the latest data currently saved locally (i.e., the aggregated scaling factor and the aggregated attention vector) and the data in the received second data packet (i.e., the scaling factor of the second slave node and the initial attention vector), so that the network card can realize real-time aggregation of two-by-two, without waiting for all initial attention vectors of the slave nodes to be received for one-time aggregation. Therefore, the calculation resources of the GPU can be effectively saved, and the calculation efficiency can be improved.

[0122] The aggregation process of the initial attention vectors of the second slave node and other slave nodes can refer to the above formula (1), which will not be described here.

[0123] Finally, the network card can write the target attention vector into the attention cache of the master node based on the identifier of the attention cache of the master node. In this way, the master node can continue to use the target attention vector for model inference to obtain new output tokens. Moreover, the entire process only needs to maintain the attention cache of the master node, without the need to maintain the attention caches of all slave nodes, so that the demand for memory capacity can be reduced, and the memory overhead can be reduced.

[0124] In addition, as mentioned earlier, the second data packet also includes a slave node bitmap. Based on this, taking a certain slave node as an example, during the aggregation process performed by the network card, the slave node bitmap can be used for aggregation, and the slave node bitmap can be updated accordingly after the aggregation is completed, that is, the node state is updated. In this way, during the aggregation process, according to the node state indicated by the slave node bitmap, repeated aggregation can be avoided, and after the aggregation is completed, the slave node bitmap is further updated, so that subsequent aggregation is also avoided, thereby improving the accuracy of aggregation.

[0125] As another possible implementation, in combination with Figure 4 As shown, based on the implementation process of the above step S104 for the switch, when the above step S105 is implemented, first, in the case that each slave node's respective first parent node receives the third data packet sent by the corresponding slave node, the initial attention vector of the corresponding slave node and the locally stored attention vector are aggregated based on the locally stored scaling factor and the scaling factor of the corresponding slave node, to obtain the aggregated attention vector and the aggregated scaling factor, and the aggregated attention vector and the aggregated scaling factor are locally stored. That is, after the switch connected to each slave node receives the third data packet sent by the corresponding slave node, the aggregation operation can be started, which can still be implemented by referring to the above formula (1), and will not be described here.

[0126] In addition, as mentioned earlier, the slave node bitmap is also included in the third data packet. Based on this, taking a certain slave node as an example, during the aggregation process performed by the parent node, i.e., the switch, the aggregation can be performed according to the slave node bitmap, and the slave node bitmap is updated accordingly after the aggregation is completed, that is, the node state is updated. In this way, during the aggregation process, according to the node state indicated by the slave node bitmap, repeated aggregation can be avoided, and further updating the slave node bitmap after the aggregation is completed can also avoid subsequent repeated aggregation, thereby improving the accuracy of the aggregation.

[0127] Then, in the case that the root node does not exist in each slave node's respective first parent node, each slave node's respective first parent node sends the aggregated attention vector to each respective second parent node in the tree structure, until the root node receives the multiple slave nodes' respective aggregated intermediate attention vectors. Among them, the first parent node's respective second parent node is connected to the respective first parent node and is located at the upper layer of the respective first parent node.

[0128] That is, after each slave node sends the third data packet to the respective first parent node, if the root node does not exist in the first parent node, each first parent node can further construct data upwards, that is, the first parent node sends the aggregated attention vector upwards layer by layer according to the level of the tree structure, until the root node performs the final aggregation process.

[0129] Then, the root node aggregates the multiple slave nodes' respective aggregated attention vectors to obtain the target attention vector. In this way, the root node can send the target attention vector to the master node to write into the master node's attention cache, so as to arouse the master node to perform the corresponding reasoning task.

[0130] S106: The main node performs model inference based on the target attention vector by a Transformer model to obtain an output token of the Transformer model.

[0131] In actual applications, the network device, i.e., the network card and the switch as the root node, can generate a scheduling event indicating that the aggregation is completed after the aggregation is completed, so that the data inference system can respond to the scheduling event to schedule the GPU of the main node to perform subsequent model inference based on the target attention vector.

[0132] Specifically, after the main node obtains the target attention vector, the residual connection and layer normalization (Add&Norm) stage and the feedforward network (FFN) stage in the Transformer model described above can be continued to be executed. Specifically, first, the target attention vector can be processed by Add&Norm to obtain a processing result, and then the processing result is nonlinearly transformed by FFN to obtain a transformed result. Finally, the transformed result can be processed again by Add&Norm to obtain a processing result, which can be used as the output of the current layer of the Transformer model. In actual applications, the Transformer model adopts a multi-layer structure, and therefore, for each layer of the Transformer model, the above process can be performed to obtain the output corresponding to each layer. Correspondingly, the output of the last layer of the Transformer model is the output token described above.

[0133] Based on the related content of the above steps S101-S106, in the embodiment of the present application, the data reasoning system can use a Transformer model as the reasoning model. Correspondingly, the data reasoning system includes a network device and a plurality of nodes, and the plurality of nodes include a master node and a plurality of slave nodes. Based on this, the master node can first generate a query vector, a key vector and a value vector based on the input tokens of the current layer of the Transformer model, and add the key vector and the value vector to the local key-value cache of the master node. Then, the master node can broadcast the query vector to the plurality of slave nodes through the network device. In this way, the plurality of nodes can perform attention calculation based on the query vector and the local key-value cache of each node to obtain an initial attention vector corresponding to each node. Then, the plurality of slave nodes can send the initial attention vector corresponding to each node to the master node through the network device, so that the network device can aggregate the initial attention vectors corresponding to each node to obtain a target attention vector. Finally, the master node performs model reasoning based on the target attention vector through the Transformer model to obtain the output tokens of the Transformer model. In this way, the master node can only broadcast the query vector, and since the query vector occupies less cache than the local key-value cache, this can help reduce the bandwidth demand of the network and reduce the time spent on the reasoning task. Moreover, aggregating the initial attention vectors corresponding to each node by using the network device can also make full use of the computing power of the network device, save the computing resources of the graphics processing unit of each node, and further reduce the time spent on the reasoning task.

[0134] In addition, as mentioned earlier, the data reasoning model can be divided into a prefill stage and a generation stage. Therefore, before step S101 is performed, the prefill stage needs to be performed first. Based on this, before step S101, the following steps 1-6 can be performed first to complete the prefill stage, which will be described in detail below.

[0135] Step 1: The master node encodes the input text to obtain an encoding result, and allocates the encoding result to the plurality of nodes.

[0136] In specific implementation, first, the data reasoning system can first allocate s / s b nodes according to the number s of tokens corresponding to the input text and the maximum number s of tokens that can be stored in the local key-value cache of each node b , and set a master node in these nodes, and the other nodes as slave nodes.

[0137] Then, the prepared master node can perform word vector and position encoding on the input text to obtain an encoding result. Then, the master node distributes the encoding result to each node according to the length of the input text, and each node can be allocated s b tokens.

[0138] Step 2: Each node in the plurality of nodes performs model inference based on the allocated encoding result through the Transformer model to obtain a query vector, a local key-value cache and a first attention vector of each node, and broadcasts the local key-value cache of each node to other nodes in the plurality of nodes.

[0139] Specifically, each node can perform matrix multiplication on the encoding result and the parameter matrix W Q ,W K ,W V to obtain a query vector, a key vector and a value vector of each node, and save the key vector and the value vector as a local key-value cache. Then, each node can perform attention calculation based on the query vector and the local key-value cache to obtain a first attention vector.

[0140] Step 3: Each node generates a second attention vector between the other nodes based on the respective query vector and the key-value cache broadcasted by the other nodes.

[0141] Here, the process of each node generating a second attention vector between the other nodes can refer to the implementation process of step S103 in the above embodiment, which will not be described here.

[0142] Step 4: Each node aggregates the first attention vector and the second attention vector between the other nodes to obtain a third attention vector of each node.

[0143] Here, the calculation process of the third attention vector of each node can refer to the implementation process of step S105 in the above embodiment, which will not be described here.

[0144] Step 5: Each node generates an output vector based on the third attention vector through the Transformer model.

[0145] Specifically, after each node obtains the third attention vector, the node can continue to perform the residual connection and layer normalization (Add & Norm) stage and the feed-forward network (FFN) stage in the above-described Transformer model. Specifically, first, the third attention vector can be processed by Add & Norm to obtain a processing result, and then the processing result is nonlinearly transformed by FFN to obtain a transformation result. Finally, the transformation result can be processed again by Add & Norm to obtain a processing result, which can be used as the output of the current layer of the Transformer model. In actual applications, the Transformer model adopts a multi-layer structure, and therefore, for each layer of the Transformer model, the above-described process can be performed to obtain the output of each layer. Correspondingly, the output of the last layer of the Transformer model can be used as the output vector.

[0146] Step 6: The master node receives the output vectors respectively sent by the plurality of slave nodes, and generates an input token of a current layer based on the output vector of the master node and the output vectors respectively sent by the plurality of slave nodes.

[0147] Here, the master node can aggregate the output vectors respectively sent by the plurality of slave nodes to obtain the input token of the current layer. For the implementation process of aggregation, the embodiments of the present application can not be specifically limited, and any method for aggregating the output vectors that exists or may exist in the future can be used for implementation.

[0148] Further, based on the data reasoning method provided in the above-described embodiments, the embodiments of the present application can also provide a data reasoning system. The data reasoning system is described below in combination with embodiments. The data reasoning system provided by the embodiments of the present application adopts a Transformer model as a reasoning model, and the data reasoning system includes a network device and a plurality of nodes, and the plurality of nodes include a master node and a plurality of slave nodes.

[0149] The master node is configured to generate a query vector, a key vector and a value vector based on an input token of a current layer of the Transformer model, add the key vector and the value vector to a local key-value cache of the master node, and broadcast the query vector to the plurality of slave nodes through the network device.

[0150] The plurality of nodes are configured to perform attention calculation based on the query vector and a local key-value cache of each node to obtain an initial attention vector corresponding to each node.

[0151] The plurality of slave nodes are configured to send the initial attention vector of each node to the master node through the network device.

[0152] aggregate the initial attention vectors corresponding to the plurality of slave nodes respectively to obtain a target attention vector;

[0153] The master node is further configured to perform model inference based on the target attention vector by using the Transformer model to obtain an output token of the Transformer model.

[0154] As an implementation form, the network device is a network card, and the plurality of nodes are in communication connection with the network card.

[0155] The master node is specifically configured to send a first data packet to the plurality of slave nodes through the network card, and the first data packet includes the query vector, the layer identifier of the current layer, an inference task identifier, and an identifier of the attention cache of the master node.

[0156] The master node is specifically configured to perform attention calculation based on the query vector and a local key-value cache of the master node to obtain an initial attention vector corresponding to the master node, and write the initial attention vector corresponding to the master node into the attention cache of the master node.

[0157] The plurality of slave nodes are specifically configured to acquire the first data packet, and perform indexing based on the layer identifier of the current layer and the inference task identifier to determine local key-value caches of the plurality of slave nodes, and perform attention calculation based on the query vector and the local key-value caches of the plurality of slave nodes to obtain initial attention vectors corresponding to the plurality of slave nodes respectively.

[0158] As an implementation form, the plurality of slave nodes are specifically configured to respectively call the network card to send a second data packet to the master node, and the second data packets sent by the plurality of slave nodes respectively include the initial attention vectors of the plurality of slave nodes respectively, a scaling factor, and an identifier of the attention cache of the master node.

[0159] The network card is specifically configured to:

[0160] In a case where the second data packet sent by a first slave node in the plurality of slave nodes is received, the initial attention vector of the first slave node and the locally stored attention vector are aggregated based on the locally stored scaling factor and the scaling factor of the first slave node to obtain an aggregated attention vector and an aggregated scaling factor, and the aggregated attention vector and the aggregated scaling factor are locally stored.

[0161] In a case where a second data packet sent by a second slave node in the plurality of slave nodes is received, the initial attention vector of the second slave node and the aggregated attention vector are aggregated based on the aggregated scaling factor and a scaling factor of the second slave node until the initial attention vectors corresponding to the plurality of slave nodes are all aggregated to obtain the target attention vector;

[0162] The target attention vector is written into the attention cache of the master node based on the identifier of the attention cache of the master node.

[0163] As an implementation form, the network device includes a plurality of switches, the plurality of nodes and the plurality of switches form a tree structure, the plurality of nodes are respectively leaf nodes of the tree structure, and the plurality of switches are respectively non-leaf nodes of the tree structure.

[0164] The master node is specifically configured to send the query vector to a root node in the tree structure, and the master node is connected to the root node.

[0165] The root node is configured to send the query vector to a first child node in the tree structure, the first child node is connected to the root node and is located at a next layer of the root node.

[0166] The first child node is configured to, in a case where the first child node is a non-leaf node, continue to send the query vector to a second child node in the tree structure until the query vector is sent to leaf nodes other than the master node in the leaf nodes, the second child node is connected to the first child node and is located at a next layer of the first child node.

[0167] As an implementation form, for each slave node in the plurality of slave nodes, the slave node is specifically configured to send a third data packet to a first parent node in the tree structure, the slave node is connected to the respective first parent node and is located at a next layer of the respective first parent node, and the third data packet sent by the slave node includes an initial attention vector of the slave node and a scaling factor.

[0168] The first parent node of each slave node is configured to, in a case where the third data packet sent by the corresponding slave node is received, aggregate the initial attention vector of the corresponding slave node and a locally stored attention vector based on a locally stored scaling factor and a scaling factor of the corresponding slave node to obtain an aggregated attention vector and an aggregated scaling factor, and locally store the aggregated attention vector and the aggregated scaling factor.

[0169] each slave node is configured to, in the case that there is no root node in a respective first parent node of each slave node, send the aggregated attention vector to a respective second parent node of the first parent node in the tree structure until the root node receives the aggregated intermediate attention vectors corresponding to the plurality of slave nodes, the second parent node of the first parent node being connected to the first parent node and being located at a layer above the first parent node;

[0170] the root node is configured to aggregate the aggregated attention vectors corresponding to the plurality of slave nodes to obtain the target attention vector.

[0171] As an implementation form, the master node is further configured to encode the input text to obtain an encoding result, and allocate the encoding result to the plurality of nodes.

[0172] Each node of the plurality of nodes is further configured to perform model inference based on the allocated encoding result by using the Transformer model to obtain a query vector, a local key-value cache and a first attention vector of each node, and broadcast the local key-value cache of each node to other nodes of the plurality of nodes.

[0173] Each node is further configured to generate a second attention vector between other nodes based on the respective query vector and the local key-value cache broadcasted by the other nodes.

[0174] Each node is further configured to aggregate the first attention vector and the second attention vector between other nodes to obtain a third attention vector of each node.

[0175] Each node is further configured to generate an output vector based on the third attention vector by using the Transformer model.

[0176] The master node is further configured to receive the output vectors respectively sent by the plurality of slave nodes, and generate the input token of the current layer based on the output vector of the master node and the output vectors respectively sent by the plurality of slave nodes.

[0177] As an implementation form, the data inference system is further configured to:

[0178] In the case that the local key-value cache of the master node is occupied to the maximum capacity, a new node is allocated to the data inference system.

[0179] The local key-value cache of the master node is moved to the new node, or the master node is changed to a slave node, and the new node is taken as a new master node.

[0180] Further, the embodiment of the present application further provides an electronic device, comprising: a processor, a memory, a system bus;

[0181] The processor and the memory are connected through the system bus;

[0182] The memory is used for storing one or more programs, the one or more programs comprising instructions which, when executed by the processor, cause the processor to perform any implementation step of the above-mentioned data reasoning method.

[0183] Further, the embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium storing instructions, when the instructions run on the electronic device, causing any implementation step of the above-mentioned data reasoning method.

[0184] From the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps of the method in the above embodiments can be implemented by means of software plus necessary universal hardware platforms. Based on such an understanding, the technical solutions of the present application can be embodied in the form of software products, which can be stored in a storage medium, such as ROM / RAM, magnetic disks, optical disks, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) execute the methods described in the various embodiments or some parts of the embodiments. It should be noted that the various embodiments in the present specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other.

[0185] For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts are referred to the method part.

[0186] It should also be noted that, in this document, the relationship terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0187] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and, while certain embodiments according to the principles set forth herein are shown and described, it is to be understood that the same are not limiting of the scope of the application as it is set forth in the appended claims, and that various modifications are made within the scope of the appended claims. Therefore, it is contemplated to cover the application in its broadest scope, including all features that can be made or used in the embodiments described herein.

Claims

1. A data inference method, characterized by, The application is applied to a data reasoning system adopting a Transformer model as a reasoning model, the data reasoning system comprises a network device and a plurality of nodes, the plurality of nodes comprise a master node and a plurality of slave nodes, and the data reasoning method comprises: The master node generates a query vector, a key vector and a value vector based on an input token of a current layer of the Transformer model, and adds the key vector and the value vector to a local key-value cache of the master node; The master node broadcasts the query vector to the plurality of slave nodes through the network device; The plurality of nodes perform attention calculation based on the query vector and respective local key-value caches of the plurality of nodes to obtain initial attention vectors corresponding to the plurality of nodes respectively; The plurality of slave nodes send respective initial attention vectors to the master node through the network device; The network device aggregates the initial attention vectors corresponding to the plurality of slave nodes respectively to obtain a target attention vector; The master node performs model reasoning based on the target attention vector through the Transformer model to obtain an output token of the Transformer model.

2. The data reasoning method of claim 1, wherein, The network device is a network card, and the plurality of nodes are in communication connection with the network card; The master node broadcasts the query vector to the plurality of slave nodes through the network device, comprising: The master node sends a first data packet to the plurality of slave nodes through the network card, wherein the first data packet comprises the query vector, a layer identifier of the current layer, a reasoning task identifier and an identifier of an attention cache of the master node; The plurality of nodes perform attention calculation based on the query vector and respective local key-value caches of the plurality of nodes to obtain initial attention vectors corresponding to the plurality of nodes respectively, comprising: The master node performs attention calculation based on the query vector and the local key-value cache of the master node to obtain an initial attention vector corresponding to the master node, and writes the initial attention vector corresponding to the master node into the attention cache of the master node; The plurality of slave nodes acquire the first data packet, and perform indexing based on the layer identifier of the current layer and the reasoning task identifier to determine respective local key-value caches of the plurality of slave nodes, and perform attention calculation based on the query vector and the respective local key-value caches of the plurality of slave nodes to obtain initial attention vectors corresponding to the plurality of slave nodes respectively.

3. The data reasoning method of claim 2, wherein, The plurality of slave nodes send respective initial attention vectors to the master node through the network device, comprising: The plurality of slave nodes respectively call the network card to send a second data packet to the master node, wherein the second data packet sent by each of the plurality of slave nodes comprises an initial attention vector of the slave node, a scaling factor and an identifier of the attention cache of the master node; The network device aggregates the initial attention vectors corresponding to the plurality of slave nodes respectively to obtain a target attention vector, comprising: The network card aggregates the initial attention vector of the first slave node and the locally stored attention vector based on the locally stored scaling factor and the scaling factor of the first slave node to obtain an aggregated attention vector and an aggregated scaling factor, and locally stores the aggregated attention vector and the aggregated scaling factor, in the case that the network card receives a second data packet sent by the first slave node in the plurality of slave nodes; The network card continues to aggregate the initial attention vector of the second slave node and the aggregated attention vector based on the aggregated scaling factor and the scaling factor of the second slave node until the initial attention vectors corresponding to the plurality of slave nodes are all aggregated to obtain the target attention vector, in the case that the network card receives a second data packet sent by the second slave node in the plurality of slave nodes; The network card writes the target attention vector into the attention cache of the master node based on the identifier of the attention cache of the master node.

4. The data reasoning method of claim 1, wherein, The network device comprises a plurality of switches, the plurality of nodes and the plurality of switches form a tree structure, the plurality of nodes are respectively leaf nodes of the tree structure, and the plurality of switches are respectively non-leaf nodes of the tree structure; The master node broadcasts the query vector to the plurality of slave nodes through the network device, comprising: The master node sends the query vector to a root node in the tree structure, and the master node is connected to the root node; The root node sends the query vector to a first child node in the tree structure, the first child node is connected to the root node and is located at the next layer of the root node; In the case that the first child node is a non-leaf node, the first child node continues to send the query vector to a second child node in the tree structure until the query vector is sent to the leaf nodes other than the master node, the second child node is connected to the first child node and is located at the next layer of the first child node.

5. The data reasoning method of claim 4, wherein, For each slave node in the plurality of slave nodes, the plurality of slave nodes send respective initial attention vectors to the master node through the network device, comprising: Each slave node sends a third data packet to a first parent node in the tree structure, each slave node is connected to a respective first parent node and is located at the next layer of the respective first parent node, and the third data packet sent by each slave node comprises the initial attention vector and the scaling factor of each slave node; The network device aggregates the initial attention vectors corresponding to the plurality of slave nodes to obtain a target attention vector, comprising: The first parent node of each slave node aggregates the initial attention vector of the corresponding slave node and the locally stored attention vector based on the locally stored scaling factor and the scaling factor of the corresponding slave node to obtain an aggregated attention vector and an aggregated scaling factor, and locally stores the aggregated attention vector and the aggregated scaling factor, in the case that the first parent node of each slave node receives a third data packet sent by the corresponding slave node. In the case that the first parent node of each slave node does not exist the root node, the first parent node of each slave node sends the aggregated attention vector to the second parent node of each slave node, until the root node receives the aggregated intermediate attention vectors corresponding to the plurality of slave nodes, the second parent node of each first parent node is connected with the corresponding first parent node and is located at the upper layer of the corresponding first parent node. The root node aggregates the aggregated attention vectors corresponding to the plurality of slave nodes to obtain the target attention vector.

6. The data reasoning method according to any one of claims 1 to 5, characterized in that, Before the master node generates the query vector, the key vector and the value vector based on the input token of the current layer of the Transformer model, the method further comprises: The master node encodes the input text to obtain an encoding result, and allocates the encoding result to the plurality of nodes; Each node of the plurality of nodes performs model inference based on the allocated encoding result through the Transformer model to obtain the query vector, the local key-value cache and the first attention vector of each node, and broadcasts the local key-value cache of each node to other nodes of the plurality of nodes; Each node generates the second attention vector between other nodes based on the respective query vector and the local key-value cache broadcasted by other nodes; Each node aggregates the first attention vector and the second attention vector between other nodes to obtain the third attention vector of each node; Each node generates an output vector based on the third attention vector through the Transformer model; The master node receives the output vectors sent by the plurality of slave nodes respectively, and generates the input token of the current layer based on the output vector of the master node and the output vectors sent by the plurality of slave nodes respectively.

7. The data reasoning method according to any one of claims 1 to 5, characterized in that, The data inference method further comprises: In the case that the local key-value cache of the master node is occupied to the maximum capacity, a new node is allocated to the data inference system; The local key-value cache of the master node is moved to the new node, or the master node is changed to a slave node, and the new node is used as a new master node.

8. A data inference system, comprising: The data inference system adopts a Transformer model as an inference model, the data inference system comprises a network device and a plurality of nodes, the plurality of nodes comprise a master node and a plurality of slave nodes; The master node is configured to generate a query vector, a key vector and a value vector based on an input token of a current layer of the Transformer model, and add the key vector and the value vector to a local key-value cache of the master node; The network device is configured to broadcast the query vector to the plurality of slave nodes; The plurality of nodes are configured to perform attention calculation based on the query vector and a local key-value cache of each of the plurality of nodes to obtain an initial attention vector corresponding to each of the plurality of nodes respectively; The plurality of slave nodes are configured to send the initial attention vector corresponding to each of the plurality of slave nodes to the master node through the network device; The network device is configured to aggregate the initial attention vector corresponding to each of the plurality of slave nodes to obtain a target attention vector; The master node is further configured to perform model inference based on the target attention vector through the Transformer model to obtain an output token of the Transformer model.

9. An electronic device, comprising: The device comprises a processor, a memory and a system bus; The processor and the memory are connected through the system bus; The memory is configured to store a program, the program comprising instructions which, when executed by the processor, cause the processor to perform the steps of the data inference method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store a computer program which, when executed by a terminal device, implements the steps of the data inference method of any one of claims 1 to 7.

Images