Data reasoning method, system and equipment and storage medium

Abstract

The invention discloses a data reasoning method, system and device and a storage medium, and the method comprises the steps that a master node generates a query vector, a key vector and a value vector based on an input lexical element of a current layer of a Transform model, and adds the key vector and the value vector to a local key value cache of the master node; the master node broadcasts query vectors to a plurality of slave nodes through network equipment; the plurality of nodes perform attention calculation based on the query vector and the local key value caches of the plurality of nodes to obtain initial attention vectors corresponding to the plurality of nodes; the plurality of slave nodes send respective initial attention vectors to the master node through the network device; the network device performs aggregation processing on the initial attention vectors corresponding to the plurality of slave nodes to obtain a target attention vector; and the main node performs model reasoning based on the target attention vector through a Transform model to obtain an output lexical element. Therefore, the reasoning time can be reduced, and the network bandwidth requirement is reduced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data reasoning method, system, device and storage medium. Background Art

[0002] Large language model inference refers to the process of using a trained large language model for inference, where the user inputs prompt information into the model and the model outputs the answer information corresponding to the prompt information. The input and output of the model are collectively referred to as context.

[0003] In practical applications, as the context of the model continues to grow, the amount of computation required for reasoning with a large language model will increase linearly. Based on this, Ring Attention (a ring attention mechanism with a distributed architecture) can be used to run the reasoning process of a large language model between multiple distributed nodes, decomposing the entire reasoning process in a distributed memory manner. However, in long-context reasoning scenarios, the length of the text sequence that the model needs to process and understand may be infinitely long, and the above solution still has flaws. The reasoning task takes a long time and requires a huge amount of network bandwidth, resulting in poor reasoning results for long contexts. Summary of the invention

[0004] Embodiments of the present application provide a data reasoning method, system, device, and storage medium to reduce the time spent on reasoning tasks and reduce the demand for network bandwidth.

[0005] In a first aspect, an embodiment of the present application provides a data reasoning method, which is applied to a data reasoning system, wherein the data reasoning system adopts a transformer model as a reasoning model, and the data reasoning system includes a network device and multiple nodes, wherein the multiple nodes include a master node and multiple slave nodes, and the data reasoning method includes:

[0006] The master node generates a query vector, a key vector, and a value vector based on the input word of the current layer of the Transformer model, and adds the key vector and the value vector to the local key-value cache of the master node;

[0007] The master node broadcasts the query vector to the plurality of slave nodes through the network device;

[0008] The multiple nodes perform attention calculation based on the query vector and the local key-value caches of the multiple nodes to obtain initial attention vectors corresponding to the multiple nodes respectively;

[0009] The multiple slave nodes send their respective initial attention vectors to the master node through the network device;

[0010] The network device aggregates the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector;

[0011] The master node performs model inference based on the target attention vector through the Transformer model to obtain the output word of the Transformer model.

[0012] Optionally, the network device is a network card, and the multiple nodes are all communicatively connected to the network card;

[0013] The master node broadcasts the query vector to the plurality of slave nodes through the network device, including:

[0014] The master node sends a first data packet to the multiple slave nodes through the network card, wherein the first data packet includes the query vector, the layer identifier of the current layer, the reasoning task identifier, and the identifier of the attention cache of the master node;

[0015] The multiple nodes perform attention calculation based on the query vector and the local key-value caches of the multiple nodes to obtain initial attention vectors corresponding to the multiple nodes, including:

[0016] The master node performs attention calculation based on the query vector and the local key-value cache of the master node to obtain an initial attention vector corresponding to the master node, and writes the initial attention vector corresponding to the master node into the attention cache of the master node;

[0017] The multiple slave nodes obtain the first data packet, and index based on the layer identifier of the current layer and the reasoning task identifier, determine the local key-value cache of each of the multiple slave nodes, and perform attention calculation based on the query vector and the local key-value cache of each of the multiple slave nodes to obtain the initial attention vectors corresponding to the multiple slave nodes respectively.

[0018] Optionally, the multiple slave nodes send their respective initial attention vectors to the master node through the network device, including:

[0019] The multiple slave nodes respectively call the network card to send a second data packet to the master node, and the second data packets respectively sent by the multiple slave nodes include the initial attention vectors, scaling factors and the identification of the attention cache of the master node of the respective multiple slave nodes;

[0020] The network device aggregates the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector, including:

[0021] When receiving a second data packet sent by a first slave node among the multiple slave nodes, the network card aggregates the initial attention vector of the first slave node and the locally stored attention vector based on the locally stored scaling factor and the scaling factor of the first slave node to obtain an aggregated attention vector and an aggregated scaling factor, and stores the aggregated attention vector and the aggregated scaling factor locally;

[0022] When receiving a second data packet sent by a second slave node among the multiple slave nodes, the network card continues to aggregate the initial attention vector of the second slave node and the aggregated attention vector based on the aggregated scaling factor and the scaling factor of the second slave node until the initial attention vectors corresponding to the multiple slave nodes are all aggregated to obtain the target attention vector;

[0023] The network card writes the target attention vector into the attention cache of the master node based on the identifier of the attention cache of the master node.

[0024] Optionally, the network device includes a plurality of switches, the plurality of nodes and the plurality of switches form a tree structure, the plurality of nodes serve as leaf nodes of the tree structure respectively, and the plurality of switches serve as non-leaf nodes of the tree structure respectively;

[0025] The master node broadcasts the query vector to the plurality of slave nodes through the network device, including:

[0026] The master node sends the query vector to a root node in the tree structure, the master node being connected to the root node;

[0027] The root node sends the query vector to a first child node in the tree structure, where the first child node is connected to the root node and is located at a layer below the root node;

[0028] When the first child node is a non-leaf node, the first child node continues to send the query vector to the second child node in the tree structure until the query vector is sent to other leaf nodes among the leaf nodes except the main node, and the second child node is connected to the first child node and is located at the next layer of the first child node.

[0029] Optionally, for each of the multiple slave nodes, the multiple slave nodes send their respective initial attention vectors to the master node through the network device, including:

[0030] Each slave node sends a third data packet to the first parent node in the tree structure, each slave node is connected to the first parent node corresponding to each slave node and is located at a layer below the first parent node corresponding to each slave node, and the third data packet sent by each slave node includes an initial attention vector and a scaling factor of each slave node;

[0031] The network device aggregates the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector, including:

[0032] Upon receiving the third data packet sent by the corresponding slave node, the first parent node of each slave node aggregates the initial attention vector of the corresponding slave node and the locally stored attention vector based on the locally stored scaling factor and the scaling factor of the corresponding slave node to obtain an aggregated attention vector and an aggregated scaling factor, and stores the aggregated attention vector and the aggregated scaling factor locally;

[0033] In the case that there is no root node in the first parent node of each slave node, the first parent node of each slave node sends the aggregated attention vector to the second parent node of each slave node in the tree structure until the root node receives the aggregated intermediate attention vectors corresponding to the plurality of slave nodes, and the second parent nodes of the first parent nodes are connected to the first parent nodes corresponding to the first parent nodes, and are located in an upper layer of the first parent nodes corresponding to the first parent nodes;

[0034] The root node aggregates the aggregated attention vectors corresponding to the multiple slave nodes to obtain the target attention vector.

[0035] Optionally, before the master node generates a query vector, a key vector, and a value vector based on the input word of the current layer of the Transformer model, the method further includes:

[0036] The master node encodes the input text to obtain an encoding result, and distributes the encoding result to the multiple nodes;

[0037] Each of the multiple nodes performs model reasoning based on the assigned encoding result through the Transformer model to obtain a query vector, a local key-value cache, and a first attention vector of each node, and broadcasts the local key-value cache of each node to other nodes in the multiple nodes;

[0038] Each node generates a second attention vector with other nodes based on its own query vector and the local key-value cache broadcasted by the other nodes;

[0039] Each node aggregates the first attention vector and the second attention vector between the node and other nodes to obtain a third attention vector for each node;

[0040] Each node generates an output vector based on the third attention vector through the Transformer model;

[0041] The master node receives the output vectors respectively sent by the plurality of slave nodes, and generates the input word element of the current layer based on its own output vector and the output vectors respectively sent by the plurality of slave nodes.

[0042] Optionally, the data reasoning method further includes:

[0043] When the local key-value cache of the master node is occupied to the maximum capacity, allocating a new node to the data reasoning system;

[0044] Move the local key-value cache of the master node to the new node, or change the master node to a slave node and use the new node as the new master node.

[0045] In a second aspect, an embodiment of the present application provides a data reasoning system, wherein the data reasoning system adopts a transformer model as a reasoning model, and the data reasoning system includes a network device and a plurality of nodes, wherein the plurality of nodes include a master node and a plurality of slave nodes;

[0046] The master node is configured to generate a query vector, a key vector, and a value vector based on an input word of a current layer of the Transformer model, and add the key vector and the value vector to a local key-value cache of the master node; and broadcast the query vector to the plurality of slave nodes through the network device;

[0047] The multiple nodes are used to perform attention calculation based on the query vector and the local key-value caches of the multiple nodes to obtain initial attention vectors corresponding to the multiple nodes respectively;

[0048] The multiple slave nodes are used to send their respective initial attention vectors to the master node through the network device;

[0049] The network device is used to aggregate the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector;

[0050] The master node is also used to perform model inference based on the target attention vector through the Transformer model to obtain the output word of the Transformer model.

[0051] In a third aspect, an embodiment of the present application provides an electronic device, the device comprising: a processor, a memory, and a system bus;

[0052] The processor and the memory are connected via the system bus;

[0053] The memory is used to store a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes any implementation step of the above-mentioned data reasoning method based on weak measurement technology.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on an electronic device, the electronic device executes any implementation step of the above-mentioned data inference method based on weak measurement technology.

[0055] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0056] In an embodiment of the present application, the data reasoning system can use a transformer model as a reasoning model. Accordingly, the data reasoning system includes a network device and multiple nodes, and the multiple nodes include a master node and multiple slave nodes. Based on this, the master node can first generate a query vector, a key vector and a value vector based on the input word element of the current layer of the Transformer model, and add the key vector and the value vector to the local key-value cache of the master node. Then, the master node can broadcast the query vector to multiple slave nodes through the network device, so that the multiple nodes can perform attention calculations based on the query vector and the local key-value caches of the multiple nodes to obtain the initial attention vectors corresponding to the multiple nodes. Then, the multiple slave nodes can send their respective initial attention vectors to the master node through the network device, so that the network device can aggregate the initial attention vectors corresponding to the multiple slave nodes to obtain the target attention vector. Finally, the master node performs model reasoning based on the target attention vector through the Transformer model to obtain the output word element of the Transformer model.

[0057] In this way, the master node can only broadcast the query vector. Compared with broadcasting the local key-value cache, since the query vector occupies a smaller cache, this can help reduce the network's bandwidth requirements and reduce the time spent on reasoning tasks. For example, in a long-context reasoning scenario, this can significantly improve the reasoning efficiency of the long context and improve the reasoning effect. In addition, by using network devices to aggregate the initial attention vectors corresponding to multiple slave nodes, the computing power of the network devices can be fully utilized, saving the computing resources of the graphics processors of multiple nodes, thereby further reducing the time spent on reasoning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of the structure of a data reasoning system provided in an embodiment of the present application;

[0059] Figure 2 A flowchart of a data reasoning method provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of the overall framework of a data reasoning method provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the overall framework of another data reasoning method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] As mentioned above, as the context of the model continues to grow, the amount of computation required to reason about a large language model increases linearly. Based on this, Ring Attention (a ring attention mechanism with a distributed architecture) can be used to run the reasoning process of a large language model across multiple distributed nodes.

[0063] Specifically, for an inference task, Ring Attention can determine a fixed master node to perform complete transformer model inference, and each node saves its own local key-value cache in the form of a ring topology and transmits it in a counterclockwise or clockwise direction. Correspondingly, the master node will continuously receive the local key-value cache from the neighboring nodes to iteratively calculate the attention vector.

[0064] However, since the main node of Ring Aattention is fixed and the local key-value cache of each node is transmitted in the form of a ring topology, the entire reasoning task needs to go through n×l transmissions and calculations, where n represents the number of nodes and l represents the total number of layers of the model. As a result, a single reasoning task takes a long time, and the longer a single reasoning task takes, the longer it stays in the memory of the graphics processing unit (GPU), which greatly affects the number of concurrent tasks in the system and causes a large overall task delay.

[0065] Moreover, each node in Ring Attention not only needs to retain its own local key-value cache, but also needs to manually transmit the local key-value cache from other nodes in the ring topology. Therefore, twice the memory space is required, resulting in a large memory overhead.

[0066] In addition, the process of transmitting the local key-value cache between nodes is equivalent to each node receiving and sending a complete local key-value cache every time an inference task is performed. Therefore, the demand for network bandwidth is huge.

[0067] It can be seen that in the above Ring Attention scheme, although the entire reasoning process can be decomposed in a distributed memory manner, in the long-context reasoning scenario, the length of the text sequence that the model needs to process and understand may reach an infinite length, and the above scheme still has flaws. The reasoning task takes a long time, the memory overhead is large, and the demand for network bandwidth is huge, resulting in poor reasoning effect on long contexts.

[0068] Based on this, in order to solve the above problems, the embodiment of the present application provides a data reasoning method, and is applied to a data processing system. Among them, the data reasoning system can use a transformer Transformer model as a reasoning model, and accordingly, the data reasoning system includes a network device and multiple nodes, and the multiple nodes include a master node and multiple slave nodes. Based on this, the above master node can first generate a query vector, a key vector and a value vector based on the input word element of the current layer of the Transformer model, and add the key vector and the value vector to the local key-value cache of the master node. Then, the master node can broadcast the query vector to multiple slave nodes through the network device, so that multiple nodes can perform attention calculation based on the above query vector and the local key-value cache of each of the multiple nodes to obtain the initial attention vectors corresponding to the multiple nodes. Then, the above multiple slave nodes can send their respective initial attention vectors to the master node through the network device, so that the network device can aggregate the initial attention vectors corresponding to the multiple slave nodes to obtain the target attention vector. Finally, the master node performs model reasoning based on the target attention vector through the Transformer model to obtain the output word element of the Transformer model.

[0069] In this way, the master node can only broadcast the query vector. Compared with broadcasting the local key-value cache, since the query vector occupies a smaller cache, this can help reduce the network's bandwidth requirements and reduce the time spent on reasoning tasks. For example, in a long-context reasoning scenario, this can significantly improve the reasoning efficiency of the long context and improve the reasoning effect. In addition, by using network devices to aggregate the initial attention vectors corresponding to multiple slave nodes, the computing power of the network devices can be fully utilized, saving the computing resources of the graphics processors of multiple nodes, thereby further reducing the time spent on reasoning tasks.

[0070] In addition, compared to Ring Attention, which requires n-1 calculation cycles to obtain an attention vector, in an embodiment of the present application, the master node and multiple slave nodes can calculate the attention vector simultaneously, and are aggregated by the network device, which has little impact on the performance of the GPU. Therefore, when the node scale n is large, the execution time of a single reasoning task can be further reduced, achieving faster reasoning speed. Furthermore, faster reasoning speed also means that the local key-value cache resides in the GPU memory for a shorter time, so the data reasoning system can have more resources allocated to other concurrent reasoning tasks.

[0071] It should be noted that the embodiments of the present application may not limit the execution subject of the data reasoning method. For example, the data reasoning method of the embodiments of the present application may be applied to data processing devices such as terminal devices or servers. Among them, the terminal device may be an electronic device such as a smart phone, a computer, a personal digital assistant (PDA), a tablet computer, etc. The server may be an independent server, a cluster server, or a cloud server.

[0072] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0073] Figure 1 A schematic diagram of the structure of a data reasoning system provided in an embodiment of the present application. Figure 1 As shown, the data reasoning system provided in the embodiment of the present application adopts a transformer model as a reasoning model. Accordingly, the data reasoning system includes a network device and a plurality of nodes, wherein the plurality of nodes include a master node and a plurality of slave nodes.

[0074] The Transformer model can be run on each of the multiple nodes as an inference model to perform data inference tasks. Specifically, the data inference model can be divided into a prefill phase and a generation phase. The two phases are exemplarily described below.

[0075] For the prefill stage of large model reasoning, the model can generate a query vector q, a key vector k, and a value vector v based on the input text, and then calculate the first output word token. Since the prefill stage is mainly computationally limited, the method of transmitting KV cache instead of Q can make the computation more local, thereby ensuring efficiency.

[0076] Based on this, in the specific implementation, first, the master node can encode the input text, and then distribute the encoding results to different nodes. Then, when calculating each layer of the transformer model, each of the multiple nodes can generate a query vector q, a key vector k and a value vector v according to the input text, wherein the key vector k and the value vector v can constitute the local key-value cache KV cache of each node. At this time, each node can further combine its own query vector q, key vector k and value vector v to perform attention calculation to obtain the first attention vector of each node. Then, all nodes can broadcast the local key-value cache KV cache to other nodes. When each node receives the local key-value cache KV cache sent by other nodes, it can perform attention calculation in combination with its own query vector q to generate the second attention vector between each node and other nodes. Then, each node can aggregate the above first attention vector and all the second attention vectors through aggregation operation to generate the third attention vector of each node. Finally, each node can continue to use the third attention vector to perform model reasoning to obtain the output vector of the transformer model, and send the output vector to the master node, which generates the first output word element token.

[0077] For the generation stage of large model reasoning, the model can generate the next output word meta-token based on each new input token and the historical key value cache KV cache.

[0078] In the specific implementation, first, when the master node calculates each layer of the transformer model, it can obtain the query vector q, key vector k and value vector v according to the input word of the current layer. Among them, the query vector q will be broadcast to multiple slave nodes, and the key vector k and value vector v will be added to the local key-value cache KV cache of the master node. Then, multiple nodes can perform attention calculations based on the above query vector q and their respective local key-value cache KV cache to obtain the initial attention vectors corresponding to the multiple nodes. Then, multiple slave nodes among the multiple nodes can send the above initial attention vectors to the master node through the network device, and the network device can aggregate the initial attention vectors corresponding to the multiple slave nodes through online aggregation operations to obtain the target attention vector. Then, the master node can continue to use the target attention vector for model reasoning to obtain the next output word of the Transformer model.

[0079] It should be noted that in the above process, if the local cache key value KV cache of the master node is full, a new node can be allocated to the system, and the local cache key value KV cache of the master node can be moved to the new node. Alternatively, the above master node is changed to a slave node, and the new node is used as the new master node. Therefore, the master node is not fixed. In addition, for different reasoning tasks, the master node may also be different, rather than fixed.

[0080] In summary, combined with the overall architecture of the above-mentioned data reasoning system, in the embodiment of the present application, in the generation stage of large model reasoning, the master node can only broadcast the query vector. Compared with broadcasting the local key-value cache, the cache occupied by the query vector is smaller. Therefore, this can help reduce the bandwidth requirements of the network and reduce the time spent on the reasoning task. In addition, by using the network device to aggregate the initial attention vectors corresponding to multiple slave nodes, the computing power of the network device can be fully utilized, saving the computing resources of the graphics processors of multiple nodes, thereby further reducing the time spent on the reasoning task. In addition, compared with Ring Attention, which requires n-1 calculation cycles to obtain an attention vector, the master node and multiple slave nodes can calculate the attention vector at the same time, and the network device is aggregated, which has little impact on the performance of the GPU. Therefore, when the node scale n is large, the execution time of a single reasoning task can be further reduced, achieving a faster reasoning speed. Furthermore, a faster reasoning speed also means that the local key-value cache resides in the GPU memory for a shorter time, so the data reasoning system can have more resources allocated to other concurrent reasoning tasks.

[0081] Based on the data reasoning system provided in the above embodiment, the embodiment of the present application can also provide a data reasoning method for the above generation process. The data reasoning method is described below in conjunction with the embodiments and drawings.

[0082] Figure 2 The flowchart of a data reasoning method provided in the embodiment of the present application. The data reasoning method provided in the embodiment of the present application describes the overall implementation of the scheme by the interaction process between each node in the data reasoning system and the interaction process between each node and the network device. Based on this, combined with Figure 2 As shown, the data reasoning method includes:

[0083] S101: The master node generates a query vector, a key vector, and a value vector based on the input word of the current layer of the Transformer model, and adds the key vector and the value vector to the local key-value cache of the master node.

[0084] In practical applications, the generation process of query vectors, key vectors and value vectors is not specifically limited in the embodiments of the present application, and any existing or future algorithm for generating query vectors, key vectors and value vectors may be used for implementation. For example, the master node may compare the input word of the current layer of the Transformer model with the parameter matrix W Q ,W K ,W V Perform matrix multiplication to obtain the query vector, key vector, and value vector corresponding to the input word.

[0085] Furthermore, when the master node adds the key vector and the value vector to the local key-value cache of the master node, if the local key-value cache of the master node is full, that is, when the local key-value cache of the master node is occupied to the maximum capacity, a new node can be allocated to the data inference system first, and the local key-value cache of the master node can be moved to the new node, or the above-mentioned master node can be changed to a slave node, and the new node can be used as the new master node.

[0086] S102: The master node broadcasts the query vector to multiple slave nodes through a network device.

[0087] In practical applications, the above network device may be a network card or a switch. The network card may be implemented by a smart network card (Smart Network Interface Card, SmartNIC) or a data processor (Data Processing Unit, DPU) that supports online aggregation operations. For ease of understanding, the broadcast process may be illustrated below by combining two different network devices, a network card and a switch.

[0088] As an example, combining Figure 3 As shown, the network device takes a network card as an example, and accordingly, multiple nodes are all connected to the network card for communication to achieve end-to-end communication transmission.

[0089] Based on this, for step S102, in specific implementation, the master node can send a first data packet to multiple slave nodes through a network card. In practical applications, the first data packet includes a query vector, a layer identifier of the current layer, an inference task identifier, and an identifier of the attention cache of the master node.

[0090] Among them, the layer identifier of the current layer can uniquely represent the current layer of the Transformer model, and the inference task identifier can uniquely represent the currently executed inference task. Therefore, the layer identifier and the inference task identifier of the current layer can help index the slave node to the local key-value cache corresponding to the current layer and the currently executed inference task. In addition, the identifier of the attention cache of the master node can uniquely represent the attention cache of the master node. Therefore, through the identifier of the attention cache of the master node, the slave node can quickly and accurately determine the address of the attention cache of the master node, which is convenient for writing back the calculated initial attention vector.

[0091] In this way, while the master node broadcasts the query vector to multiple slave nodes, it can also send other information to multiple slave nodes, so that multiple slave nodes can not only calculate their own initial attention based on the above query vector, but also index their own local key-value cache through the layer identifier and inference task identifier, and write the calculated initial attention vector back to the attention cache of the above master node.

[0092] In addition, in an embodiment of the present application, in order to improve the model reasoning effect, the offset information between the input features can be further considered when performing attention calculations, so as to capture and understand the features and the relationship between the features. Based on this, in an embodiment of the present application, the above-mentioned first data packet may also include offset information (offset) corresponding to the payload (payload). Wherein, payload refers to the data information contained in the first data packet. In this way, attention calculations can be further performed based on the offset corresponding to the payload in the future. By considering the offset between the features, the model can be helped to more accurately identify and interpret the data, thereby improving the reasoning performance of the model.

[0093] As another example, combining Figure 4 As shown, the network device takes a switch that supports online aggregation operations as an example. Since the switch is naturally located at the convergence position of the network, it is more convenient to perform aggregation processing. Accordingly, there can be multiple switches, and the multiple nodes are interconnected through forwarding of multiple switches.

[0094] Furthermore, a tree structure can be constructed based on the above multiple nodes and multiple switches, that is, the above multiple nodes and multiple switches constitute a tree structure. In practical applications, the tree structure may include leaf nodes and non-leaf nodes, and accordingly, the above multiple nodes are respectively used as leaf nodes of the tree structure, and the multiple switches are respectively used as non-leaf nodes of the tree structure. Among them, the non-leaf nodes also include the root node of the tree structure, and the intermediate nodes between the root node and the leaf nodes.

[0095] In the tree structure, two connected layers of nodes are parent-child nodes, that is, the upper layer node is the parent node of the lower layer node it is connected to, and the lower layer node is the child node of the upper layer node it is connected to. Accordingly, multiple nodes in the above data reasoning system can maintain their own parent node information and child node information.

[0096] Based on this, for step S102, in a specific implementation, first, the master node can send a query vector to the root node in the tree structure, wherein the master node is connected to the root node. That is, the master node unicasts the query vector to the switch serving as the root node. In practical applications, the master node can also unicast other data in the first data packet, such as the identifiers of each node in the tree structure and the identifier of the attention cache of the master node, to the root node.

[0097] Next, the root node may send a query vector to the first child node in the tree structure. The first child node is connected to the root node and is located at the next layer of the root node. Here, the root node may query its own child node, i.e., the first child node, according to the identifiers of each node in the tree structure sent by the master node, and send a query vector to the first child node.

[0098] In this way, when the first child node is a leaf node and not a master node, the first leaf node can represent a slave node in the data reasoning system. Therefore, the query vector has been sent to the slave node, and the entire sending process can be ended.

[0099] In the case where the first child node is a non-leaf node, the first child node can continue to send the query vector to the second child node in the tree structure until the query vector is sent to the other leaf nodes except the master node in the leaf node, and the second child node is connected to the first child node and is located at the next layer of the first child node. In other words, when the first child node is a non-leaf node but an intermediate node, the query vector can be sent to the next layer of nodes with the help of the switch as the intermediate node, and so on until the query vector is sent to the other leaf nodes, that is, multiple slave nodes, and the entire sending process can be ended.

[0100] S103: The multiple nodes perform attention calculation based on the query vector and the local key-value caches of the multiple nodes to obtain initial attention vectors corresponding to the multiple nodes.

[0101] In an embodiment of the present application, for step S103, in the specific implementation, first, the master node can perform attention calculation based on the query vector and the local key-value cache of the master node, obtain the initial attention vector corresponding to the master node, and write the initial attention vector corresponding to the master node into the attention cache of the master node.

[0102] Here, after the master node calculation is completed, the data inference system can first put the current inference task into the waiting queue and switch to other inference tasks for scheduling to prevent the GPU of the master node from idling.

[0103] Then, multiple slave nodes can obtain the first data packet sent by the master node through the network card, and index based on the layer identifier and the reasoning task identifier of the current layer to determine the local key value cache of each of the multiple slave nodes. In actual applications, multiple slave nodes can index based on the above-mentioned reasoning task identifier and the layer identifier of the current layer to determine the local key value cache currently required by each of them.

[0104] Then, multiple slave nodes can perform attention calculation based on the query vector and the local key-value caches of the multiple slave nodes to obtain initial attention vectors corresponding to the multiple slave nodes.

[0105] In practical applications, after receiving the first data packet, the multiple slave nodes may each generate a task with the highest priority to quickly start the attention calculation process. Alternatively, the multiple slave nodes may generate a scheduling operation that supports task termination and preemption to reduce the waiting time of the attention calculation process.

[0106] In addition, it should be noted that the above-mentioned attention calculation process can be implemented using any existing or future attention algorithm, and the embodiments of the present application do not specifically limit this.

[0107] S104: Multiple slave nodes send their respective initial attention vectors to the master node through the network device.

[0108] As mentioned above, the above network device may be a network card or a switch. Based on this, for ease of understanding, the implementation process of step S104 is described below in combination with two different network devices.

[0109] As an example, combining Figure 3As shown, the network device takes a network card as an example, and the above step S104 may include: multiple slave nodes respectively call the network card to send a second data packet to the master node, wherein the second data packets respectively sent by the multiple slave nodes include the initial attention vectors of the multiple slave nodes, the scaling factor and the identification of the attention cache of the master node. In this way, the network card can perform attention aggregation on each initial attention vector based on the second data packets respectively sent by the multiple slave nodes.

[0110] In addition, the second data packet sent by each slave node may also include a slave node bitmap. A slave node bitmap refers to a data structure that uses bits to represent the state of a slave node. For example, taking 32 slave nodes as an example, a 32-bit bitmap can be used as a slave node bitmap, and each bit (1 bit) in the slave node bitmap corresponds to a node, for example, the 0th bit represents the state of node 1, the 1st bit represents the state of node 2, and so on, the 31st bit represents the state of node 32. Accordingly, when the value of a certain bit is 0, it can be indicated that the initial attention vector corresponding to the slave node has not been aggregated, and when the value is 1, it can be indicated that the initial attention vector corresponding to the slave node has been aggregated. Therefore, when the values ​​of all bits in the slave node bitmap are 1, it can be indicated that the entire aggregation process is completed.

[0111] As another example, combining Figure 4 As shown, the network device takes a switch as an example, for each of the multiple slave nodes, the above step S104 may include: each slave node sends a third data packet to the first parent node in the tree structure, wherein each slave node is respectively connected to the first parent node corresponding to each other, and is located at the next layer of the first parent node corresponding to each other, and the third data packet sent by each slave node includes the initial attention vector and scaling factor of each slave node. Here, since the multiple slave nodes are all leaf nodes in the tree structure, each slave node has its own first parent node, and the first parent nodes of each slave node can be consistent. Then, each slave node uploads the initial attention vector to the switch as the first parent node, so that the switch as the first parent node aggregates the received initial attention vector.

[0112] In addition, the third data packet sent by each slave node also includes a slave node bitmap. A slave node bitmap refers to a data structure that uses bits to represent the state of a slave node. For example, taking 32 slave nodes as an example, a 32-bit bitmap can be used as a slave node bitmap, and each bit (1 bit) in the slave node bitmap corresponds to a node, for example, the 0th bit represents the state of node 1, the 1st bit represents the state of node 2, and so on, the 31st bit represents the state of node 32. Accordingly, when the value of a certain bit is 0, it can be indicated that the initial attention vector corresponding to the slave node has not been aggregated, and when the value is 1, it can be indicated that the initial attention vector corresponding to the slave node has been aggregated. Therefore, when the values ​​of all bits in the slave node bitmap are 1, it can be indicated that the entire aggregation process is completed.

[0113] It should be noted that the second data packet and the third data packet are both data packets sent from the slave node, and the data content included in the two are the same. For ease of understanding, in the embodiment of the present application, in the scenario where the network device is a network card, the second data packet is used to describe the data packet sent from the slave node to the master node, and in the scenario where the network device is a switch, the third data packet is used to describe the data packet sent from the slave node to the corresponding parent node in the tree structure.

[0114] S105: The network device aggregates the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector.

[0115] As a possible implementation, combining Figure 3 As shown, based on the implementation process of step S104 for the network card, when the above step S105 is specifically implemented, first, when the network card receives the second data packet sent by the first slave node among multiple slave nodes, based on the locally stored scaling factor and the scaling factor of the first slave node, the initial attention vector of the first slave node and the locally stored attention vector are aggregated to obtain the aggregated attention vector and the aggregated scaling factor, and the aggregated attention vector and the aggregated scaling factor are stored locally.

[0116] For example, the aggregation process is implemented using the SOFTMAX function, and the log value of the denominator of the SOFTMAX function is based on 2. Based on this, the above-mentioned aggregation process can be shown as the following formula (1):

[0117]

[0118] Among them, a0 is the attention vector after aggregation, a is the attention vector stored locally on the network card, l is the scaling factor stored locally on the network card, l0 is the scaling factor after aggregation, and a ′ is the initial attention vector of the first slave node, l ′is the scaling factor of the first slave node.

[0119] It should be noted that the above formula (1) specifically describes a process of aggregation processing. The subsequent aggregation processing can be implemented by using the above formula (1), which will not be described in detail below.

[0120] In addition, in practical applications, when the network card performs the above aggregation operation, a software lock or a hardware lock can be used to ensure the atomicity of the operation to avoid errors when the initial attention vectors of multiple slave nodes are aggregated at the same time.

[0121] Then, when the network card receives the second data packet sent by the second slave node among the multiple slave nodes, it continues to perform aggregation processing on the initial attention vector and the aggregated attention vector of the second slave node based on the aggregated scaling factor and the scaling factor of the second slave node, until the initial attention vectors corresponding to the multiple slave nodes are all aggregated to obtain the target attention vector. In other words, the network card can start the aggregation process after receiving the second data packet corresponding to a slave node, and can directly perform aggregation processing based on the latest data currently stored locally (i.e., the above-mentioned aggregated scaling factor and aggregated attention vector) and the data in the received second data packet (i.e., the above-mentioned scaling factor and initial attention vector of the second slave node), so that the network card can achieve real-time pairwise aggregation without having to wait until the initial attention vectors of all slave nodes are received for one-time aggregation. Therefore, it can effectively save the computing resources of the GPU and improve computing efficiency.

[0122] The process of aggregating the initial attention vectors of the second slave node and other slave nodes can be found in the above formula (1), which will not be repeated here.

[0123] Finally, the network card can write the target attention vector into the attention cache of the master node based on the identifier of the attention cache of the master node. In this way, the subsequent master node can continue to use the target attention vector for model reasoning to obtain new output words. In addition, the entire process only needs to maintain the attention cache of the master node, and there is no need to maintain the attention cache of each slave node. Therefore, the demand for memory capacity can be reduced and memory overhead can be reduced.

[0124] In addition, as mentioned above, the second data packet also includes a slave node bitmap. Based on this, taking a certain slave node as an example, during the process of the network card performing aggregation processing, aggregation can be performed based on the slave node bitmap, and the slave node bitmap can be updated accordingly after the aggregation is completed, that is, the node status is updated. In this way, during the aggregation process, repeated aggregation can be avoided based on the node status indicated by the slave node bitmap, and further updating the slave node bitmap after the aggregation is completed can also avoid subsequent aggregation duplication, thereby improving the accuracy of aggregation.

[0125] As another possible implementation, combining Figure 4 As shown, based on the implementation process of step S104 for the switch, when the above step S105 is specifically implemented, first, when the first parent node of each slave node receives the third data packet sent by the corresponding slave node, based on the locally stored scaling factor and the scaling factor of the corresponding slave node, the initial attention vector of the corresponding slave node and the locally stored attention vector are aggregated to obtain the aggregated attention vector and the aggregated scaling factor, and the aggregated attention vector and the aggregated scaling factor are stored locally. That is to say, after receiving the third data packet sent by the corresponding slave node, the switch to which each slave node is connected can start the aggregation operation, and the aggregation operation can still be implemented with reference to the above formula (1), which will not be repeated here.

[0126] In addition, as mentioned above, the third data packet also includes a slave node bitmap. Based on this, taking a slave node as an example, when the parent node, i.e., the switch, performs aggregation processing, it can perform aggregation based on the slave node bitmap, and update the slave node bitmap accordingly after the aggregation is completed, that is, update the node status. In this way, during the aggregation process, repeated aggregation can be avoided based on the node status indicated by the slave node bitmap, and further updating the slave node bitmap after the aggregation is completed can also avoid subsequent aggregation duplication, thereby improving the accuracy of aggregation.

[0127] Next, when there is no root node in the first parent node of each slave node, the first parent node of each slave node sends the aggregated attention vector to the second parent node of each slave node in the tree structure until the root node receives the aggregated intermediate attention vectors corresponding to multiple slave nodes. The second parent nodes of the first parent nodes are connected to the first parent nodes of each slave node and are located in the upper layer of the first parent nodes of each slave node.

[0128] That is to say, after each slave node sends the third data packet to its respective first parent node, if there is no root node in the first parent node, each first parent node can further construct data upward, that is, the first parent node sends the aggregated attention vector upward layer by layer according to the tree structure level until it reaches the root node for the final aggregation process.

[0129] Then, the root node aggregates the aggregated attention vectors corresponding to the multiple slave nodes to obtain the target attention vector. In this way, the root node can send the target attention vector to the master node to write it into the master node's attention cache, thereby invoking the master node to perform the corresponding reasoning task.

[0130] S106: The master node performs model inference based on the target attention vector through the Transformer model to obtain the output word of the Transformer model.

[0131] In practical applications, the network devices, namely the above-mentioned network card and the switch as the root node, can generate a scheduling event indicating the completion of aggregation after the aggregation is completed, so that the data inference system can respond to the scheduling event to schedule the GPU of the main node to perform subsequent model inference based on the target attention vector.

[0132] Specifically, after the main node obtains the target attention vector, it can continue to execute the residual connection and layer normalization (Add&Norm) stage and the feedforward network (FFN) stage in the above Transformer model. Specifically, first, the target attention vector can be processed by Add&Norm to obtain the processing result, and then the processing result is nonlinearly transformed by FFN to obtain the transformation result. Finally, Add&Norm can be applied again to process the transformation result to obtain the processing result, which can be used as the output of the current layer of the Transformer model. In practical applications, the Transformer model adopts a multi-layer structure. Therefore, for each layer of the Transformer model, it can be processed according to the above process to obtain the corresponding output of each layer. Correspondingly, the output of the last layer of the Transformer model is the above-mentioned output word.

[0133] Based on the relevant contents of the above steps S101-S106, it can be known that in an embodiment of the present application, the data reasoning system can use the transformer model as the reasoning model. Accordingly, the data reasoning system includes a network device and multiple nodes, and the multiple nodes include a master node and multiple slave nodes. Based on this, the master node can first generate a query vector, a key vector and a value vector based on the input word element of the current layer of the Transformer model, and add the key vector and the value vector to the local key-value cache of the master node. Then, the master node can broadcast the query vector to multiple slave nodes through the network device, so that multiple nodes can perform attention calculation based on the query vector and the local key-value cache of each of the multiple nodes to obtain the initial attention vectors corresponding to the multiple nodes. Then, the multiple slave nodes can send their respective initial attention vectors to the master node through the network device, so that the network device can aggregate the initial attention vectors corresponding to the multiple slave nodes to obtain the target attention vector. Finally, the master node performs model reasoning based on the target attention vector through the Transformer model to obtain the output word element of the Transformer model. In this way, the master node can only broadcast the query vector. Compared with broadcasting the local key-value cache, the query vector occupies a smaller cache. Therefore, this can help reduce the network bandwidth requirements and reduce the time spent on reasoning tasks. In addition, by using network devices to aggregate the initial attention vectors corresponding to multiple slave nodes, the computing power of the network devices can be fully utilized, saving the computing resources of the graphics processors of multiple nodes, thereby further reducing the time spent on reasoning tasks.

[0134] In addition, as mentioned above, the data inference model can be divided into the prefill stage and the generation stage. Therefore, before executing step S101, the prefill stage needs to be executed first. Based on this, before step S101, the following steps 1 to 6 can also be executed to complete the prefill stage, which is described in detail below.

[0135] Step 1: The master node encodes the input text to obtain the encoding result, and distributes the encoding result to multiple nodes.

[0136] In the specific implementation, first, the data reasoning system can first calculate the number of word meta-tokens s corresponding to the input text and the maximum number of word meta-tokens s that can be stored in the local key-value cache of each node. b , can be allocated s / s b nodes, and set a master node among these nodes, and the other nodes serve as slave nodes.

[0137] Next, the master node can perform word vector and position encoding on the input text to obtain the encoding result. Then, the master node distributes the encoding result to each node according to the length of the input text. Each node can be assigned s b A word.

[0138] Step 2: Each of the multiple nodes performs model inference based on the assigned encoding results through the Transformer model to obtain the query vector, local key-value cache and first attention vector of each node, and broadcasts the local key-value cache of each node to other nodes in the multiple nodes.

[0139] Specifically, each node can compare the encoding result with the parameter matrix W Q ,W K ,W V Perform matrix multiplication to obtain the query vector, key vector, and value vector of each node, and save the key vector and value vector as a local key-value cache. Then, each node can perform attention calculation based on the query vector and the local key-value cache to obtain the first attention vector.

[0140] Step 3: Each node generates a second attention vector with other nodes based on its own query vector and the key-value cache broadcast by other nodes.

[0141] Here, the process of each node generating the second attention vector with other nodes can be referred to the implementation process of step S103 in the above embodiment, which will not be repeated here.

[0142] Step 4: Each node aggregates the first attention vector and the second attention vectors between other nodes to obtain the third attention vector of each node.

[0143] Here, the calculation process of the third attention vector of each node can refer to the implementation process of step S105 in the above embodiment, which will not be repeated here.

[0144] Step 5: Each node passes through the Transformer model to generate an output vector based on the third attention vector.

[0145] Specifically, after each node obtains the third attention vector, it can continue to execute the residual connection and layer normalization (Add&Norm) stage and the feedforward network (FFN) stage in the above Transformer model. Specifically, first, the third attention vector can be processed by Add&Norm to obtain the processing result, and then the processing result can be nonlinearly transformed by FFN to obtain the transformation result. Finally, Add&Norm can be applied again to process the transformation result to obtain the processing result, which can be used as the output of the current layer of the Transformer model. In practical applications, the Transformer model adopts a multi-layer structure. Therefore, for each layer of the Transformer model, it can be processed according to the above process to obtain the corresponding output of each layer. Correspondingly, the output of the last layer of the Transformer model can obtain the above output vector.

[0146] Step 6: The master node receives the output vectors sent by the multiple slave nodes, and generates the input word element of the current layer based on its own output vector and the output vectors sent by the multiple slave nodes.

[0147] Here, the master node can aggregate the output vectors sent by multiple slave nodes to obtain the input word of the current layer. The implementation process of aggregation is not specifically limited in the embodiment of the present application, and can be implemented by any method of aggregating output vectors that is currently available in the pupa or may appear in the future.

[0148] Furthermore, based on the data reasoning method provided in the above embodiment, the embodiment of the present application can also provide a data reasoning system. The data reasoning system is described below in conjunction with the embodiment. The data reasoning system provided in the embodiment of the present application adopts a transformer model as a reasoning model, and the data reasoning system includes a network device and multiple nodes, and the multiple nodes include a master node and multiple slave nodes;

[0149] The master node is configured to generate a query vector, a key vector, and a value vector based on an input word of a current layer of the Transformer model, and add the key vector and the value vector to a local key-value cache of the master node; and broadcast the query vector to the plurality of slave nodes through the network device;

[0150] The multiple nodes are used to perform attention calculation based on the query vector and the local key-value caches of the multiple nodes to obtain initial attention vectors corresponding to the multiple nodes respectively;

[0151] The multiple slave nodes are used to send their respective initial attention vectors to the master node through the network device;

[0152] The network device is used to aggregate the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector;

[0153] The master node is also used to perform model inference based on the target attention vector through the Transformer model to obtain the output word of the Transformer model.

[0154] As an implementation mode, the network device is a network card, and the multiple nodes are all communicatively connected to the network card;

[0155] The master node is specifically configured to send a first data packet to the plurality of slave nodes through the network card, wherein the first data packet includes the query vector, the layer identifier of the current layer, the inference task identifier, and the identifier of the attention cache of the master node;

[0156] The master node is specifically used to perform attention calculation based on the query vector and the local key-value cache of the master node, obtain an initial attention vector corresponding to the master node, and write the initial attention vector corresponding to the master node into the attention cache of the master node;

[0157] The multiple slave nodes are specifically used to obtain the first data packet, and to perform indexing based on the layer identifier of the current layer and the reasoning task identifier, to determine the local key-value caches of the multiple slave nodes respectively, and to perform attention calculation based on the query vector and the local key-value caches of the multiple slave nodes respectively, to obtain the initial attention vectors corresponding to the multiple slave nodes respectively.

[0158] As an implementation mode, the multiple slave nodes are specifically used to respectively call the network card to send a second data packet to the master node, and the second data packets respectively sent by the multiple slave nodes include the initial attention vectors, scaling factors and identifiers of the attention cache of the master node of the respective multiple slave nodes;

[0159] The network card is specifically used for:

[0160] Upon receiving a second data packet sent by a first slave node among the plurality of slave nodes, based on a locally stored scaling factor and a scaling factor of the first slave node, aggregate the initial attention vector of the first slave node and the locally stored attention vector to obtain an aggregated attention vector and an aggregated scaling factor, and store the aggregated attention vector and the aggregated scaling factor locally;

[0161] When receiving a second data packet sent by a second slave node among the multiple slave nodes, continue to aggregate the initial attention vector of the second slave node and the aggregated attention vector based on the aggregated scaling factor and the scaling factor of the second slave node until the initial attention vectors corresponding to the multiple slave nodes are all aggregated to obtain the target attention vector;

[0162] Based on the identifier of the attention cache of the master node, the target attention vector is written into the attention cache of the master node.

[0163] As an implementation mode, the network device includes a plurality of switches, the plurality of nodes and the plurality of switches form a tree structure, the plurality of nodes serve as leaf nodes of the tree structure, and the plurality of switches serve as non-leaf nodes of the tree structure;

[0164] The master node is specifically used to send the query vector to the root node in the tree structure, and the master node is connected to the root node;

[0165] The root node is used to send the query vector to a first child node in the tree structure, where the first child node is connected to the root node and is located at a layer below the root node;

[0166] The first child node is used to continue sending the query vector to the second child node in the tree structure when the first child node is a non-leaf node, until the query vector is sent to other leaf nodes among the leaf nodes except the main node, and the second child node is connected to the first child node and is located at a layer below the first child node.

[0167] As an implementation manner, for each slave node among the multiple slave nodes, each slave node is specifically used to send a third data packet to the first parent node in the tree structure, each slave node is respectively connected to the first parent node corresponding to each slave node, and is located at a layer below the first parent node corresponding to each slave node, and the third data packet sent by each slave node includes the initial attention vector and scaling factor of each slave node;

[0168] The first parent node of each slave node is configured to aggregate the initial attention vector of the corresponding slave node and the locally stored attention vector based on the locally stored scaling factor and the scaling factor of the corresponding slave node when receiving the third data packet sent by the corresponding slave node, to obtain an aggregated attention vector and an aggregated scaling factor, and to store the aggregated attention vector and the aggregated scaling factor locally;

[0169] Each of the slave nodes is used for, when there is no root node in the respective first parent node of each of the slave nodes, the respective first parent node sends the aggregated attention vector to the respective second parent node in the tree structure, until the root node receives the aggregated intermediate attention vectors respectively corresponding to the plurality of slave nodes, and the respective second parent nodes of the first parent nodes are respectively connected to the respective first parent nodes and are located in an upper layer of the respective first parent nodes;

[0170] The root node is used to aggregate the aggregated attention vectors corresponding to the multiple slave nodes to obtain the target attention vector.

[0171] As an implementation mode, the master node is further used to encode the input text to obtain an encoding result, and distribute the encoding result to the multiple nodes;

[0172] Each of the multiple nodes is further configured to perform model inference based on the assigned encoding result through the Transformer model to obtain a query vector, a local key-value cache, and a first attention vector of each node, and broadcast the local key-value cache of each node to other nodes in the multiple nodes;

[0173] Each of the nodes is further configured to generate a second attention vector between the other nodes based on the respective query vectors and the local key-value cache broadcasted by the other nodes;

[0174] Each node is further configured to aggregate the first attention vector and the second attention vector between the first node and the other node to obtain a third attention vector for each node;

[0175] Each of the nodes is further configured to generate an output vector based on the third attention vector through the Transformer model;

[0176] The master node is further configured to receive the output vectors respectively sent by the plurality of slave nodes, and to generate the input word element of the current layer based on its own output vector and the output vectors respectively sent by the plurality of slave nodes.

[0177] As an implementation method, the data reasoning system is also used to:

[0178] When the local key-value cache of the master node is occupied to the maximum capacity, allocating a new node to the data reasoning system;

[0179] Move the local key-value cache of the master node to the new node, or change the master node to a slave node and use the new node as the new master node.

[0180] Furthermore, an embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a system bus;

[0181] The processor and the memory are connected via the system bus;

[0182] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor performs any implementation step of the above-mentioned data reasoning method.

[0183] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on an electronic device, any implementation step of the above-mentioned data inference method is implemented.

[0184] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus the necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments. It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0185] As for the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method part.

[0186] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0187] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data reasoning method, characterized in that: Applied to a data reasoning system, the data reasoning system adopts a transformer model as a reasoning model, the data reasoning system includes a network device and multiple nodes, the multiple nodes include a master node and multiple slave nodes, and the data reasoning method includes: The master node generates a query vector, a key vector, and a value vector based on the input word of the current layer of the Transformer model, and adds the key vector and the value vector to the local key-value cache of the master node; The master node broadcasts the query vector to the plurality of slave nodes through the network device; The multiple nodes perform attention calculation based on the query vector and the local key-value caches of the multiple nodes to obtain initial attention vectors corresponding to the multiple nodes respectively; The multiple slave nodes send their respective initial attention vectors to the master node through the network device; The network device aggregates the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector; The master node performs model inference based on the target attention vector through the Transformer model to obtain the output word of the Transformer model.

2. The data inference method according to claim 1, characterized in that: The network device is a network card, and the multiple nodes are all communicatively connected to the network card; The master node broadcasts the query vector to the plurality of slave nodes through the network device, including: The master node sends a first data packet to the multiple slave nodes through the network card, wherein the first data packet includes the query vector, the layer identifier of the current layer, the reasoning task identifier, and the identifier of the attention cache of the master node; The multiple nodes perform attention calculation based on the query vector and the local key-value caches of the multiple nodes to obtain initial attention vectors corresponding to the multiple nodes, including: The master node performs attention calculation based on the query vector and the local key-value cache of the master node to obtain an initial attention vector corresponding to the master node, and writes the initial attention vector corresponding to the master node into the attention cache of the master node; The multiple slave nodes obtain the first data packet, and index based on the layer identifier of the current layer and the reasoning task identifier, determine the local key-value cache of each of the multiple slave nodes, and perform attention calculation based on the query vector and the local key-value cache of each of the multiple slave nodes to obtain the initial attention vectors corresponding to the multiple slave nodes respectively.

3. The data inference method according to claim 2, characterized in that: The multiple slave nodes send their respective initial attention vectors to the master node through the network device, including: The multiple slave nodes respectively call the network card to send a second data packet to the master node, and the second data packets respectively sent by the multiple slave nodes include the initial attention vectors, scaling factors and the identification of the attention cache of the master node of the respective multiple slave nodes; The network device aggregates the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector, including: When receiving a second data packet sent by a first slave node among the multiple slave nodes, the network card aggregates the initial attention vector of the first slave node and the locally stored attention vector based on the locally stored scaling factor and the scaling factor of the first slave node to obtain an aggregated attention vector and an aggregated scaling factor, and stores the aggregated attention vector and the aggregated scaling factor locally; When receiving a second data packet sent by a second slave node among the multiple slave nodes, the network card continues to aggregate the initial attention vector of the second slave node and the aggregated attention vector based on the aggregated scaling factor and the scaling factor of the second slave node until the initial attention vectors corresponding to the multiple slave nodes are all aggregated to obtain the target attention vector; The network card writes the target attention vector into the attention cache of the master node based on the identifier of the attention cache of the master node.

4. The data inference method according to claim 1, characterized in that: The network device comprises a plurality of switches, the plurality of nodes and the plurality of switches form a tree structure, the plurality of nodes are respectively used as leaf nodes of the tree structure, and the plurality of switches are respectively used as non-leaf nodes of the tree structure; The master node broadcasts the query vector to the plurality of slave nodes through the network device, including: The master node sends the query vector to a root node in the tree structure, the master node being connected to the root node; The root node sends the query vector to a first child node in the tree structure, where the first child node is connected to the root node and is located at a layer below the root node; When the first child node is a non-leaf node, the first child node continues to send the query vector to the second child node in the tree structure until the query vector is sent to other leaf nodes among the leaf nodes except the main node, and the second child node is connected to the first child node and is located at the next layer of the first child node.

5. The data inference method according to claim 4, characterized in that: For each of the multiple slave nodes, the multiple slave nodes send their respective initial attention vectors to the master node through the network device, including: Each slave node sends a third data packet to the first parent node in the tree structure, each slave node is connected to the first parent node corresponding to each slave node and is located at a layer below the first parent node corresponding to each slave node, and the third data packet sent by each slave node includes an initial attention vector and a scaling factor of each slave node; The network device aggregates the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector, including: Upon receiving the third data packet sent by the corresponding slave node, the first parent node of each slave node aggregates the initial attention vector of the corresponding slave node and the locally stored attention vector based on the locally stored scaling factor and the scaling factor of the corresponding slave node to obtain an aggregated attention vector and an aggregated scaling factor, and stores the aggregated attention vector and the aggregated scaling factor locally; In the case that there is no root node in the first parent node of each slave node, the first parent node of each slave node sends the aggregated attention vector to the second parent node of each slave node in the tree structure until the root node receives the aggregated intermediate attention vectors corresponding to the plurality of slave nodes, and the second parent nodes of the first parent nodes are connected to the first parent nodes corresponding to the first parent nodes, and are located in an upper layer of the first parent nodes corresponding to the first parent nodes; The root node aggregates the aggregated attention vectors corresponding to the multiple slave nodes to obtain the target attention vector.

6. The data inference method according to any one of claims 1 to 5, characterized in that: Before the master node generates a query vector, a key vector, and a value vector based on an input word of a current layer of the Transformer model, the method further includes: The master node encodes the input text to obtain an encoding result, and distributes the encoding result to the multiple nodes; Each of the multiple nodes performs model reasoning based on the assigned encoding result through the Transformer model to obtain a query vector, a local key-value cache, and a first attention vector of each node, and broadcasts the local key-value cache of each node to other nodes in the multiple nodes; Each node generates a second attention vector with other nodes based on its own query vector and the local key-value cache broadcasted by the other nodes; Each node aggregates the first attention vector and the second attention vector between the node and other nodes to obtain a third attention vector for each node; Each node generates an output vector based on the third attention vector through the Transformer model; The master node receives the output vectors respectively sent by the plurality of slave nodes, and generates the input word element of the current layer based on its own output vector and the output vectors respectively sent by the plurality of slave nodes.

7. The data inference method according to any one of claims 1 to 5, characterized in that: The data reasoning method further includes: When the local key-value cache of the master node is occupied to the maximum capacity, allocating a new node to the data reasoning system; Move the local key-value cache of the master node to the new node, or change the master node to a slave node and use the new node as the new master node.

8. A data reasoning system, characterized in that: The data reasoning system adopts a transformer model as a reasoning model, and the data reasoning system includes a network device and a plurality of nodes, wherein the plurality of nodes include a master node and a plurality of slave nodes; The master node is used to generate a query vector, a key vector and a value vector based on an input word of a current layer of the Transformer model, and add the key vector and the value vector to a local key-value cache of the master node; and, broadcasting the query vector to the plurality of slave nodes through the network device; The multiple nodes are used to perform attention calculation based on the query vector and the local key-value caches of the multiple nodes to obtain initial attention vectors corresponding to the multiple nodes respectively; The multiple slave nodes are used to send their respective initial attention vectors to the master node through the network device; The network device is used to aggregate the initial attention vectors corresponding to the multiple slave nodes to obtain a target attention vector; The master node is also used to perform model inference based on the target attention vector through the Transformer model to obtain the output word of the Transformer model.

9. An electronic device, characterized in that: The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store a program, wherein the program includes instructions, and when the instructions are executed by the processor, the processor executes the steps of the data reasoning method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a terminal device, the steps of the data inference method described in any one of claims 1 to 7 are implemented.

Images