Data inference method, network device, medium, electronic device and program product

Abstract

The present disclosure relates to a data reasoning method, a network device, a medium, an electronic device, and a program product, and belongs to the field of computer technology. The method can solve bandwidth bottlenecks, reduce latency, and improve system performance. A data reasoning method includes: receiving an initial attention vector from multiple nodes, the initial attention vector including multiple initial elements; determining a local attention vector including multiple local elements stored locally on the network device; performing aggregation of local element i with the corresponding initial element by j-1, j, j+1, and j+2 in the pipeline stage of the network device, wherein the j-1 stage determines the weighted value of the initial element and stores it in the HPV; the j stage reads the local element i from its memory and stores it in the HPV; the j+1 stage determines the weighted value of the local element i and stores it in the HPV; the j+2 stage aggregates the local element i with the initial element based on the weighted value of the local element i and the initial element to obtain the i-th element in the target attention vector, and the target attention vector is used for data reasoning.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a data reasoning method, a network device, a medium, an electronic device, and a program product. Background Art

[0002] Large language model inference uses a trained large language model to perform inference. Users input prompts into the large language model, and the large language model outputs corresponding answers. The input and output of the large language model are collectively referred to as context.

[0003] As the context length of large language models continues to grow, the computational complexity of large language model inference increases linearly. Currently, the inference process of large language models is typically run across multiple distributed nodes, breaking down the entire inference process using distributed memory. Specifically, the context is segmented along the sequence length dimension to obtain multiple subsequences. Each distributed node is responsible for a subsequence of finite length and independently calculates the attention vector for the subsequence it is responsible for. The master node among the multiple distributed nodes then performs a full reduction on the attention vectors calculated by all distributed nodes to obtain a global attention vector. This requires the master node to obtain all attention vectors before performing the full reduction calculation. Once the calculation is complete, the calculation results are updated to all distributed nodes. This approach can easily cause computational and network bottlenecks at the master node, affecting system performance. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a data inference method, network device, medium, electronic device and program product that can solve the bandwidth bottleneck problem, reduce latency and improve system performance.

[0005] To achieve the above objectives, the present disclosure provides a data inference method, which is applied to a network device in a data inference system. The data inference system includes the network device and multiple nodes. The data inference method includes:

[0006] Receiving initial attention vectors from the multiple nodes, wherein the initial attention vectors are obtained by each of the multiple nodes performing attention calculations on the context subsequences for which they are responsible, and include multiple initial elements;

[0007] determining a local attention vector stored locally by the network device and corresponding to the initial attention vector, wherein the local attention vector includes a plurality of local elements;

[0008] The j-1th, jth, j+1th and j+2th stages of the pipeline stages of the network device for performing vector aggregation perform aggregation of the local element i and the corresponding initial element, wherein the local element i is stored in the memory of the jth stage, the j-1th stage determines the weighted value of the initial element and stores the weighted value of the initial element in the packet header vector of the network device, the jth stage reads the local element i from the memory of the stage and stores it in the packet header vector, the j+1th stage obtains the local element i from the packet header vector, determines the weighted value of the local element i and stores the weighted value of the local element i in the packet header vector, the j+2th stage aggregates the local element i and the initial element based on the weighted value of the local element i in the packet header vector and the weighted value of the initial element to obtain the i-th element in the target attention vector, wherein the target attention vector is used by the master node among the multiple nodes for data reasoning.

[0009] The present disclosure further provides a network device, which is applied to a data inference system. The data inference system includes the network device and a plurality of nodes. The network device includes:

[0010] A receiving module, configured to receive initial attention vectors from the multiple nodes, wherein the initial attention vectors are obtained by each of the multiple nodes performing attention calculations on the context subsequences for which they are responsible, and include multiple initial elements;

[0011] a determining module, configured to determine a local attention vector corresponding to the initial attention vector and stored locally by the network device, wherein the local attention vector includes a plurality of local elements;

[0012] Pipeline stages for performing vector aggregation, wherein the j-1th stage, the jth stage, the j+1th stage and the j+2th stage in the pipeline stages for performing vector aggregation perform aggregation of a local element i with a corresponding initial element, wherein the local element i is stored in the memory of the jth stage, the j-1th stage determines the weighted value of the initial element and stores the weighted value of the initial element in the packet header vector of the network device, the jth stage reads the local element i from the memory of the stage and stores it in the packet header vector, the j+1th stage obtains the local element i from the packet header vector, determines the weighted value of the local element i and stores the weighted value of the local element i in the packet header vector, the j+2th stage aggregates the local element i with the initial element based on the weighted value of the local element i in the packet header vector and the weighted value of the initial element to obtain the i-th element in the target attention vector, wherein the target attention vector is used by a master node among the multiple nodes for data reasoning.

[0013] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any method in the present disclosure when the program is executed by a processor.

[0014] The present disclosure also provides an electronic device, comprising:

[0015] a memory having a computer program stored thereon;

[0016] A processor is configured to execute a computer program in a memory to implement the steps of any method in the present disclosure.

[0017] The present disclosure also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods in the present disclosure when executed by a processor.

[0018] By adopting the above technical solution, the initial attention vectors of each node are aggregated through the network device to obtain the target attention vector. This aggregation is an online aggregation. Therefore, on the one hand, since the network device is the physical central node in the data reasoning system, its original bandwidth is relatively large and it is physically closer to other nodes. On the other hand, since online aggregation is adopted, the bandwidth bottleneck problem is solved, the latency is reduced, and the system performance is improved. In addition, since attention aggregation is performed on the network device, the node side does not need to perform attention aggregation, thereby saving storage and computing resources on the node side, and can also make full use of the high bandwidth and low latency advantages of the network device, significantly improving the efficiency of data reasoning. Furthermore, since there is no need for two-to-two communication between nodes, the bandwidth demand for data transmission is reduced. In addition, in the related art, each level in the hardware pipeline level of the RMT switch has fixed, private computing resources, and different pipeline levels cannot directly share memory data, which limits the implementation of complex operations. By adopting the above-mentioned technical solution, it can meet complex distributed computing needs, realize complex aggregation operations, support in-network computing in more scenarios, and realize cross-level pipeline state reading and writing operations, thereby improving the efficiency of aggregation.

[0019] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following detailed description, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0021] Figure 1 It is a flowchart of a data reasoning method according to an embodiment of the present disclosure.

[0022] Figure 2 It is a schematic diagram of pipeline aggregation according to an embodiment of the present disclosure.

[0023] Figure 3 2 is a schematic diagram of a changing trend of the number of loops according to an embodiment of the present disclosure.

[0024] Figure 4 FIG. 4 is a schematic diagram of batch processing of multiple vector elements in each stage according to an embodiment of the present disclosure.

[0025] Figure 5 Schematic diagram of a horizontal expansion aggregation operation according to an embodiment of the present disclosure.

[0026] Figure 6 is a schematic block diagram of a network device according to an embodiment of the present disclosure.

[0027] Figure 7 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0028] The following describes the specific embodiments of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0029] Figure 1 The data reasoning method according to an embodiment of the present disclosure is a flow chart of a data reasoning method. The data reasoning method can be applied to a network device in a data reasoning system, which includes a network device and multiple nodes.

[0030] A network device is a convergence point in a data inference system, or a physical central node. For example, in a data inference system, multiple nodes communicate through a switch, so the network device is the switch. This switch can be a programmable switch based on a pipelined reconfigurable match-action table (RMT) architecture.

[0031] like Figure 1 As shown, the data inference method may include the following steps S11 to S13.

[0032] In step S11, initial attention vectors are received from multiple nodes, wherein the initial attention vectors are obtained by each of the multiple nodes performing attention calculations on the context subsequences they are responsible for and include multiple initial elements.

[0033] Regarding attention calculation, taking the Transformer model as an example, each node performs attention calculation on its own context subsequence based on the query vector (q), the locally cached key vector (K), and the value vector (V), generating an initial attention vector. A context subsequence is a subsequence obtained by segmenting the context along its sequence length dimension. The input and output of the inference model are collectively referred to as the context.

[0034] In addition, the initial attention vector will include multiple elements, which are called initial elements.

[0035] In step S12, a local attention vector corresponding to the initial attention vector and stored locally in the network device is determined, where the local attention vector includes multiple local elements.

[0036] For example, multiple nodes can send initial attention vectors to a network device in the form of data packets, and the data packets can carry corresponding IDs. The network device can then determine the local attention vector corresponding to the initial attention vector based on the IDs carried in the data packets sent by each node.

[0037] In addition, if the local attention vector does not exist locally in the network device, the network device can directly save the first received initial attention vector locally as the local attention vector.

[0038] In addition, the local attention vector includes multiple elements, which are called local elements.

[0039] In step S13, the j-1th, jth, j+1th, and j+2th stages of the pipeline stage for performing vector aggregation of the network device perform aggregation of the local element i with the corresponding initial element, wherein the local element i is stored in the memory of the jth stage, the j-1th stage determines the weighted value of the initial element and stores the weighted value of the initial element in the packet header vector of the network device, the jth stage reads the local element i from the memory of the stage and stores it in the packet header vector, the j+1th stage obtains the local element i from the packet header vector, determines the weighted value of the local element i, and stores the weighted value of the local element i in the packet header vector, and the j+2th stage aggregates the local element i with the initial element based on the weighted value of the local element i in the packet header vector and the weighted value of the initial element to obtain the i-th element in the target attention vector. By aggregating each local element with each initial element in this manner, all elements in the target attention vector can be obtained. The target attention vector is used by the master node among the multiple nodes for data reasoning.

[0040] Figure 2It is a schematic diagram of pipeline aggregation according to an embodiment of the present disclosure. Assume that the local attention vector a={a0,a1,…,a3} and the initial attention vector a ′ ={a0',a1',…,a3'} for aggregation. The aggregation operation adopts weighted average, and the weights are w and w' respectively. Then the aggregation operation is:

[0041] a i ←wa i +w'a i ' (1)

[0042] Assume that a0 is located at s j Level, then in s j-1 The first stage obtains w'a0' by looking up the table and saves w'a0' in the packet header vector (PHV) (this operation is called pre-calculation). j Level, from s j Read a0 from the memory of the first level and save it in PHV (this operation is called extraction), and then j+1 At the s level, wa0 is obtained by looking up the table and stored in PHV (this operation is called calculation). j+2 Level, perform arithmetic summation on w'a0' and wa0 carried in PHV, and write the summation result into s j+2 The local attention vector a = {a0, a1, ..., a3} and the initial attention vector a are updated. ′ ={a0',a1',…,a3'}. In addition, through pre-calculation, extraction, calculation, and update, the vector aggregation operation of MEM_R (memory read) -> TBL_LKP (flow table lookup) -> ALU_EXE (simple arithmetic operation) -> MEM_W (memory write) is implemented.

[0043] Figure 2 In the aggregation operation, a0 is transferred from s j Level shift to s j+2 In particular, when s j-1 Level, s j Level, s j+1 Level, s j+2 When a stage in the pipeline reaches the last stage of the pipeline for performing vector aggregation in the pipeline stage of the network device, the state in the PHV can be looped back to the initial stage of the pipeline for performing vector aggregation in the pipeline stage by recirculating.init For example, if the nth to n+mth stages in the pipeline are used to perform vector aggregation, then the nth stage is the initial stage s init , where n and m are integers greater than or equal to 1. Figure 2 As shown, s j+2 The stage is the last stage of the pipeline stage for performing vector aggregation in the pipeline stage, then the update of a2 is looped back to the initial stage s of the pipeline stage for performing vector aggregation init , and finally write the updated result of a2 into s init+1 Level. Note that s init It is fixed and refers to the initial stage of the pipeline used to perform vector aggregation. Loopback is a technology used in programmable network devices (such as programmable switches) that allows packets to return and re-enter the switch pipeline for additional processing after passing through certain processing stages of the switch. This process is often used to address the problem of limited computing power or resources at each stage in the switch pipeline to implement complex network functions or operations.

[0044] By adopting the above pipeline operation, the aggregation operation of multiple elements in a single vector can be fully pipelined. Figure 2 As shown, when updating, from s j-1 Aggregate a0 from level s j Stage a1 begins aggregation... From the perspective of the pipeline stages performing vector aggregation, under full load, each stage can simultaneously perform pre-calculation, extraction, calculation, and update operations on different elements. Furthermore, in a pipelined switch, each hardware stage processes only one packet at a time, and loopbacks do not cause congestion, ensuring line-speed packet processing.

[0045] In addition, for multiple initial attention vectors, the starting level for aggregating each initial attention vector can be determined by the following method: that is, obtaining the starting level in the pipeline level for aggregating the current initial attention vector, and using the second level after the starting level as the starting level in the pipeline level for aggregating the next initial attention vector. For example, the aggregation of initial attention vector k is from level s j Starting from level s, the aggregation of the initial attention vector k+1 is j+2 start.

[0046] By adopting the above technical solution, the initial attention vectors of each node are aggregated through the network device to obtain the target attention vector. This aggregation is an online aggregation. Therefore, on the one hand, since the network device is the physical central node in the data reasoning system, its original bandwidth is relatively large and it is physically closer to other nodes. On the other hand, since online aggregation is adopted, the bandwidth bottleneck problem is solved, the latency is reduced, and the system performance is improved. In addition, since attention aggregation is performed on the network device, the node side does not need to perform attention aggregation, thereby saving storage and computing resources on the node side, and can also make full use of the high bandwidth and low latency advantages of the network device, significantly improving the efficiency of data reasoning. Furthermore, since there is no need for two-to-two communication between nodes, the bandwidth demand for data transmission is reduced. In addition, in the related art, each level in the hardware pipeline level of the RMT switch has fixed, private computing resources, and different pipeline levels cannot directly share memory data, which limits the implementation of complex operations. By adopting the above-mentioned technical solution, it can meet complex distributed computing needs, realize complex aggregation operations, support in-network computing in more scenarios, and realize cross-level pipeline state reading and writing operations, thereby improving the efficiency of aggregation.

[0047] In some embodiments, the initial attention vectors of multiple nodes can be aggregated on multiple network devices. For example, a single network device aggregates the initial attention vectors of some nodes. The data inference method according to an embodiment of the present disclosure may further include setting the number of aggregations performed on each network device based on a flat interval of the number of loopbacks.

[0048] Assume n agg Indicates the number of aggregation operations performed on the network device, n stage Indicates the number of stages used for vector aggregation in the hardware pipeline stage, n a Indicates the number of levels occupied by the vector a to be aggregated. If n a <n stage , then use the update method described above, n agg In theory, it can be infinite. But in practice, as n agg As the number of loops increases, n recircle It is not a linear growth, but a Figure 3 Therefore, when planning the number of aggregation operations performed by each network device, n agg fall into Figure 3 The flat intervals shown, especially the first stable interval, do not trigger a loop, that is, the throughput and latency of the network device will not be affected. In addition, by reducing n a , which can extend the length of the flat interval.

[0049] By adopting the above technical solution, the number of loops can be reduced, the aggregation delay can be reduced, and the aggregation performance can be improved.

[0050] In some embodiments, the data reasoning method according to an embodiment of the present disclosure may further include: aggregating the multiple local elements and the multiple initial elements in parallel in a pipeline stage for performing vector aggregation. The specific aggregation process is the same as the process described above.

[0051] That is, if each level has sufficient table lookup and logic resources, the level can support multiple parallel update operations, so vertical scalability can be achieved by batch processing multiple vector elements in each level. Figure 4 As shown, four groups of pre-calculation, extraction, calculation and update operations are performed simultaneously in each stage.

[0052] By adopting the above technical solution, it is possible to increase system capacity through vertical expansion, that is, to support the aggregation of longer vectors and achieve scalability of aggregation.

[0053] In some embodiments, the data inference method according to the embodiment of the present disclosure may further include: dividing the initial attention vector and the local attention vector into vector segments of preset lengths respectively; and sequentially aggregating the local elements and initial elements in each vector segment using a pipeline stage for performing vector aggregation.

[0054] The above embodiment is for vertical expansion, but due to the hardware resources of network equipment, the size of each batch processing level cannot be infinite. Therefore, when the length of the vector exceeds the capacity of a single pipeline, horizontal expansion can be achieved through looping.

[0055] For example, both the initial attention vector and the local attention vector can be divided into several vector segments of fixed length, each vector segment containing a number of elements. In addition, the number of vector segments into which the initial attention vector is divided is the same as the number of vector segments into which the local attention vector is divided, and the number of elements in the vector segments of the initial attention vector is the same as the number of elements in the vector segments of the local attention vector. Each stage can uniquely identify the elements in the vector by the vector ID and the vector segment index. For example, each stage in the pipeline stage used to perform vector aggregation can uniquely identify the local elements and the initial elements by the vector IDs of the initial attention vector and the local attention vector and the segment index of the vector segment, so as to ensure that the elements in the memory are read and written correctly. When performing the aggregation operation, the same as the process of the aggregation operation described above, first determine the starting level s of the first element of the 0th vector segment. start , then from s startThe pipeline stage starts to perform the aggregation of vector segment 0. If it reaches the end of the pipeline stage used to perform vector aggregation, it will loop back. After updating vector segment 0, it will immediately update vector segment 1 until all vector segments are updated. Figure 5 As shown in the figure, the network device can support vector aggregation of length 16 by only four horizontal loops. In addition, Figure 5 The assumption is s start =s init .

[0056] By adopting the above technical solution, it is possible to increase system capacity through horizontal expansion, that is, to support the aggregation of longer vectors and achieve scalability of aggregation.

[0057] Figure 6 This is a schematic block diagram of a network device according to an embodiment of the present disclosure. The network device can be applied to a data reasoning system, which includes the network device and multiple nodes.

[0058] like Figure 6 As shown, the network device 60 may include: a receiving module 61, for receiving an initial attention vector from a plurality of nodes, wherein the initial attention vector is obtained by each of the plurality of nodes performing attention calculation on the context subsequence for which they are responsible and includes a plurality of initial elements; a determining module 62, for determining a local attention vector corresponding to the initial attention vector stored locally in the network device, wherein the local attention vector includes a plurality of local elements; a pipeline stage 63 for performing vector aggregation, wherein the j-1th stage, the jth stage, the j+1th stage, and the j+2th stage of the pipeline stage for performing vector aggregation perform aggregation of the local element i with the corresponding initial element, wherein the local element i Stored in the memory of the jth level, the j-1th level determines the weighted value of the initial element and stores the weighted value of the initial element in the packet header vector of the network device, the jth level reads the local element i from the memory of the level and stores it in the packet header vector, the j+1th level obtains the local element i from the packet header vector, determines the weighted value of the local element i and stores the weighted value of the local element i in the packet header vector, the j+2th level aggregates the local element i and the initial element based on the weighted value of the local element i in the packet header vector and the weighted value of the initial element to obtain the i-th element in the target attention vector, wherein the target attention vector is used by the main node among the multiple nodes for data reasoning.

[0059] By adopting the above technical solution, the initial attention vectors of each node are aggregated through the network device to obtain the target attention vector. This aggregation is an online aggregation. Therefore, on the one hand, since the network device is the physical central node in the data reasoning system, its original bandwidth is relatively large and it is physically closer to other nodes. On the other hand, since online aggregation is adopted, the bandwidth bottleneck problem is solved, the latency is reduced, and the system performance is improved. In addition, since attention aggregation is performed on the network device, the node side does not need to perform attention aggregation, thereby saving storage and computing resources on the node side, and can also make full use of the high bandwidth and low latency advantages of the network device, significantly improving the efficiency of data reasoning. Furthermore, since there is no need for two-to-two communication between nodes, the bandwidth demand for data transmission is reduced. In addition, in the related art, each level in the hardware pipeline level of the RMT switch has fixed, private computing resources, and different pipeline levels cannot directly share memory data, which limits the implementation of complex operations. By adopting the above-mentioned technical solution, it can meet complex distributed computing needs, realize complex aggregation operations, support in-network computing in more scenarios, and realize cross-level pipeline state reading and writing operations, thereby improving the efficiency of aggregation.

[0060] Optionally, if any one of the j-1th stage, the jth stage, the j+1th stage, and the j+2th stage is the last stage of the pipeline stage for performing vector aggregation, the state in the header vector is looped back to the initial stage of the pipeline stage for performing vector aggregation in a loopback manner.

[0061] Optionally, the plurality of local elements and the plurality of original elements are aggregated in parallel in the pipeline stage 63 for performing vector aggregation.

[0062] Optionally, the pipeline stage 63 for performing vector aggregation is further configured to:

[0063] Splitting the initial attention vector and the local attention vector into vector segments of preset lengths respectively;

[0064] The local elements and the original elements in each vector segment are sequentially aggregated using the pipeline stage for performing vector aggregation.

[0065] Optionally, each stage in the pipeline stage for performing vector aggregation uniquely determines the local element and the initial element through the vector IDs of the initial attention vector and the local attention vector and the segment index of the vector segment.

[0066] Optionally, the initial attention vector is aggregated on a plurality of the network devices, and the network device 60 further includes a setting module for:

[0067] The number of times the aggregation is performed on each of the network devices is set according to a flat interval of the number of loopbacks.

[0068] The specific implementation of the operations performed by each module in the network device according to the embodiment of the present disclosure has been described in detail in the relevant method and will not be repeated here.

[0069] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any method in the present disclosure when the program is executed by a processor.

[0070] The present disclosure also provides an electronic device, comprising:

[0071] a memory having a computer program stored thereon;

[0072] A processor is configured to execute a computer program in a memory to implement the steps of any method in the present disclosure.

[0073] The present disclosure also provides a computer program product, comprising a computer program, which implements the steps of any one of the methods in the present disclosure when executed by a processor.

[0074] Figure 7 FIG. 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. Figure 7 As shown, the electronic device 700 may include: a processor 701 , a memory 702 , and may further include one or more of a multimedia component 703 , an input / output (I / O) interface 704 , and a communication component 705 .

[0075] The processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned data reasoning method. The memory 702 is used to store various types of data to support the operation of the electronic device 700. For example, this data may include instructions for any application or method operating on the electronic device 700, as well as application-related data, such as contact information, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 702 or sent through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules, and the above-mentioned other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0076] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the above-mentioned data inference method.

[0077] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When executed by a processor, the program instructions implement the steps of the aforementioned data reasoning method. For example, the computer-readable storage medium may be the aforementioned memory 702 including the program instructions. The program instructions may be executed by the processor 701 of the electronic device 700 to perform the aforementioned data reasoning method.

[0078] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a processor. When the computer program is executed by the processor, the steps of the above-mentioned data reasoning method are implemented.

[0079] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the scope of protection of the present disclosure.

[0080] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0081] In addition, the various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A data reasoning method, characterized in that: A network device applied to a data inference system, wherein the data inference system comprises the network device and a plurality of nodes, and the data inference method comprises: Receiving initial attention vectors from the multiple nodes, wherein the initial attention vectors are obtained by each of the multiple nodes performing attention calculations on the context subsequences for which they are responsible, and include multiple initial elements; determining a local attention vector stored locally by the network device and corresponding to the initial attention vector, wherein the local attention vector includes a plurality of local elements; The j-1th, jth, j+1th and j+2th stages of the pipeline stages of the network device for performing vector aggregation perform aggregation of the local element i and the corresponding initial element, wherein the local element i is stored in the memory of the jth stage, the j-1th stage determines the weighted value of the initial element and stores the weighted value of the initial element in the packet header vector of the network device, the jth stage reads the local element i from the memory of the stage and stores it in the packet header vector, the j+1th stage obtains the local element i from the packet header vector, determines the weighted value of the local element i and stores the weighted value of the local element i in the packet header vector, the j+2th stage aggregates the local element i and the initial element based on the weighted value of the local element i in the packet header vector and the weighted value of the initial element to obtain the i-th element in the target attention vector, wherein the target attention vector is used by the master node among the multiple nodes for data reasoning.

2. The method according to claim 1, characterized in that The method further comprises: If any one of the j-1th stage, the jth stage, the j+1th stage, and the j+2th stage is the last stage of the pipeline stage for performing vector aggregation, the state in the packet header vector is looped back to the initial stage of the pipeline stage for performing vector aggregation in a loopback manner.

3. The method according to claim 1, characterized in that The method further comprises: Aggregating the plurality of local elements and the plurality of original elements in parallel in the pipeline stage for performing vector aggregation.

4. The method according to claim 1, wherein The method further comprises: Splitting the initial attention vector and the local attention vector into vector segments of preset lengths respectively; The local elements and the original elements in each vector segment are sequentially aggregated using the pipeline stage for performing vector aggregation.

5. The method according to claim 4, characterized in that The method further comprises: Each stage in the pipeline stage for performing vector aggregation uniquely identifies the local element and the initial element through the vector IDs of the initial attention vector and the local attention vector and the segment index of the vector segment.

6. The method according to claim 1, characterized in that Aggregating the initial attention vectors on a plurality of the network devices, the method further comprises: The number of times the aggregation is performed on each of the network devices is set according to a flat interval of the number of loopbacks.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Obtain the starting stage in the pipeline stage for aggregating the current initial attention vector, and use the second stage after the starting stage as the starting stage in the pipeline stage for aggregating the next initial attention vector.

8. A network device, characterized in that: Applied to a data inference system, the data inference system includes the network device and multiple nodes, the network device includes: A receiving module, configured to receive initial attention vectors from the multiple nodes, wherein the initial attention vectors are obtained by each of the multiple nodes performing attention calculations on the context subsequences for which they are responsible, and include multiple initial elements; a determining module, configured to determine a local attention vector corresponding to the initial attention vector and stored locally by the network device, wherein the local attention vector includes a plurality of local elements; Pipeline stages for performing vector aggregation, wherein the j-1th stage, the jth stage, the j+1th stage and the j+2th stage in the pipeline stages for performing vector aggregation perform aggregation of a local element i with a corresponding initial element, wherein the local element i is stored in the memory of the jth stage, the j-1th stage determines the weighted value of the initial element and stores the weighted value of the initial element in the packet header vector of the network device, the jth stage reads the local element i from the memory of the stage and stores it in the packet header vector, the j+1th stage obtains the local element i from the packet header vector, determines the weighted value of the local element i and stores the weighted value of the local element i in the packet header vector, the j+2th stage aggregates the local element i with the initial element based on the weighted value of the local element i in the packet header vector and the weighted value of the initial element to obtain the i-th element in the target attention vector, wherein the target attention vector is used by a master node among the multiple nodes for data reasoning.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images