Data reasoning method, network device, medium, electronic device and program product

Abstract

The invention relates to a data reasoning method, network equipment, a medium, electronic equipment and a program product, belongs to the technical field of computers, and can solve the bandwidth bottleneck problem, reduce time delay and improve system performance. The data reasoning method is applied to network equipment in a data reasoning system, and comprises the following steps: receiving an initial attention vector and an initial scaling factor from a plurality of nodes of the data reasoning system; aiming at each initial attention vector and each initial scaling factor, determining a corresponding local attention vector and a local scaling factor, determining the maximum value of the initial scaling factor and the local scaling factor and the absolute value of the difference value between the initial scaling factor and the local scaling factor, and if the absolute value of the difference value is smaller than or equal to a preset value, executing the step 2; if yes, the local scaling factor is updated based on the absolute value of the difference value and the maximum value, the local attention vector is updated based on the initial attention vector, the local attention vector and the absolute value of the difference value to conduct vector aggregation to obtain a target attention vector, and the target attention vector is used for data reasoning by a main node in the multiple nodes.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a data reasoning method, a network device, a medium, an electronic device, and a program product. Background Art

[0002] Large language model inference refers to the process of using a trained large language model for inference, where the user inputs prompt information into the large language model, and the large language model outputs the answer information corresponding to the prompt information. The input and output of the large language model are collectively referred to as context.

[0003] As the context length of a large language model continues to grow, the amount of computation required for reasoning with a large language model will increase linearly. At present, the reasoning process of a large language model is usually run between multiple distributed nodes, and the entire reasoning process is decomposed in a distributed memory manner. Specifically, the context sequence length dimension is divided to obtain multiple subsequences. Each distributed node is responsible for a subsequence of a finite length, and each node independently calculates the attention vector of the subsequence it is responsible for. Then, the master node among the multiple distributed nodes performs a full reduction on the attention vectors calculated by all distributed nodes to obtain a global attention vector. This requires the master node to obtain all attention vectors before performing a full reduction calculation. After the calculation is completed, the calculation results are updated to all distributed nodes. This method is likely to cause calculation and network bottlenecks at the master node, affecting system performance. Summary of the invention

[0004] The purpose of the present disclosure is to provide a data reasoning method, network equipment, medium, electronic equipment and program product, which can solve the bandwidth bottleneck problem, reduce latency and improve system performance.

[0005] In order to achieve the above object, the present disclosure provides a data reasoning method, which is applied to a network device in a data reasoning system, wherein the data reasoning system includes the network device and a plurality of nodes, and the data reasoning method includes:

[0006] Receiving initial attention vectors and initial scaling factors from the multiple nodes, wherein the initial attention vectors and the initial scaling factors are obtained by each of the multiple nodes performing attention calculations on the context subsequences for which they are responsible;

[0007] Initial attention vector and initial scaling factor for each node:

[0008] Determining a local attention vector and a local scaling factor corresponding to the initial attention vector and the initial scaling factor stored locally by the network device;

[0009] Determine a maximum value of the initial scaling factor and the local scaling factor and an absolute value of a difference between the initial scaling factor and the local scaling factor;

[0010] If the absolute value of the difference is less than or equal to a preset value, the local scaling factor is updated based on the absolute value of the difference and the maximum value, and the local attention vector is updated based on the initial attention vector, the local attention vector and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes respectively to obtain a target attention vector, wherein the target attention vector is used by a main node among the multiple nodes to perform data inference to obtain an inference result.

[0011] The present disclosure further provides a network device, which is applied to a data reasoning system. The data reasoning system includes the network device and a plurality of nodes. The network device includes:

[0012] A receiving module, configured to receive an initial attention vector and an initial scaling factor from the multiple nodes, wherein the initial attention vector and the initial scaling factor are obtained by each of the multiple nodes performing attention calculation on the context subsequence for which it is responsible;

[0013] Aggregation module for the initial attention vector and initial scaling factor for each node:

[0014] Determining a local attention vector and a local scaling factor corresponding to the initial attention vector and the initial scaling factor stored locally by the network device;

[0015] Determine a maximum value of the initial scaling factor and the local scaling factor and an absolute value of a difference between the initial scaling factor and the local scaling factor;

[0016] If the absolute value of the difference is less than or equal to a preset value, the local scaling factor is updated based on the absolute value of the difference and the maximum value, and the local attention vector is updated based on the initial attention vector, the local attention vector and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes respectively to obtain a target attention vector, wherein the target attention vector is used by a main node among the multiple nodes to perform data inference to obtain an inference result.

[0017] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any method described in the present disclosure when the program is executed by a processor.

[0018] The present disclosure also provides an electronic device, comprising:

[0019] a memory having a computer program stored thereon;

[0020] A processor is used to execute the computer program in the memory to implement the steps of any method described in the present disclosure.

[0021] The present disclosure also provides a computer program product, comprising a computer program, which implements the steps of any method described in the present disclosure when the computer program is executed by a processor.

[0022] By adopting the above technical solution, the initial attention vectors of each node are aggregated through the network device to obtain the target attention vector. This aggregation is an online aggregation. Therefore, on the one hand, since the network device is the physical central node in the data reasoning system, its original bandwidth is relatively large and it is physically closer to other nodes. On the other hand, since online aggregation is adopted, the bandwidth bottleneck problem is solved, the latency is reduced, and the system performance is improved. In addition, since the attention aggregation is performed on the network device, the node side does not need to perform attention aggregation, thereby saving storage and computing resources on the node side. Furthermore, since there is no need for two-to-two communication between nodes, the bandwidth requirement for data transmission is reduced.

[0023] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0025] Figure 1 It is a flow chart of a data reasoning method according to an embodiment of the present disclosure.

[0026] Figure 2 It is a schematic diagram of multiple nodes transmitting initial attention vectors and initial scaling factors to a network device according to an embodiment of the present disclosure.

[0027] Figure 3a It is a schematic diagram of updating the local scaling factor by looking up the first table according to an embodiment of the present disclosure.

[0028] Figure 3b and 3c They are schematic diagrams of the second table and the third table according to embodiments of the present disclosure.

[0029] Figure 3d is a schematic diagram of a fourth table according to an embodiment of the present disclosure.

[0030] Figure 3e and 3f They are schematic diagrams of the fifth table and the sixth table according to embodiments of the present disclosure.

[0031] Figure 4 is a schematic block diagram of a network device according to an embodiment of the present disclosure.

[0032] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0033] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0034] Figure 1 The data inference method is applied to a network device in a data inference system, and the data inference system includes a network device and multiple nodes.

[0035] A network device refers to a convergence point in a data inference system, or it can also be called a physical central node. For example, in a data inference system, multiple nodes communicate through switches, so the network device is the switch. For another example, if the data inference system is implemented using multi-chip interconnection, then the network device is the input / output (IO) chip in the multi-chip interconnection.

[0036] like Figure 1 As shown, the data reasoning method according to the embodiment of the present disclosure may include the following steps S11 to S14.

[0037] In step S11, an initial attention vector a′ and an initial scaling factor γ′ are received from multiple nodes, wherein the initial attention vector a′ and the initial scaling factor γ′ are obtained by each of the multiple nodes performing attention calculations on the context subsequences they are responsible for.

[0038] For attention calculation, taking the data inference system using the Transformer model as the inference model as an example, each node will perform attention calculation on the context subsequence it is responsible for based on the query vector (q), the locally cached key vector (K) and the value vector (V), and obtain the initial attention vector a′ and the initial scaling factor γ′. The context subsequence refers to the subsequence obtained by splitting along the sequence length dimension of the context. The input and output of the inference model are collectively referred to as the context. The initial scaling factor γ′ can be various defined scaling factors. For example, the initial scaling factor γ′ can be an additional scalar for the initial attention vector a′: γ′=log∑e qK .

[0039] Figure 2FIG. 1 is a schematic diagram of multiple nodes transmitting initial attention vectors and initial scaling factors to a network device according to an embodiment of the present disclosure. Figure 2 As shown, multiple nodes 11 1 to 11 n The respective initial attention vectors and initial scaling factors are sent to the network device 10 respectively.

[0040] In step S12, for each node's initial attention vector a′ and initial scaling factor γ′, a local attention vector a′ corresponding to the initial attention vector a′ and initial scaling factor γ′ stored locally in the network device is determined. l and the local scaling factor γ l .

[0041] For example, multiple nodes can send the initial attention vector a′ and the initial scaling factor γ′ to the network device in the form of a data packet, and the data packet can carry the corresponding ID. The network device can determine the local attention vector a′ corresponding to the initial attention vector a′ and the initial scaling factor γ′ according to the ID carried in the data packet sent by each node. l and the local scaling factor γ l .

[0042] In addition, if the network device does not have a local attention vector a before l and the local scaling factor γ l , then the network device can directly save the first received initial attention vector a′ and initial scaling factor γ′ locally as the local attention vector a l and the local scaling factor γ l .

[0043] In step S13, the initial scaling factor γ′ and the local scaling factor γ are determined. l The maximum value of γ max As well as the initial scaling factor γ′ and the local scaling factor γ l The absolute value of the difference in,

[0044] In step S14, if the absolute value of the difference If it is less than or equal to the preset value, then it is based on the absolute value of the difference and the maximum value γ max Update local scaling factor γ l , and based on the initial attention vector a′, the local attention vector a l and the absolute value of the difference Update the local attention vector a l Aggregate the initial attention vectors a′ corresponding to multiple nodes to obtain the target attention vector a目标 , where the target attention vector a 目标 It is used by the master node among multiple nodes to perform data reasoning to obtain reasoning results.

[0045] For example, the network device receives the initial attention vector a′ and the initial scaling factor γ′ from n nodes, and the network device first determines the initial scaling factor γ′ of the first node and the local scaling factor γ of the network device. l The maximum value of γ max As well as the initial scaling factor γ′ of the first node and the local scaling factor γ of the network device l The absolute value of the difference If the absolute value of the difference If the value is less than or equal to the preset value, the network device will and the maximum value γ max Update local scaling factor γ l , and based on the initial attention vector a′ of the first node and the local attention vector a of the network device l and the absolute value of the difference Update the local attention vector a l In this way, the aggregation with the initial attention vector a′ of the first node is realized. Then, for the initial scaling factor γ′ and the initial attention vector a′ of the second node, the network device adopts the same processing method as the initial scaling factor γ′ and the initial attention vector a′ of the first node, and the cycle is repeated until the aggregation of the initial attention vectors a′ of n nodes is completed, and the target attention vector a is obtained. 目标 .

[0046] In addition, if the absolute value of the difference is greater than the preset value, which means that the smaller scaling factor has little effect on the larger scaling factor (for example, less than one thousandth), so the smaller scaling factor can be ignored. Then, the network device can use the maximum value γ max The corresponding scaling factor is used to update the local scaling factor γ l , and use the maximum value of γ max The corresponding attention vector is used to update the local attention vector a l For example, in the absolute value of the difference If the initial scaling factor γ′ is greater than the local scaling factor γ l , the network device replaces the local scaling factor γ with the initial scaling factor γ′ l , replace the local attention vector a with the initial attention vector a′ l , if the initial scaling factor γ′ is smaller than the local scaling factor γ l , then the network device maintains the local scaling factor γ land the local attention vector a l constant.

[0047] In addition, the target attention vector is the result of aggregating the initial attention vectors of all nodes in the data reasoning system. For example, the data reasoning system includes multiple network devices, each of which is responsible for aggregating the initial attention vectors of some of the multiple nodes to obtain an intermediate attention vector, and then one of the multiple network devices aggregates each intermediate attention vector to obtain a target attention vector, which is used by the master node among the multiple nodes to perform data reasoning to obtain the reasoning result.

[0048] By adopting the above technical solution, the initial attention vectors of each node are aggregated through the network device to obtain the target attention vector. This aggregation is an online aggregation. Therefore, on the one hand, since the network device is the physical central node in the data reasoning system, its original bandwidth is relatively large and it is physically closer to other nodes. On the other hand, since online aggregation is adopted, the bandwidth bottleneck problem is solved, the latency is reduced, and the system performance is improved. In addition, since the attention aggregation is performed on the network device, the node side does not need to perform attention aggregation, thereby saving storage and computing resources on the node side. Furthermore, since there is no need for two-to-two communication between nodes, the bandwidth requirement for data transmission is reduced.

[0049] In some embodiments, the local scaling factor may be updated by table lookup. That is, updating the local scaling factor based on the absolute value and the maximum value of the difference in step S14 may be implemented in the following manner.

[0050] First, according to the absolute value of the difference, the logarithmic function value of the absolute value of the difference is searched from the first table, wherein the first table records the corresponding relationship between each range of the absolute value of the difference and the logarithmic function value.

[0051] The local scaling factor is then updated by adding the found logarithmic function value to the maximum value.

[0052] For example, suppose the local scaling factor is updated using the following function: Since network devices generally do not have sufficient computing power and memory space, they cannot directly solve the above function. To this end, the data inference method according to the embodiment of the present disclosure adopts a table lookup method. The first table can be obtained in the following way. First, according to the minimum weight requirement, determine the absolute value of the difference For example, assuming the minimum weight w is 0.004, the absolute value of the difference is determined based on this. Should not exceed 8, then the absolute value of the difference The value range of is 0 to 8. Then, the absolute value of the difference The value range of is divided into multiple sub-value ranges. Each sub-value range corresponds to a different logarithmic function value of the absolute value of the difference, that is, The values ​​of are different. In the same sub-value range, different absolute values ​​of the difference The logarithmic function value corresponding to the same absolute value of the difference. In this way, a first table recording the correspondence between each range of the absolute value of the difference and the logarithmic function value can be obtained. By using the first table for table lookup, the update of the local scaling factor can be achieved. The schematic diagram of updating the local scaling factor by looking up the first table is shown in FIG. Figure 3a shown.

[0053] In addition, to improve accuracy, the absolute value of the difference In the form of fixed-point numbers.

[0054] By adopting the above technical solution, it is possible to use a table lookup method to replace complex operations, so that attention vector aggregation can be achieved on network devices with insufficient computing power and memory space, reducing the hardware resource requirements and hardware complexity of network devices, and avoiding numerical overflow and excessive resource overhead. For example, when low-bit quantization (such as 8 bits) is used, even for binary functions, it is not necessary to maintain too large a table, so it will not consume too much memory resources.

[0055] In some embodiments, the local attention vector may be updated by table lookup. That is, updating the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference described in step S14 may be implemented in the following manner.

[0056] First, based on the size of the initial scaling factor and the local scaling factor, a table for the initial attention vector and a table for the local attention vector are selected from the second table and the third table, wherein the second table records the correspondence between the attention vector, the absolute value of the difference and the first weighted value, and the third table records the correspondence between the attention vector, the absolute value of the difference and the second weighted value.

[0057] For example, suppose that the local attention vector uses function a l =wa 1 +(1-w)a 2 Update, where a 1 and a 2 represents the attention vector; the weight of the attention vector corresponding to a larger scaling factor is w, and the weight of the attention vector corresponding to a smaller scaling factor is (1-w). For example, if the initial scaling factor γ′ is greater than the local scaling factor γl , then the above function is a l =wa′+(1-w)a l , if the initial scaling factor γ′ is smaller than the local scaling factor γ l , then the above function is a l =wa l +(1-w)a′; the first weighted value in the second table is a function The second weighted value in the third table is a function of The value of , where a represents the attention vector. Then, if the initial scaling factor γ′ is greater than the local scaling factor γ l , then the second table is the table for the initial attention vector a′, and the third table is the table for the local attention vector a l If the initial scaling factor γ′ is less than the local scaling factor γ l , then the third table is the table for the initial attention vector a′, and the second table is the table for the local attention vector a l Table of contents. Figure 3b and 3c They are schematic diagrams of the second table and the third table according to embodiments of the present disclosure.

[0058] Then, based on the initial attention vector and the absolute value of the difference, the weighted value for the initial attention vector is looked up from the table for the initial attention vector, and based on the local attention vector and the absolute value of the difference, the weighted value for the local attention vector is looked up from the table for the local attention vector.

[0059] Then, the local attention vector is updated according to the weighted value for the initial attention vector and the weighted value for the local attention vector.

[0060] In this way, the local attention vector can be updated by looking up the table. It can be in the form of a fixed-point number. The more bits the fixed-point number has, the higher the accuracy of the updated local attention vector will be, but the size of the table (such as the second table and the third table) will also be larger. Therefore, a compromise can be made between the size of the table and the accuracy of the updated local attention vector.

[0061] By adopting the above technical solution, it is possible to use a table lookup method to replace complex operations, so that attention vector aggregation can be achieved on network devices with insufficient computing power and memory space, reducing the hardware resource requirements and hardware complexity of network devices, and avoiding numerical overflow and excessive resource overhead. For example, when low-bit quantization (such as 8 bits) is used, even for binary functions, it is not necessary to maintain too large a table, so it will not consume too much memory resources.

[0062] In some embodiments, the local attention vector may be updated by table lookup. That is, updating the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference described in step S14 may be implemented in the following manner.

[0063] First, the weight w corresponding to the absolute value of the difference is searched from the fourth table, wherein the fourth table records the corresponding relationship between each range of the absolute value of the difference and the weight w.

[0064] For example, suppose that the local attention vector uses function a l =wa 1 +(1-w)a 2 Update, where a 1 and a 2 represents the attention vector. Then the fourth table can be obtained as follows. First, according to the minimum weight requirement, determine the absolute value of the difference For example, assuming the minimum weight w is 0.004, the absolute value of the difference is determined based on this. Should not exceed 8, then the absolute value of the difference The value range of is 0 to 8. Then, the absolute value of the difference The value range of is divided into multiple sub-value ranges. Each sub-value range corresponds to a different weight w value, that is, The values ​​of are different. In the same sub-value range, different absolute values ​​of the difference Corresponding to the same weight w value. In this way, a fourth table recording the corresponding relationship between each range of the absolute value of the difference and the weight value can be obtained. Figure 3d is a schematic diagram of a fourth table according to an embodiment of the present disclosure.

[0065] In addition, to improve accuracy, the absolute value of the difference It can be in the form of fixed-point numbers. For example, the absolute value of the difference can be in the form of fixed-point numbers. The first m bits of are used as the index of the fourth table. Assuming m is 4, the index can be recorded as

[0066] Then, based on the size of the initial scaling factor and the local scaling factor, a table for the initial attention vector and a table for the local attention vector are selected from the fifth table and the sixth table, wherein the fifth table records the correspondence between the attention vector, the weight and the third weighted value, and the sixth table records the correspondence between the attention vector, the weight and the fourth weighted value.

[0067] For example, suppose that the local attention vector uses function a l =wa 1 +(1-w)a2 Update, where a 1 and a 2 represents the attention vector; the weight of the attention vector corresponding to a larger scaling factor is w, and the weight of the attention vector corresponding to a smaller scaling factor is (1-w). For example, if the initial scaling factor γ′ is greater than the local scaling factor γ l , then the above function is a l =wa′+(1-w)a l , if the initial scaling factor γ′ is smaller than the local scaling factor γ l , then the above function is a l =wa l +(1-w)a′; the third weighted value in the fifth table is the value of the function wa, and the fourth weighted value in the sixth table is the value of the function (1-w)a, where a represents the attention vector. Then, if the initial scaling factor γ′ is greater than the local scaling factor γ l , then the fifth table is the table for the initial attention vector a′, and the sixth table is the table for the local attention vector a l If the initial scaling factor γ′ is less than the local scaling factor γ l , then the sixth table is the table for the initial attention vector a′, and the fifth table is the table for the local attention vector a l Table of contents. Figure 3e and 3f They are schematic diagrams of the fifth table and the sixth table according to embodiments of the present disclosure.

[0068] Then, based on the initial attention vector and the found weight, the weighted value for the initial attention vector is looked up from the table for the initial attention vector, and based on the local attention vector and the found weight, the weighted value for the local attention vector is looked up from the table for the local attention vector.

[0069] Then, the local attention vector is updated according to the weighted value for the initial attention vector and the weighted value for the local attention vector.

[0070] In this way, the local attention vector can be updated by looking up the table. It can be in the form of a fixed-point number. The more bits the fixed-point number has, the higher the accuracy of the updated local attention vector will be, but the size of the table (such as the fifth table and the sixth table) will also be larger. Therefore, a compromise can be made between the size of the table and the accuracy of the updated local attention vector.

[0071] By adopting the above technical solution, it is possible to use a table lookup method to replace complex operations, so that attention vector aggregation can be achieved on network devices with insufficient computing power and memory space, reducing the hardware resource requirements and hardware complexity of network devices, and avoiding numerical overflow and excessive resource overhead. For example, when low-bit quantization (for example, 8 bits) is used, even for binary functions, it is not necessary to maintain too large a table, so it will not consume too much memory resources.

[0072] In some embodiments, in addition to updating the local scaling factor by table lookup, the local scaling factor can also be updated by fixed-point multiplication. That is, updating the local scaling factor based on the absolute value and the maximum value of the difference in step S14 can be implemented as follows.

[0073] First, the absolute value range of the difference in which the absolute value of the difference lies is determined.

[0074] For example, suppose the local scaling factor is the function Update. Since network devices generally do not have sufficient computing power and memory space, they cannot directly solve the above function. Therefore, the above function can be converted into a linear function in the following way. First, let in, is the integer part of the absolute value of the difference, Direct shift operation can be performed. is the fractional part of the absolute value of the difference, It is essentially an exponential function with values ​​between (0,1), which can be approximated by multi-segment linear functions. For example, the range of the absolute value of the difference is divided into several sub-ranges, each of which corresponds to a linear function of the fractional part of the absolute value of the difference, for example, The slope k of the linear function corresponding to each sub-range 1 is different, the intercept b of the linear function corresponding to each sub-range 1 The determination method of the value range of the absolute value of the difference has been described in detail above and will not be repeated here. In addition, determining the absolute value range of the difference in which the absolute value of the difference is located is to determine the sub-value range in which the absolute value of the difference is located.

[0075] Then, based on the linear function corresponding to the absolute value range of the difference, the linear function value of the fractional part of the absolute value of the difference is determined, and a shift operation is performed on the integer part of the absolute value of the difference to obtain a shift operation result.

[0076] Then, based on the linear function value, the shift operation result and the maximum value, the local scaling factor is updated.

[0077] Use the function with the local scaling factor Update, and For example, we can calculate the product of the linear function value and the shift operation result, and use Taylor expansion to calculate log(1+x), and then use the Taylor expansion value and γ max Add them together to get the updated local scaling factor.

[0078] By adopting the above technical solution, fixed-point multiplication can be used to replace complex calculations, so that attention vector aggregation can be achieved on network devices with insufficient computing power and memory space, reducing the hardware resource requirements and hardware complexity of network devices, and avoiding problems such as numerical overflow and excessive resource overhead.

[0079] In some embodiments, in addition to updating the local attention vector by table lookup, the local attention vector can also be updated by fixed-point multiplication. That is, updating the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference described in step S14 can be implemented as follows.

[0080] First, the absolute value range of the difference in which the absolute value of the difference lies is determined.

[0081] For example, suppose the local attention vector adopts the function a k =wa 1 +(1-w)a 2 Update, where the weight a 1 and a 2 Represents the attention vector. Since network devices generally do not have sufficient computing power and memory space, they cannot directly solve the above function. Therefore, the following method can be used to convert First, in order to achieve a more stable accuracy (resolution), the functions can be grouped evenly according to the range of w. For example, w is reduced by a set value (such as 0.1) as a fixed step size, and The value range of is divided into several sub-value ranges. The determination method of the value range of has been described in detail above and will not be repeated here. A linear approximation is performed on the w variation curve of each sub-value range, for example, where k 2 and b 2 Represent the slope and intercept of the linear function respectively. When calculating, the absolute value of the difference is compared with the endpoints of each sub-range in parallel to determine the absolute value range of the difference. Therefore, only the endpoint values ​​of each sub-range and the slope and intercept values ​​of each linear function need to be maintained, which greatly reduces memory overhead.

[0082] Then, according to the absolute value of the difference, the weight is determined based on a linear function corresponding to the absolute value range of the difference.

[0083] After determining the absolute value range of the difference, the linear function corresponding to the absolute value range of the difference can be known. According to the linear function, the corresponding weight can be calculated.

[0084] Then, based on the initial attention vector, the local attention vector and the determined weight, the local attention vector is updated.

[0085] For example, assuming that the weight of the attention vector corresponding to the larger scaling factor is w, and the weight of the attention vector corresponding to the smaller scaling factor is (1-w), then the initial scaling factor γ′ and the local scaling factor γ l The size of the initial attention vector determines whether the weight is w or (1-w) and whether the weight of the local attention vector is w or (1-w). Then, the function a can be used l =wa 1 +(1-w)a 2 Update the local attention vector.

[0086] By adopting the above technical solution, it is possible to use fixed-point multiplication to replace complex calculations. Therefore, on the one hand, it is possible to realize the aggregation of attention vectors on network devices with insufficient computing power and memory space, and on the other hand, it reduces the hardware resource requirements and hardware complexity of network devices. On the other hand, it can compress the memory resources occupied by the lookup table, and can use higher-precision data types (such as 16 bits) for aggregation to improve accuracy, which can avoid problems such as numerical overflow and excessive resource overhead.

[0087] Figure 4 It is a schematic block diagram of a network device according to an embodiment of the present disclosure. The network device can be applied to a data reasoning system, which includes a network device and multiple nodes. The network device refers to a convergence point in a data reasoning system, or can also be called a physical central node. For example, in a data reasoning system, multiple nodes communicate through a switch, and the network device is the switch. For another example, if the data reasoning system is implemented using a multi-chip interconnect, the network device is an input / output (IO) chip in the multi-chip interconnect.

[0088] like Figure 4As shown, the network device 30 includes: a receiving module 31, which is used to receive an initial attention vector and an initial scaling factor from multiple nodes, wherein the initial attention vector and the initial scaling factor are obtained by each of the multiple nodes performing attention calculations on the context subsequences they are responsible for; an aggregation module 32, which is used to determine, for the initial attention vector and the initial scaling factor of each node, a local attention vector and a local scaling factor corresponding to the initial attention vector and the initial scaling factor stored locally in the network device, determine the maximum value of the initial scaling factor and the local scaling factor and the absolute value of the difference between the initial scaling factor and the local scaling factor, if the absolute value of the difference is less than or equal to a preset value, update the local scaling factor based on the absolute value of the difference and the maximum value, and update the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes respectively, and obtain a target attention vector, wherein the target attention vector is used by a master node among the multiple nodes to perform data inference to obtain an inference result.

[0089] By adopting the above technical solution, the initial attention vectors of each node are aggregated through the network device to obtain the target attention vector. This aggregation is an online aggregation. Therefore, on the one hand, since the network device is the physical central node in the data reasoning system, its original bandwidth is relatively large and it is physically closer to other nodes. On the other hand, since online aggregation is adopted, the bandwidth bottleneck problem is solved, the latency is reduced, and the system performance is improved. In addition, since the attention aggregation is performed on the network device, the node side does not need to perform attention aggregation, thereby saving storage and computing resources on the node side. Furthermore, since there is no need for two-to-two communication between nodes, the bandwidth requirement for data transmission is reduced.

[0090] Optionally, the aggregation module 32 is further used to: if the absolute value of the difference is greater than a preset value, use the scaling factor corresponding to the maximum value to update the local scaling factor, and use the attention vector corresponding to the maximum value to update the local attention vector.

[0091] Optionally, the aggregation module 32 updates the local scaling factor based on the absolute value and the maximum value of the difference, including:

[0092] According to the absolute value of the difference, searching for the logarithmic function value of the absolute value of the difference from the first table, wherein the first table records the corresponding relationship between each range of the absolute value of the difference and the logarithmic function value;

[0093] Update the local scaling factor by adding the found logarithmic function value to the maximum value.

[0094] Optionally, the aggregation module 32 updates the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference, including:

[0095] Based on the size of the initial scaling factor and the local scaling factor, selecting a table for the initial attention vector and a table for the local attention vector from the second table and the third table, wherein the second table records the corresponding relationship between the attention vector, the absolute value of the difference, and the first weighted value, and the third table records the corresponding relationship between the attention vector, the absolute value of the difference, and the second weighted value;

[0096] According to the initial attention vector and the absolute value of the difference, find a weighted value for the initial attention vector from a table for the initial attention vector;

[0097] According to the local attention vector and the absolute value of the difference, find a weighted value for the local attention vector from a table for the local attention vector;

[0098] The local attention vector is updated according to the weighted value for the initial attention vector and the weighted value for the local attention vector.

[0099] Optionally, the aggregation module 32 updates the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference, including:

[0100] Find the weight corresponding to the absolute value of the difference from the fourth table, wherein the fourth table records the corresponding relationship between each range of the absolute value of the difference and the weight;

[0101] Based on the size of the initial scaling factor and the local scaling factor, selecting a table for the initial attention vector and a table for the local attention vector from a fifth table and a sixth table, wherein the fifth table records the corresponding relationship between the attention vector, the weight, and the third weighted value, and the sixth table records the corresponding relationship between the attention vector, the weight, and the fourth weighted value;

[0102] According to the initial attention vector and the found weight, find a weighted value for the initial attention vector from a table for the initial attention vector;

[0103] According to the local attention vector and the found weight, find a weighted value for the local attention vector from a table for the local attention vector;

[0104] The local attention vector is updated according to the weighted value for the initial attention vector and the weighted value for the local attention vector.

[0105] Optionally, the aggregation module 32 updates the local scaling factor based on the absolute value and the maximum value of the difference, including:

[0106] Determine the absolute value range of the difference in which the absolute value of the difference lies;

[0107] Determine the linear function value of the fractional part of the absolute value of the difference based on the linear function corresponding to the absolute value range of the difference;

[0108] Perform a shift operation on the integer part of the absolute value of the difference to obtain a shift operation result;

[0109] Update the local scaling factor based on the linear function value, the shift operation result and the maximum value.

[0110] Optionally, the aggregation module 32 updates the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference, including:

[0111] Determine the absolute value range of the difference in which the absolute value of the difference lies;

[0112] Determine the weight according to the absolute value of the difference and the linear function corresponding to the absolute value range of the difference;

[0113] Based on the initial attention vector, the local attention vector, and the determined weight, the local attention vector is updated.

[0114] Optionally, the absolute value of the difference is a fixed-point number.

[0115] Regarding the network device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0116] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any method in the present disclosure when the program is executed by a processor.

[0117] The present disclosure also provides an electronic device, comprising:

[0118] a memory having a computer program stored thereon;

[0119] A processor is used to execute a computer program in a memory to implement the steps of any method in the present disclosure.

[0120] The present disclosure also provides a computer program product, comprising a computer program, which implements the steps of any method in the present disclosure when executed by a processor.

[0121] Figure 5 FIG. 7 is a block diagram of an electronic device 700 according to an exemplary embodiment. Figure 5 As shown, the electronic device 700 may include: a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0122] The processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above-mentioned data reasoning method. The memory 702 is used to store various types of data to support the operation of the electronic device 700. For example, these data may include instructions for any application or method used to operate on the electronic device 700, and application-related data, such as contact data, messages sent and received, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 702 or sent via the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules, and the other interface modules may be keyboards, mice, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more thereof, so the corresponding communication component 705 may include: Wi-Fi module, Bluetooth module, NFC module.

[0123] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the above-mentioned data reasoning method.

[0124] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned data reasoning method are implemented. For example, the computer-readable storage medium can be the above-mentioned memory 702 including program instructions, and the above-mentioned program instructions can be executed by the processor 701 of the electronic device 700 to complete the above-mentioned data reasoning method.

[0125] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a processor. When the computer program is executed by the processor, the steps of the above-mentioned data reasoning method are implemented.

[0126] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments. Within the technical concept of the present disclosure, a variety of simple modifications can be made to the technical solution of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0127] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present disclosure will not further describe various possible combinations.

[0128] In addition, various embodiments of the present disclosure may be arbitrarily combined, and as long as they do not violate the concept of the present disclosure, they should also be regarded as the contents disclosed by the present disclosure.

Claims

1. A data reasoning method, characterized in that: A network device applied to a data reasoning system, wherein the data reasoning system comprises the network device and a plurality of nodes, and the data reasoning method comprises: Receiving initial attention vectors and initial scaling factors from the multiple nodes, wherein the initial attention vectors and the initial scaling factors are obtained by each of the multiple nodes performing attention calculations on the context subsequences for which they are responsible; Initial attention vector and initial scaling factor for each node: Determining a local attention vector and a local scaling factor corresponding to the initial attention vector and the initial scaling factor stored locally by the network device; Determine a maximum value of the initial scaling factor and the local scaling factor and an absolute value of a difference between the initial scaling factor and the local scaling factor; If the absolute value of the difference is less than or equal to a preset value, the local scaling factor is updated based on the absolute value of the difference and the maximum value, and the local attention vector is updated based on the initial attention vector, the local attention vector and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes respectively to obtain a target attention vector, wherein the target attention vector is used by a main node among the multiple nodes to perform data inference to obtain an inference result.

2. The method according to claim 1, characterized in that The method further comprises: If the absolute value of the difference is greater than the preset value, the scaling factor corresponding to the maximum value is used to update the local scaling factor, and the attention vector corresponding to the maximum value is used to update the local attention vector.

3. The method according to claim 1, characterized in that The updating of the local scaling factor based on the absolute value of the difference and the maximum value comprises: According to the absolute value of the difference, searching for the logarithmic function value of the absolute value of the difference from a first table, wherein the first table records the corresponding relationship between each range of the absolute value of the difference and the logarithmic function value; The local scaling factor is updated by adding the found logarithmic function value to the maximum value.

4. The method according to any one of claims 1 to 3, characterized in that The updating the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference comprises: Based on the sizes of the initial scaling factor and the local scaling factor, selecting a table for the initial attention vector and a table for the local attention vector from a second table and a third table, wherein the second table records the corresponding relationship between the attention vector, the absolute value of the difference, and the first weighted value, and the third table records the corresponding relationship between the attention vector, the absolute value of the difference, and the second weighted value; According to the initial attention vector and the absolute value of the difference, searching for a weighted value for the initial attention vector from the table for the initial attention vector; According to the local attention vector and the absolute value of the difference, searching for a weighted value for the local attention vector from the table for the local attention vector; Update the local attention vector according to the weighted value for the initial attention vector and the weighted value for the local attention vector.

5. The method according to any one of claims 1 to 3, characterized in that The updating the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference comprises: Searching for a weight corresponding to the absolute value of the difference from a fourth table, wherein the fourth table records the corresponding relationship between each range of the absolute value of the difference and the weight; Based on the sizes of the initial scaling factor and the local scaling factor, selecting a table for the initial attention vector and a table for the local attention vector from a fifth table and a sixth table, wherein the fifth table records the corresponding relationship between the attention vector, the weight, and the third weighted value, and the sixth table records the corresponding relationship between the attention vector, the weight, and the fourth weighted value; According to the initial attention vector and the found weight, searching for a weighted value for the initial attention vector from the table for the initial attention vector; According to the local attention vector and the found weight, searching for a weighted value for the local attention vector from the table for the local attention vector; Update the local attention vector according to the weighted value for the initial attention vector and the weighted value for the local attention vector.

6. The method according to claim 1, characterized in that The updating of the local scaling factor based on the absolute value of the difference and the maximum value comprises: Determine the absolute value range of the difference within which the absolute value of the difference lies; Determine a linear function value of a fractional part of the absolute value of the difference based on the linear function corresponding to the absolute value range of the difference; Performing a shift operation on the integer part of the absolute value of the difference to obtain a shift operation result; The local scaling factor is updated based on the linear function value, the shift operation result and the maximum value.

7. The method according to claim 1, characterized in that The updating the local attention vector based on the initial attention vector, the local attention vector and the absolute value of the difference comprises: Determine the absolute value range of the difference within which the absolute value of the difference lies; Determining a weight according to the absolute value of the difference and based on a linear function corresponding to a range of the absolute value of the difference; Based on the initial attention vector, the local attention vector and the determined weight, the local attention vector is updated.

8. The method according to any one of claims 1 to 7, characterized in that The absolute value of the difference is a fixed-point number.

9. A network device, characterized in that: Applied to a data reasoning system, the data reasoning system comprises the network device and a plurality of nodes, the network device comprises: A receiving module, configured to receive an initial attention vector and an initial scaling factor from the multiple nodes, wherein the initial attention vector and the initial scaling factor are obtained by each of the multiple nodes performing attention calculation on the context subsequence for which it is responsible; Aggregation module for the initial attention vector and initial scaling factor for each node: Determining a local attention vector and a local scaling factor corresponding to the initial attention vector and the initial scaling factor stored locally by the network device; Determine a maximum value of the initial scaling factor and the local scaling factor and an absolute value of a difference between the initial scaling factor and the local scaling factor; If the absolute value of the difference is less than or equal to a preset value, the local scaling factor is updated based on the absolute value of the difference and the maximum value, and the local attention vector is updated based on the initial attention vector, the local attention vector and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes respectively to obtain a target attention vector, wherein the target attention vector is used by a main node among the multiple nodes to perform data inference to obtain an inference result.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 8 are implemented.

11. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images