Data Inference Method, Network Device, Medium, Electronic Device and Program Product

Abstract

The present disclosure relates to a data inference method, a network device, a medium, an electronic device, and a program product, belonging to the field of computer technology, capable of solving the bandwidth bottleneck problem, reducing latency, and improving system performance. A data inference method, applied to a network device in a data inference system, includes: receiving initial attention vectors and initial scaling factors from multiple nodes of the data inference system; for each initial attention vector and initial scaling factor, determining corresponding local attention vectors and local scaling factors, determining the maximum value of the initial and local scaling factors and the absolute value of the difference between the initial and local scaling factors, if the absolute value of the difference is less than or equal to a preset value, then updating the local scaling factor based on the absolute value of the difference and the maximum value, and updating the local attention vector based on the initial and local attention vectors and the absolute value of the difference to perform vector aggregation to obtain a target attention vector, and the target attention vector is used by the master node among the multiple nodes for data inference.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a data inference method, a network device, a medium, an electronic device, and a program product. Background Art

[0002] Large language model inference refers to the process of using a trained large language model for inference, where a user inputs prompt information to the large language model, and the large language model outputs answer information corresponding to the prompt information. Herein, the input and output of the large language model are collectively referred to as context.

[0003] As the context length of the large language model continues to increase, the computational amount of large language model inference will increase linearly. Currently, the inference process of the large language model is usually run among multiple distributed nodes, and the entire inference process is decomposed in the way of distributed memory. Specifically, it is sliced along the sequence length dimension of the context to obtain multiple subsequences, each distributed node is responsible for a limited-length subsequence, independently calculates the attention vector of the responsible subsequence, and then the master node in multiple distributed nodes performs a full reduction on the attention vectors calculated by all distributed nodes to obtain the global attention vector. This requires the master node to obtain all the attention vectors before performing the full reduction calculation, and after the calculation is completed, update the calculation result to all distributed nodes. This method is prone to causing computational and network bottlenecks of the master node, affecting system performance. Summary of the Invention

[0004] The purpose of the present disclosure is to provide a data inference method, a network device, a medium, an electronic device, and a program product, which can solve the bandwidth bottleneck problem, reduce latency, and improve system performance.

[0005] To achieve the above object, the present disclosure provides a data inference method, which is applied to a network device in a data inference system. The data inference system includes the network device and multiple nodes. The data inference method includes:

[0006] Receiving initial attention vectors and initial scaling factors from the multiple nodes, where the initial attention vectors and the initial scaling factors are obtained by the multiple nodes respectively performing attention calculations on the responsible context subsequences;

[0007] For the initial attention vectors and initial scaling factors of each node:

[0008] Determining local attention vectors and local scaling factors corresponding to the initial attention vectors and the initial scaling factors locally saved by the network device;

[0009] Determine the maximum value between the initial scaling factor and the local scaling factor, and the absolute value of the difference between the initial scaling factor and the local scaling factor;

[0010] If the absolute value of the difference is less than or equal to a preset value, update the local scaling factor based on the absolute value of the difference and the maximum value, and update the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes respectively, obtaining a target attention vector, where the target attention vector is used by the master node among the multiple nodes for data inference to obtain an inference result.

[0011] The present disclosure also provides a network device, which is applied to a data inference system. The data inference system includes the network device and multiple nodes. The network device includes:

[0012] A receiving module, configured to receive an initial attention vector and an initial scaling factor from the multiple nodes, where the initial attention vector and the initial scaling factor are obtained by the multiple nodes respectively performing attention calculation on the responsible context subsequence;

[0013] An aggregation module, configured to, for the initial attention vector and the initial scaling factor of each node:

[0014] Determine the local attention vector and the local scaling factor stored locally in the network device corresponding to the initial attention vector and the initial scaling factor;

[0015] Determine the maximum value between the initial scaling factor and the local scaling factor, and the absolute value of the difference between the initial scaling factor and the local scaling factor;

[0016] If the absolute value of the difference is less than or equal to a preset value, update the local scaling factor based on the absolute value of the difference and the maximum value, and update the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes respectively, obtaining a target attention vector, where the target attention vector is used by the master node among the multiple nodes for data inference to obtain an inference result.

[0017] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the steps of any method described in the present disclosure are implemented.

[0018] The present disclosure also provides an electronic device, including:

[0019] A memory, on which a computer program is stored;

[0020] A processor for executing the computer program in the memory to implement the steps of the method according to any one of the present disclosure.

[0021] The present disclosure also provides a computer program product, including a computer program, which implements the steps of the method according to any one of the present disclosure when executed by a processor.

[0022] By adopting the above technical solution, the initial attention vectors of each node are aggregated by a network device to obtain a target attention vector, and this aggregation is an online aggregation. Therefore, on the one hand, since the network device is a physical central node in the data inference system, its original bandwidth is relatively large and it is physically closer to other nodes. On the other hand, since online aggregation is adopted, the bandwidth bottleneck problem is solved, the latency is reduced, and the system performance is improved. In addition, since the attention aggregation is executed on the network device, the node side does not need to execute attention aggregation, thus saving the storage and computing resources of the node side. Further, since the nodes do not need to communicate pairwise, the bandwidth requirement for data transmission is reduced.

[0023] Other features and advantages of the present disclosure will be described in detail in the subsequent specific implementation part. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings are used to provide a further understanding of the present disclosure, and constitute a part of the specification, and are used to explain the present disclosure together with the following specific implementation, but do not constitute a limitation to the present disclosure. In the drawings:

[0025] Figure 1 is a flowchart of a data inference method according to an embodiment of the present disclosure.

[0026] Figure 2 is a schematic diagram of multiple nodes transmitting initial attention vectors and initial scaling factors to a network device according to an embodiment of the present disclosure.

[0027] Figure 3a is a schematic diagram of updating a local scaling factor by looking up a table in a first table according to an embodiment of the present disclosure.

[0028] Figure 3b and 3c are schematic diagrams of a second table and a third table respectively according to an embodiment of the present disclosure.

[0029] Figure 3d is a schematic diagram of a fourth table according to an embodiment of the present disclosure.

[0030] Figure 3e and 3f are schematic diagrams of a fifth table and a sixth table respectively according to an embodiment of the present disclosure.

[0031] Figure 4 It is a schematic block diagram of a network device according to an embodiment of the present disclosure.

[0032] Figure 5 It is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0033] The following will describe the detailed implementation manners of the present disclosure with reference to the accompanying drawings. It should be understood that the detailed implementation manners described herein are only used to illustrate and explain the present disclosure, and are not used to limit the present disclosure.

[0034] Figure 1 It is a flowchart of a data inference method according to an embodiment of the present disclosure. The data inference method is applied to a network device in a data inference system, and the data inference system includes a network device and multiple nodes.

[0035] The network device refers to the convergence point in the data inference system, or can also be called the physical central node. For example, in the data inference system, multiple nodes communicate through a switch, then the network device is the switch. Another example is that if the data inference system is implemented using multi-chip interconnection, then the network device is the input / output (Input / Output, IO) chip in the multi-chip interconnection.

[0036] Such as Figure 1 shown, the data inference method according to an embodiment of the present disclosure may include the following steps S11 to S14.

[0037] In step S11, receive the initial attention vector a' and the initial scaling factor γ' from multiple nodes, where the initial attention vector a' and the initial scaling factor γ' are obtained by each of the multiple nodes performing attention calculation on the context subsequence it is responsible for.

[0038] For the attention calculation, taking the data inference system using a Transformer model as the inference model as an example, each node will perform attention calculation on the context subsequence it is responsible for based on the query vector (q), the key vector (K) cached locally, and the value vector (V), and obtain the initial attention vector a' and the initial scaling factor γ'. The context subsequence refers to the subsequence obtained by slicing along the sequence length dimension of the context. The input and output of the inference model are collectively referred to as the context. The initial scaling factor γ' can be various defined scaling factors. For example, the initial scaling factor γ' can be an additional scalar for the initial attention vector a': γ' = log∑e qK .

[0039] Figure 2Schematic diagram of multiple nodes transmitting initial attention vectors and initial scaling factors to a network device according to an embodiment of the present disclosure. As Figure 2 shown, multiple nodes 111 to 11 n respectively send their respective initial attention vectors and initial scaling factors to the network device 10.

[0040] In step S12, for the initial attention vector a' and the initial scaling factor γ' of each node, determine the local attention vector a l and the local scaling factor γ l stored locally in the network device corresponding to the initial attention vector a' and the initial scaling factor γ'.

[0041] Exemplarily, multiple nodes may send the initial attention vector a' and the initial scaling factor γ' to the network device in the form of data packets, and corresponding IDs may be carried in the data packets. The network device can then determine the local attention vector a l and the local scaling factor γ l corresponding to the initial attention vector a' and the initial scaling factor γ' according to the IDs carried in the data packets sent by each node.

[0042] In addition, if the local attention vector a l and the local scaling factor γ l do not exist locally in the network device before, the network device can directly save the first received initial attention vector a' and the initial scaling factor γ' locally as the local attention vector a l and the local scaling factor γ l .

[0043] In step S13, determine the maximum value γ l among the initial scaling factor γ' and the local scaling factor γ max and the absolute value of the difference between the initial scaling factor γ' and the local scaling factor γ l where, In step S14, if the absolute value of the difference

[0044] is less than or equal to a preset value, then update the local scaling factor γ based on the absolute value of the difference and the maximum value γ , and update the local attention vector a based on the initial attention vector a', the local attention vector a max and the absolute value of the difference l to aggregate the initial attention vectors a' corresponding to multiple nodes respectively to obtain the target attention vector a l where, to aggregate the initial attention vectors a' corresponding to multiple nodes respectively to obtain the target attention vector a l 目标 ​, where the target attention vector a 目标 is used by the master node among multiple nodes for data inference to obtain an inference result.

[0045] Exemplarily, if the network device receives the initial attention vector a' and the initial scaling factor γ' from n nodes, the network device first determines the maximum value γ l between the initial scaling factor γ' of the first node and the local scaling factor γ max of the network device, l as well as the absolute value of the difference between the initial scaling factor γ' of the first node and the local scaling factor γ of the network device. If the absolute value of the difference max is less than or equal to a preset value, the network device updates the local scaling factor γ l based on the absolute value of the difference l and the maximum value γ and updates the local attention vector a l based on the initial attention vector a' of the first node, the local attention vector a 目标 of the network device, and the absolute value of the difference.

[0046] In this way, aggregation with the initial attention vector a' of the first node is achieved. Then, for the initial scaling factor γ' and the initial attention vector a' of the second node, the network device processes them in the same way as for the initial scaling factor γ' and the initial attention vector a' of the first node. By repeating this process, aggregation of the initial attention vectors a' of n nodes is completed until the target attention vector a is obtained. max In addition, if the absolute value of the difference l is greater than the preset value, it means that the influence of the smaller scaling factor on the larger scaling factor is very small (e.g., less than one-thousandth). Therefore, the smaller scaling factor can be ignored. Then, the network device can use the scaling factor max corresponding to the maximum value γ l to update the local scaling factor γ and use the attention vector l corresponding to the maximum value γ l to update the local attention vector a l . For example, in the case where the absolute value of the difference l is greater than the preset value, if the initial scaling factor γ' is greater than the local scaling factor γ l of the network device, the network device replaces the local scaling factor γl remain unchanged.

[0047] In addition, the target attention vector is the result of aggregating the initial attention vectors of all nodes in the data inference system. For example, the data inference system includes multiple network devices. Each network device is responsible for aggregating the initial attention vectors of some of the multiple nodes to obtain intermediate attention vectors. Then, one of the multiple network devices aggregates the intermediate attention vectors to obtain the target attention vector, which is used by the master node among the multiple nodes for data inference to obtain the inference result.

[0048] By adopting the above technical solution, the initial attention vectors of each node are aggregated by network devices to obtain the target attention vector, and this aggregation is an online aggregation. Therefore, on the one hand, since the network device is a physical central node in the data inference system, its original bandwidth is relatively large and it is physically closer to other nodes. On the other hand, due to the online aggregation adopted, the bandwidth bottleneck problem is solved, the latency is reduced, and the system performance is improved. In addition, since the attention aggregation is executed on the network device, the node side does not need to execute attention aggregation, thus saving the storage and computing resources on the node side. Further, since the nodes do not need to communicate pairwise, the bandwidth requirement for data transmission is reduced.

[0049] In some embodiments, the local scaling factor can be updated by means of table lookup. That is, the update of the local scaling factor based on the absolute value of the difference and the maximum value described in step S14 can be implemented in the following manner.

[0050] First, according to the absolute value of the difference, look up the logarithmic function value of the absolute value of the difference in the first table, where the first table records the corresponding relationship between each range of the absolute value of the difference and the logarithmic function value.

[0051] Then, update the local scaling factor by adding the found logarithmic function value to the maximum value.

[0052] For example, assume that the local scaling factor is updated using the following function: Since network devices generally do not have sufficient computing power and memory space, network devices cannot directly solve the above function. Therefore, the data inference method according to the embodiments of the present disclosure adopts the method of table lookup. The first table can be obtained in the following manner. First, according to the minimum weight requirement, determine the value range of the absolute value of the difference For example, assume that the minimum weight w is 0.004, and accordingly determine that the absolute value of the difference should not exceed 8, then the absolute value of the difference The value range of is from 0 to 8. Then, the absolute value of the difference is divided into multiple sub - value ranges. Each sub - value range corresponds to a different logarithmic function value of the absolute value of the difference, that is the value of Figure 3a is different. Within the same sub - value range, different absolute values of the difference

[0053] correspond to the same logarithmic function value of the absolute value of the difference. In this way, a first table recording the corresponding relationship between each range of the absolute value of the difference and the logarithmic function value can be obtained. By using the first table to look up the table, the update of the local scaling factor can be achieved. The schematic diagram of updating the local scaling factor by looking up the table in the first table is as shown.

[0054] In addition, to improve the accuracy, the absolute value of the difference

[0055] adopts the form of fixed - point numbers.

[0056] By adopting the above technical solution, the method of looking up the table can be used to replace complex operations, so that the aggregation of attention vectors can be achieved on network devices with insufficient computing power and memory space, reducing the hardware resource requirements and hardware complexity of network devices, and also avoiding problems such as numerical overflow and excessive resource overhead. For example, when using low - bit quantization (such as 8 bits), even for binary functions, it is not necessary to maintain a large table, so it will not consume too much memory resources.

[0057] In some embodiments, the local attention vector can be updated by looking up the table. That is, the update of the local attention vector based on the initial attention vector, local attention vector, and the absolute value of the difference described in step S14 can be achieved through the following method. l For example, assume that the local attention vector is updated using the function a = wa1+(1 - w)a2, where l a1 and a2 represent attention vectors; the weight of the attention vector corresponding to the larger scaling factor is w, and the weight of the attention vector corresponding to the smaller scaling factor is (1 - w). For example, if the initial scaling factor γ′ is greater than the local scaling factor γ l then the above function is a lIf the initial scaling factor γ′ is less than the local scaling factor γ l then the above function is a l = wa l + (1 - w)a′; The first weighted value in the second table is the value of the function and the second weighted value in the third table is the value of the function where a represents the attention vector. Then, if the initial scaling factor γ′ is greater than the local scaling factor γ l then the second table is the table for the initial attention vector a′ and the third table is the table for the local attention vector a l ; If the initial scaling factor γ′ is less than the local scaling factor γ l then the third table is the table for the initial attention vector a′ and the second table is the table for the local attention vector a l . Figure 3b and 3c are schematic diagrams of the second table and the third table according to embodiments of the present disclosure, respectively.

[0058] Then, according to the initial attention vector and the absolute value of the difference, the weighted value for the initial attention vector is looked up from the table for the initial attention vector, and according to the local attention vector and the absolute value of the difference, the weighted value for the local attention vector is looked up from the table for the local attention vector.

[0059] Then, the local attention vector is updated according to the weighted value for the initial attention vector and the weighted value for the local attention vector.

[0060] In this way, the update of the local attention vector can be achieved by looking up the table. In addition, the absolute value of the difference can be in the form of a fixed-point number. The more digits of the fixed-point number, the higher the accuracy of the updated local attention vector, but the larger the scale of the table (such as the second table, the third table). Therefore, a trade-off can be made between the scale of the table and the accuracy of the updated local attention vector.

[0061] By adopting the above technical solution, it is possible to use the method of looking up the table to replace complex operations, so that the aggregation of attention vectors can also be achieved on network devices with insufficient computing power and memory space, reducing the hardware resource requirements and hardware complexity of the network device, and also avoiding problems such as numerical overflow and excessive resource overhead. For example, when using low-bit quantization (such as 8 bits), even for binary functions, it is not necessary to maintain a too large table, so it will not consume too much memory resources.

[0062] In some embodiments, the local attention vector may be updated by table lookup. That is, the updating of the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference in step S14 may be implemented as follows.

[0063] First, the weight w corresponding to the absolute value of the difference is searched from the fourth table, wherein the fourth table records the corresponding relationship between each range of the absolute value of the difference and the weight w.

[0064] For example, suppose that the local attention vector adopts function a l =wa1+(1-w)a2 for update, where a1 and a2 represent the attention vectors. The fourth table can be obtained as follows. First, according to the minimum weight requirement, determine the absolute value of the difference For example, assuming the minimum weight w is 0.004, the absolute value of the difference is determined based on this. Should not exceed 8, then the absolute value of the difference The value range of is 0 to 8. Then, the absolute value of the difference The value range of is divided into multiple sub-value ranges. Each sub-value range corresponds to a different weight w value, that is, The values of are different. In the same sub-value range, different absolute values of the difference Corresponding to the same weight w value. In this way, a fourth table can be obtained that records the corresponding relationship between each range of the absolute value of the difference and the weight value. Figure 3d is a schematic diagram of a fourth table according to an embodiment of the present disclosure.

[0065] In addition, to improve accuracy, the absolute value of the difference It can be in the form of fixed-point numbers. For example, the absolute value of the difference can be in the form of fixed-point numbers. The first m bits of are used as the index of the fourth table. Assuming m is 4, the index can be recorded as

[0066] Then, based on the size of the initial scaling factor and the local scaling factor, a table for the initial attention vector and a table for the local attention vector are selected from the fifth table and the sixth table, wherein the fifth table records the correspondence between the attention vector, the weight and the third weighted value, and the sixth table records the correspondence between the attention vector, the weight and the fourth weighted value.

[0067] For example, suppose that the local attention vector adopts function a lis updated as \(a = wa_1+(1 - w)a_2\), where \(a_1\) and \(a_2\) represent attention vectors; the weight of the attention vector corresponding to the larger scaling factor is \(w\), and the weight of the attention vector corresponding to the smaller scaling factor is \((1 - w)\). For example, if the initial scaling factor \(\gamma'\) is greater than the local scaling factor \(\gamma\) l , then the above function is \(a\) l \(= wa'+(1 - w)a\) l . If the initial scaling factor \(\gamma'\) is less than the local scaling factor \(\gamma\) l , then the above function is \(a\) l \(= wa\) l \(+(1 - w)a'\); the third weighted value in the fifth table is the value of the function \(wa\), and the fourth weighted value in the sixth table is the value of the function \((1 - w)a\), where \(a\) represents the attention vector. Then, if the initial scaling factor \(\gamma'\) is greater than the local scaling factor \(\gamma\) l , then the fifth table is the table for the initial attention vector \(a'\), and the sixth table is the table for the local attention vector \(a\) l . If the initial scaling factor \(\gamma'\) is less than the local scaling factor \(\gamma\) l , then the sixth table is the table for the initial attention vector \(a'\), and the fifth table is the table for the local attention vector \(a\) l . Figure 3e and 3f are respectively schematic diagrams of the fifth table and the sixth table according to embodiments of the present disclosure.

[0068] Then, according to the initial attention vector and the found weight, the weighted value for the initial attention vector is looked up from the table for the initial attention vector, and according to the local attention vector and the found weight, the weighted value for the local attention vector is looked up from the table for the local attention vector.

[0069] Then, according to the weighted value for the initial attention vector and the weighted value for the local attention vector, the local attention vector is updated.

[0070] In this way, the update of the local attention vector can be achieved by looking up the table. In addition, the absolute value of the difference can be in the form of a fixed-point number. The more digits of the fixed-point number, the higher the accuracy of the updated local attention vector, but the larger the scale of the table (such as the fifth table, the sixth table). Therefore, a trade-off can be made between the scale of the table and the accuracy of the updated local attention vector.

[0071] By adopting the above technical solution, it is possible to use the method of looking up a table to replace complex operations, so that the aggregation of attention vectors can be achieved on network devices with insufficient computing power and memory space, reducing the hardware resource requirements and hardware complexity of network devices, and also avoiding problems such as numerical overflow and excessive resource overhead. For example, when using low-bit quantization (e.g., 8 bits), even for binary functions, it is not necessary to maintain a large table, so it will not consume excessive memory resources.

[0072] In some embodiments, in addition to using the method of looking up a table to update the local scaling factor, the local scaling factor can also be updated by means of fixed-point multiplication. That is, the update of the local scaling factor based on the absolute value of the difference and the maximum value described in step S14 can be implemented in the following manner.

[0073] First, determine the range of the absolute value of the difference.

[0074] For example, assume that the local scaling factor is updated using the function Since network devices generally do not have sufficient computing power and memory space, the network device cannot directly solve the above function. Therefore, the above function can be transformed into a linear function in the following manner. First, let where is the integer part of the absolute value of the difference, can be directly shifted, is the fractional part of the absolute value of the difference, which is essentially an exponential function with a value range of (0,1) and can be approximated by a piecewise linear function. For example, the value range of the absolute value of the difference is divided into several sub-value ranges, and each sub-value range corresponds to a linear function about the fractional part of the absolute value of the difference. For example, The slope k1 of the linear function corresponding to each sub-value range is different, and the intercept b1 of the linear function corresponding to each sub-value range is also different. The determination method of the value range of the absolute value of the difference has been described in detail above and will not be elaborated here. In addition, determining the range of the absolute value of the difference where the absolute value of the difference is located is to determine the sub-value range where the absolute value of the difference is located.

[0075] Then, based on the linear function corresponding to the range of the absolute value of the difference, determine the linear function value of the fractional part of the absolute value of the difference, and perform a shift operation on the integer part of the absolute value of the difference to obtain the shift operation result.

[0076] Then, update the local scaling factor based on the linear function value, the shift operation result, and the maximum value.

[0077] Taking the local scaling factor updated using the function for update, and moreover For example, the product of the linear function value and the shift operation result can be calculated, and log(1 + x) can be calculated using Taylor expansion, and then the Taylor expansion value is added to γ max to obtain an updated local scaling factor.

[0078] By adopting the above technical solution, it is possible to use the method of fixed-point multiplication to replace complex operations, so that the aggregation of attention vectors can be achieved on network devices with insufficient computing power and memory space, reducing the hardware resource requirements and hardware complexity of network devices, and also avoiding problems such as numerical overflow and excessive resource overhead.

[0079] In some embodiments, in addition to updating the local attention vector by using the look-up table method, the local attention vector can also be updated by using the fixed-point multiplication method. That is, the updating of the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference described in step S14 can be implemented by the following method.

[0080] First, determine the range of the absolute value of the difference.

[0081] For example, assume that the local attention vector is updated using the function a k = wa1 + (1 - w)a2, where the weights a1 and a2 represent attention vectors. Since network devices generally do not have sufficient computing power and memory space, the network device cannot directly solve the above function. Therefore, the following method can be used to transform it into a piecewise linear function. First, in order to achieve more stable accuracy (resolution), it can be evenly grouped according to the change range of w. For example, taking w decreasing by a set value (such as 0.1) as a fixed step size, the value range of is divided into several sub-value ranges, The determination method of the value range of has been described in detail above and will not be elaborated here. For the w change curve of each sub-value range, a linear approximation is performed. For example, where k2 and b2 respectively represent the slope and intercept of the linear function. During calculation, the absolute value of the difference is compared with the endpoints of each sub-value range in parallel, and the range of the absolute value of the difference where the absolute value of the difference is located can be determined. Therefore, only the endpoint values of each sub-value range and the slope values and intercept values of each linear function need to be maintained, greatly reducing the memory overhead.

[0082] Then, based on the absolute value of the difference, determine the weight based on the linear function corresponding to the range of the absolute value of the difference.

[0083] After determining the range of the absolute value of the difference, the linear function corresponding to this range of the absolute value of the difference can be known. Based on this linear function, the corresponding weight can be calculated.

[0084] Then, based on the initial attention vector, the local attention vector, and the determined weight, the local attention vector is updated.

[0085] Exemplarily, assuming that the weight of the attention vector corresponding to the larger scaling factor is w, and the weight of the attention vector corresponding to the smaller scaling factor is (1 - w), then the weight of the initial attention vector being w or (1 - w) and the weight of the local attention vector being w or (1 - w) can be determined according to the magnitudes of the initial scaling factor γ′ and the local scaling factor γ l , and then the function a l = wa1 + (1 - w)a2 can be used to update the local attention vector.

[0086] By adopting the above technical solution, the way of fixed-point multiplication can be used to replace complex operations. Therefore, on the one hand, the aggregation of attention vectors can be achieved on network devices with insufficient computing power and memory space. On the other hand, the hardware resource requirements and hardware complexity of the network device are reduced. On the other hand, the memory resources occupied by the lookup table can be compressed, and a higher-precision data type (such as 16bit) can be used for aggregation to improve the precision, and the problems of numerical overflow and excessive resource overhead can be avoided.

[0087] Figure 4 It is a schematic block diagram of a network device according to an embodiment of the present disclosure. This network device can be applied to a data inference system, and the data inference system includes a network device and multiple nodes. The network device refers to the convergence point in the data inference system, or can also be called the physical central node. For example, in the data inference system, multiple nodes communicate with each other through a switch, then the network device is the switch. For another example, if the data inference system is implemented using multi-chip interconnection, then the network device is the input / output (Input / Output, IO) chip in the multi-chip interconnection.

[0088] Such as Figure 4As shown in the figure, the network device 30 includes: a receiving module 31, configured to receive initial attention vectors and initial scaling factors from multiple nodes, where the initial attention vectors and initial scaling factors are obtained by each of the multiple nodes performing attention calculation on the context subsequence it is responsible for; an aggregation module 32, configured to, for the initial attention vector and initial scaling factor of each node, determine a local attention vector and a local scaling factor corresponding to the initial attention vector and initial scaling factor locally stored in the network device, determine the maximum value between the initial scaling factor and the local scaling factor and the absolute value of the difference between the initial scaling factor and the local scaling factor, and if the absolute value of the difference is less than or equal to a preset value, update the local scaling factor based on the absolute value of the difference and the maximum value, and update the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes respectively, obtaining a target attention vector, where the target attention vector is used by the master node among the multiple nodes for data inference to obtain an inference result.

[0089] By adopting the above technical solution, the network device aggregates the initial attention vectors of each node to obtain a target attention vector, and this aggregation is an online aggregation. Therefore, on the one hand, since the network device is the physical central node in the data inference system, its original bandwidth is relatively large and it is physically closer to other nodes. On the other hand, due to the adoption of online aggregation, the bandwidth bottleneck problem is solved, the latency is reduced, and the system performance is improved. In addition, since the attention aggregation is executed on the network device, the node side does not need to perform attention aggregation, thus saving the storage and computing resources on the node side. Further, since the nodes do not need to communicate pairwise, the bandwidth requirement for data transmission is reduced.

[0090] Optionally, the aggregation module 32 is further configured to: if the absolute value of the difference is greater than the preset value, update the local scaling factor using the scaling factor corresponding to the maximum value, and update the local attention vector using the attention vector corresponding to the maximum value.

[0091] Optionally, the aggregation module 32 updates the local scaling factor based on the absolute value of the difference and the maximum value, including:

[0092] According to the absolute value of the difference, look up the logarithmic function value of the absolute value of the difference in a first table, where the first table records the corresponding relationship between each range of the absolute value of the difference and the logarithmic function value;

[0093] Update the local scaling factor by adding the found logarithmic function value to the maximum value.

[0094] Optionally, the aggregation module 32 updates the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference, including:

[0095] Based on the magnitudes of the initial scaling factor and the local scaling factor, select the table for the initial attention vector and the table for the local attention vector from the second table and the third table, where the second table records the correspondence between the attention vector, the absolute value of the difference, and the first weighting value, and the third table records the correspondence between the attention vector, the absolute value of the difference, and the second weighting value;

[0096] According to the initial attention vector and the absolute value of the difference, look up the weighting value for the initial attention vector from the table for the initial attention vector;

[0097] According to the local attention vector and the absolute value of the difference, look up the weighting value for the local attention vector from the table for the local attention vector;

[0098] Update the local attention vector according to the weighting value for the initial attention vector and the weighting value for the local attention vector.

[0099] Optionally, the aggregation module 32 updates the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference, including:

[0100] Look up the weight corresponding to the absolute value of the difference from the fourth table, where the fourth table records the correspondence between each range of the absolute value of the difference and the weight;

[0101] Based on the magnitudes of the initial scaling factor and the local scaling factor, select the table for the initial attention vector and the table for the local attention vector from the fifth table and the sixth table, where the fifth table records the correspondence between the attention vector, the weight, and the third weighting value, and the sixth table records the correspondence between the attention vector, the weight, and the fourth weighting value;

[0102] According to the initial attention vector and the looked-up weight, look up the weighting value for the initial attention vector from the table for the initial attention vector;

[0103] According to the local attention vector and the looked-up weight, look up the weighting value for the local attention vector from the table for the local attention vector;

[0104] Update the local attention vector according to the weighting value for the initial attention vector and the weighting value for the local attention vector.

[0105] Optionally, the aggregation module 32 updates the local scaling factor based on the absolute value of the difference and the maximum value, including:

[0106] Determine the range of the absolute value of the difference in which the absolute value of the difference is located;

[0107] Determine the linear function value of the fractional part of the absolute value of the difference based on the linear function corresponding to the range of the absolute value of the difference;

[0108] Perform a shift operation on the integer part of the absolute value of the difference to obtain a shift operation result;

[0109] Update the local scaling factor based on the linear function value, the shift operation result, and the maximum value.

[0110] Optionally, the aggregation module 32 updates the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference, including:

[0111] Determine the range of the absolute value of the difference in which the absolute value of the difference is located;

[0112] Based on the absolute value of the difference, determine the weight according to the linear function corresponding to the range of the absolute value of the difference;

[0113] Update the local attention vector based on the initial attention vector, the local attention vector, and the determined weight.

[0114] Optionally, the absolute value of the difference is a fixed-point number.

[0115] Regarding the network device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0116] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the present disclosure are implemented.

[0117] The present disclosure also provides an electronic device, including:

[0118] A memory, on which a computer program is stored;

[0119] A processor, configured to execute the computer program in the memory to implement the steps of any one of the methods in the present disclosure.

[0120] The present disclosure also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in the present disclosure are implemented.

[0121] Figure 5 It is a block diagram of an electronic device 700 shown according to an exemplary embodiment. As Figure 5 shown, the electronic device 700 may include: a processor 701, a memory 702. The electronic device 700 may further include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0122] Among them, the processor 701 is used to control the overall operation of the electronic device 700 to complete all or part of the steps in the above data inference method. The memory 702 is used to store various types of data to support the operation of the electronic device 700. These data may include, for example, instructions for any application or method operating on the electronic device 700, as well as application-related data, such as contact data, received and sent messages, pictures, audio, video, and so on. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touch screen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signal may be further stored in the memory 702 or transmitted through the communication component 705. The audio component also includes at least one speaker for outputting audio signals. The I / O interface 704 provides an interface between the processor 701 and other interface modules. The above other interface modules may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, near field communication (NFC), 2G, 3G or 4G, or a combination of one or more of them. Accordingly, the communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module.

[0123] In an exemplary embodiment, the electronic device 700 can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components, and is used to execute the above data inference method.

[0124] In another exemplary embodiment, a computer-readable storage medium including program instructions is further provided. When the program instructions are executed by a processor, the steps of the above data inference method are implemented. For example, the computer-readable storage medium can be the above memory 702 including program instructions, and the above program instructions can be executed by the processor 701 of the electronic device 700 to complete the above data inference method.

[0125] In another exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program that can be executed by a processor. When the computer program is executed by the processor, the steps of the above data inference method are implemented.

[0126] The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings. However, the present disclosure is not limited to the specific details in the above embodiments. Within the scope of the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure, and these simple modifications all fall within the protection scope of the present disclosure.

[0127] In addition, it should be noted that, among the various specific technical features described in the above specific embodiments, they can be combined in any suitable manner without conflict. To avoid unnecessary repetition, the present disclosure will not separately describe various possible combination methods.

[0128] In addition, any combination can be made between various different embodiments of the present disclosure, as long as it does not violate the idea of the present disclosure, and it should also be regarded as the content disclosed by the present disclosure.

Claims

1. A data reasoning method, characterized in that: A network device applied to a data inference system, wherein the data inference system comprises the network device and a plurality of nodes, and the data inference method comprises: Receiving initial attention vectors and initial scaling factors from the multiple nodes, wherein the initial attention vectors and the initial scaling factors are obtained by each of the multiple nodes performing attention calculations on the context subsequences for which they are responsible; Initial attention vector and initial scaling factor for each node: Determining a local attention vector and a local scaling factor corresponding to the initial attention vector and the initial scaling factor, which are stored locally on the network device; Determining a maximum value of the initial scaling factor and the local scaling factor and an absolute value of a difference between the initial scaling factor and the local scaling factor; If the absolute value of the difference is less than or equal to a preset value, updating the local scaling factor based on the absolute value of the difference and the maximum value, and updating the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes to obtain a target attention vector, wherein the target attention vector is used by a master node among the multiple nodes to perform data inference to obtain an inference result; The method further comprises: If the absolute value of the difference is greater than the preset value, the scaling factor corresponding to the maximum value is used to update the local scaling factor, and the attention vector corresponding to the maximum value is used to update the local attention vector.

2. The method according to claim 1, characterized in that The updating of the local scaling factor based on the absolute value of the difference and the maximum value comprises: According to the absolute value of the difference, searching for the logarithmic function value of the absolute value of the difference from a first table, wherein the first table records the corresponding relationship between each range of the absolute value of the difference and the logarithmic function value; The local scaling factor is updated by adding the found logarithmic function value to the maximum value.

3. The method according to any one of claims 1 to 2, characterized in that The updating of the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference includes: Based on the sizes of the initial scaling factor and the local scaling factor, selecting a table for the initial attention vector and a table for the local attention vector from a second table and a third table, wherein the second table records a correspondence between the attention vector, the absolute value of the difference, and the first weighted value, and the third table records a correspondence between the attention vector, the absolute value of the difference, and the second weighted value; According to the initial attention vector and the absolute value of the difference, searching for a weighted value for the initial attention vector from the table for the initial attention vector; According to the local attention vector and the absolute value of the difference, searching for a weighted value for the local attention vector from the table for the local attention vector; Update the local attention vector according to the weighted value for the initial attention vector and the weighted value for the local attention vector.

4. The method according to any one of claims 1 to 2, characterized in that The updating of the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference includes: searching a weight corresponding to the absolute value of the difference from a fourth table, wherein the fourth table records a corresponding relationship between each range of the absolute value of the difference and the weight; Based on the magnitudes of the initial scaling factor and the local scaling factor, selecting a table for the initial attention vector and a table for the local attention vector from a fifth table and a sixth table, wherein the fifth table records a correspondence between attention vectors, weights, and third weighted values, and the sixth table records a correspondence between attention vectors, weights, and fourth weighted values; According to the initial attention vector and the found weight, searching for a weighted value for the initial attention vector from the table for the initial attention vector; According to the local attention vector and the found weight, searching for a weighted value for the local attention vector from the table for the local attention vector; The local attention vector is updated according to the weighted value for the initial attention vector and the weighted value for the local attention vector.

5. The method according to claim 1, wherein The updating of the local scaling factor based on the absolute value of the difference and the maximum value comprises: Determining the absolute value range of the difference within which the absolute value of the difference lies; Determining a linear function value of a fractional portion of the absolute value of the difference based on a linear function corresponding to the absolute value range of the difference; Performing a shift operation on the integer part of the absolute value of the difference to obtain a shift operation result; The local scaling factor is updated based on the linear function value, the shift operation result, and the maximum value.

6. The method according to claim 1, wherein The updating of the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference includes: Determining the absolute value range of the difference within which the absolute value of the difference lies; Determining a weight based on the absolute value of the difference and a linear function corresponding to the absolute value range of the difference; The local attention vector is updated based on the initial attention vector, the local attention vector, and the determined weight.

7. The method according to claim 1, characterized in that The absolute value of the difference is a fixed-point number.

8. A network device, characterized in that: Applied to a data inference system, the data inference system includes the network device and multiple nodes, the network device includes: A receiving module, configured to receive initial attention vectors and initial scaling factors from the multiple nodes, wherein the initial attention vectors and the initial scaling factors are obtained by each of the multiple nodes performing attention calculations on the context subsequences for which they are responsible; Aggregation module for the initial attention vector and initial scaling factor for each node: Determining a local attention vector and a local scaling factor corresponding to the initial attention vector and the initial scaling factor, which are stored locally on the network device; Determining a maximum value of the initial scaling factor and the local scaling factor and an absolute value of a difference between the initial scaling factor and the local scaling factor; If the absolute value of the difference is less than or equal to a preset value, updating the local scaling factor based on the absolute value of the difference and the maximum value, and updating the local attention vector based on the initial attention vector, the local attention vector, and the absolute value of the difference to aggregate the initial attention vectors corresponding to the multiple nodes to obtain a target attention vector, wherein the target attention vector is used by a master node among the multiple nodes to perform data inference to obtain an inference result; The aggregation module is further configured to: If the absolute value of the difference is greater than the preset value, the scaling factor corresponding to the maximum value is used to update the local scaling factor, and the attention vector corresponding to the maximum value is used to update the local attention vector.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images