A reconfigurable array lower transformer network model data multiplexing scheme and implementation method

Abstract

The application relates to the technical field of data multiplexing, and discloses a reconfigurable array-based Transformer network model data multiplexing scheme and an implementation method. A weight sharing method based on reverse circulation is used to construct an L-layer MSA module using L / 2 layers of parameters. The first L / 2 layers use different parameters, and the last L / 2 layers repeatedly use the parameters of the first L / 2 layers, so that weight sharing is realized. The application solves the problems of large memory access and high training cost of the Transformer network by using the weight sharing method, greatly reduces resource occupation, enables the Transformer model to run on the terminal side, and simultaneously ensures that the precision loss is within an acceptable range.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data multiplexing, in particular to a data multiplexing scheme and implementation method of a Transformer network model under a reconfigurable array. BACKGROUND

[0002] Edge devices usually have limited computing power, storage and energy consumption. Since the Transformer model is usually large and computationally intensive, directly deploying it on edge devices may lead to performance degradation, increased latency and shortened battery life. Although the pre-trained language model of the Transformer network can achieve good results on many downstream tasks after fine-tuning, the Transformer model itself is large and requires more computing resources, so it is difficult to deploy these models to hardware. Through data multiplexing, the number of model reloading and computing can be reduced, thereby reducing the burden on the device.

[0003] When designing data flow paths, data multiplexing is the main means to achieve high performance and low power consumption, which can be further divided into time multiplexing and space multiplexing. Time multiplexing occurs when the same data is reused in different cycles, while space multiplexing occurs when the same data is reused at different PEs.

[0004] Weight sharing is a simple and effective technique to reduce model size, which has been used in Transformer model compression in NLP. There are many ways to share parameters, such as sharing only parameters between different layers of the feedforward network, or sharing only parameters of the attention mechanism, while ALBERT shares all parameters of all layers. This method can significantly reduce the model size without seriously affecting the model performance, and can effectively improve the stability of the model. Transformer networks mostly use weight sharing methods. Using hierarchical sharing methods can further improve the performance of the model, as parameter sharing between different layers can alleviate the risk of overfitting.

[0005] As shown in Table 1 below, the existing data multiplexing technology of the Transformer network.

[0006] Table 1 Existing data multiplexing technology of the Transformer network

[0007]

[0008] However, the existing data multiplexing scheme in the architecture has the disadvantage of being difficult to deploy on end-side devices with limited computing resources. SUMMARY

[0009] In view of the above deficiencies in the prior art, the purpose of the present application is to provide a reconfigurable array under the Transformer network model data multiplexing scheme and implementation method.

[0010] To achieve the above purpose, the present application adopts the following technical solutions:

[0011] A reconfigurable array under the Transformer network model data multiplexing scheme uses a weight sharing method based on reverse cycle to construct an L layer MSA module using L / 2 layers (half of the Transformer layers in the module) of parameters, the first L / 2 layers use different parameters, and the last L / 2 layers reuse the parameters of the first L / 2 layers, thereby realizing weight sharing.

[0012] Further, the implementation of the weight sharing method is shown in Algorithm 1, CreateNewLayer is a function for creating an MSA module; first, stack the first L / 2 layers, the parameters of these layers are independent of each other, and the last L / 2 layers use the weight sharing method of reverse cycle to reuse the parameters of the first L / 2 layers;

[0013]

[0014] Further, the present application provides an implementation method of reconfigurable array under the Transformer network model data multiplexing, comprising the following steps:

[0015] S1. Divide the PE array into four PE groups, each PE group contains 4x4 PEs, and is a cluster;

[0016] S2. The PEs in the first group read image features and position features from the blocked input image data; the PEs in the second group read the feature information in the first group of PEs to realize data processing and storage of various operations of the encoder module; the PEs in the third group read the feature information in the second group of PEs to realize data processing and storage of various operations of the decoder module; the PEs in the fourth group prepare for various operations of the downstream tasks of the Transformer network; different PE groups read image features and weights from different input channels, and the output results are stored in the PE groups of the entire array.

[0017] Further, the data processing of the encoder module is realized by the PEs in the second group in cooperation, each input vector is mapped to a PE, after extracting the image features, the next operation is performed.

[0018] Further, the calculation process of the core unit MSA module of the encoder module is decomposed into parallel vector / matrix operations, each operation uses a row of image features and a row of image weights, and generates a row of output results; the output results are accumulated together to generate feature maps of different sizes after mapping.

[0019] Further, the input vector of the one-dimensional image comes from the storage structure of the first group of PE arrays.

[0020] Further, when the features of the image block are extracted in parallel to the first row of PE, the image weight is generated, the window shift partition method and the image weight are used to realize the vector transformation and matrix operation of the scaled dot attention, and finally the accumulation of the output data is performed to form the final output result.

[0021] Compared with the prior art, the application has the following beneficial effects: the weight sharing method is adopted to solve the problems of large memory access amount and high training cost of the Transformer network, greatly reducing the resource occupation, so that the Transformer model can run on the end side, and the precision loss is within the acceptable range. BRIEF DESCRIPTION OF DRAWINGS

[0022] Other features, objects and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0023] Figure 1 The figure is a schematic diagram of the data multiplexing scheme of the application;

[0024] Figure 2 The figure is a schematic diagram of the image data input PE array;

[0025] Figure 3 The figure is a schematic diagram of the PE array grouping;

[0026] Figure 4 The figure is a schematic diagram of the multi-level distributed storage structure;

[0027] Figure 5 The figure is a data flow diagram of the MSA module calculation;

[0028] Figure 6 The figure is a data multiplexing simulation waveform of the MSA module;

[0029] Figure 7 The figure is the accuracy and parameter amount of the comparative network model;

[0030] Figure 8 The figure is the accuracy and FLOPs of the comparative network model. DETAILED DESCRIPTION

[0031] The application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that for those skilled in the art, without departing from the concept of the application, a number of modifications and improvements can be made. These all belong to the protection scope of the application.

[0032] Example 1

[0033] Based on the design of data reuse schemes for the MSA modules of Transformer networks under reconfigurable array structures, the complexity of mapping Transformer networks to reconfigurable array structures stems from the data processing and weight sharing of the MSA modules. One design scheme for data reuse of Transformer network models under reconfigurable array structures is as follows: Figure 1 As shown, an L-layer MSA module using L / 2 layer parameters is constructed by using a weight sharing method based on reverse loop. The first L / 2 layers use different parameters, and the subsequent L / 2 layers reuse the parameters of the first L / 2 layers, thereby realizing weight sharing between high and low layers.

[0034] The implementation method for data reuse of Transformer network models under reconfigurable arrays divides the PE array into four PE groups, each PE group (Processing Element Group, PEG) containing 4×4 PEs, forming a cluster; the PEs in the first group read image features and location features from the block-based input image data, such as... Figure 2 As shown in the diagram. The PEs in the second group read feature information from the PEs in the first group to perform data processing and storage for various operations in the encoder module; the PEs in the third group read feature information from the PEs in the second group to perform data processing and storage for various operations in the decoder module; the PEs in the fourth group prepare for various operations in the downstream tasks of the Transformer network. Different PE groups read image features and weights from different input channels, and the output results are stored in the PE groups of the entire array. The PE array is grouped as follows: Figure 3 As shown.

[0035] Data processing in the encoder module is achieved through collaborative processing of PEs within the second group. Each input vector is mapped into a PE, and after extracting image features, it proceeds to the next step of computation. The core unit of the encoder module is the MSA module, whose computation process is decomposed into parallel vector / matrix operations (supporting parallel vector multiplication; matrices can be split into vectors for similar operations). Each operation uses one row of image features and one row of image weights, generating one row of output results. The output results are accumulated to generate feature maps of different sizes after mapping. The input vector of the one-dimensional image comes from the storage structure of the first group of PE arrays. This storage structure is a private storage layer composed of multiple Scratch Pad Memory (SPM) units, which, together with the distributed cache structure and the PE array, constitute a multi-level distributed storage structure. The multi-level distributed storage structure is as follows: Figure 4As shown, the on-chip cache is composed of SPM and Cache hybrid. The upper SPM layer is the private storage layer of PE layer, storing the data of operation core, but the consistency between operation cores is managed by software. The middle PE layer is a PEG composed of 4x4 PEs. The lower Cache layer is the shared storage layer of PE layer, and the communication between data is realized through the cross-array adjacent interconnection network. The MSA module calculates the data flow graph as shown in Figure 5

[0036] Because the input data set of the training Transformer network model has a large number of images, the work of mapping all images to the PE array is extremely huge, which will greatly affect the training efficiency. Therefore, when the features of the image block are extracted in parallel to the first row of PE, the image weight is generated. Using the window shift partition method and the image weight, the vector transformation and matrix operation of the scaled dot product attention are realized, and finally the accumulation of the output data is performed to form the final output result. Compared with the existing design, the data flow graph can adapt to the operation of each module of the Transformer network by maximizing the use of PE local storage structure, direct communication between PEs and spatial parallelism, and reducing the loss in the data transmission process.

[0037] The data multiplexing of the MSA module of the Transformer network is a scheme to reuse parameters and reduce network parameters, thereby realizing NxTransformer layer (i.e., encoder module). The weight sharing method mentioned in the previous section is implemented as shown in Algorithm 1, which describes the process of assigning parameters to each layer of the MSA module. CreateNewLayer is a function to create an MSA module. First, stack L / 2 layers, and the parameters of these layers are independent of each other. The last L / 2 layers use the reverse cyclic weight sharing method to reuse the parameters of the first L / 2 layers.

[0038]

[0039]

[0040] Embodiment 2

[0041] According to the instruction set of the reconfigurable array structure, as shown in Table 1 below, four types of instructions including arithmetic logic operation, data transfer, branch jump and network operation are designed. Then, the assembly code is translated into binary and stored in the instruction memory. Finally, simulation verification is performed using QuestaSim 10.1d simulation software.

[0042] Table 1 Instruction set of reconfigurable array structure

[0043]

[0044] ​

[0045] Firstly, the reconfigurable PE array is grouped, and the first group of PE array is used to realize the reading of image features and position features of image data. After the input image data is blocked, the feature sequence after reading is respectively put into 16 PEs of PE00, PE01, PE02, PE03, PE10, PE11, PE12, PE13, PE20, PE21, PE22, PE23, PE30, PE31, PE32, and PE33. Then, each PE performs LDI operation on the input data in parallel, and the purpose is data integration, which is conducive to the feature reading and the next vector-matrix operation of the second group of PE array. The feature reading code and its binary are shown in Table 2.

[0046] Table 2 Feature reading code and its binary

[0047]

[0048] Then, the second group of PE array uses LDI (Load Direct Instruction) operation to read the feature information in the first group of PE array, and stores it into four PEs of PE00, PE01, PE02, and PE03. According to the window shift partition method, the image weight is stored into four PEs of PE10, PE11, PE12, and PE13 by using LDI operation. In eight PEs of PE00, PE01, PE02, PE03, PE10, PE11, PE12, and PE13, corresponding vector transformation and matrix operation are performed by using MAC operation. The output of MSA module calculation is written into four PEs of PE20, PE21, PE22, and PE23 by using SLA (Service Level Agreement) operation. The result is integrated and stored into four PEs of PE30, PE31, PE32, and PE33 by using STI (Set Interrupt Flag) operation.

[0049] Finally, the Swin-T (Swin-Transformer) optimal subnetwork is based on the weight sharing method of reverse cycle, and realizes the data multiplexing of the MSA module of the Transformer network in the second group of PE arrays, i.e. within a single cluster. By using the LDI operation, the PE10 image weight is read from the 0th address and stored in the R1 register, and then the PE10 image weight is read from the 1st address and stored in the R2 register. By using the SLA operation, the output result is written into the R9 register, thereby realizing the data multiplexing of the first MSA module within the PE10. Then, the PE10 image weight is read from the 2nd address and the 3rd address and stored in the R3 register and the R4 register, respectively, and the output result is written into the R8 register, thereby realizing the data multiplexing of the second MSA module within the PE10. Subsequently, the PE10 image weight is read from the 4th address and the 5th address and stored in the R5 register and the R6 register, respectively, and the output result is written into the R7 register, thereby realizing the data multiplexing of the third MSA module within the PE10, and the PE10 realizes the weight sharing method based on the reverse cycle.

[0050] The MSA module data multiplexing simulation waveform is shown in Figure 6 The data multiplexing process of the first MSA module within the PE10 (0000000000000100) is shown. The PE10 image weight is read from the 0th address and the 1st address and stored in the R1 register and the R2 register, respectively, and the output result is written into the R9 register. In order to realize the data multiplexing of the MSA module of the Transformer network, data multiplexing also needs to be performed on the PE11 (0000000000000101), the PE12 (0000000000000110), and the PE13 (0000000000000111), respectively, and thus the sharing of all weights is realized.

[0051] The MSA module data multiplexing is performed on the Swin-T network under the reconfigurable structure. The comparison of the network model performance is shown in Table 3. The parameter quantity of the compressed Swin-T optimal subnetwork is 13.8M, the FLOPs is 1.9G, and the accuracy is 89.2%. After data multiplexing, the parameter quantity of the Swin-T network is 7.5M, the FLOPs is 0.9G, and the accuracy is 88.2%. It can be concluded that the weight sharing method based on the reverse cycle greatly reduces the parameter quantity and the FLOPs of the compressed Swin-T optimal subnetwork, with a reduction of 6.3M in the parameter quantity and 1G in the FLOPs.

[0052] Table 3 Comparison of network model performance

[0053]

[0054] In order to intuitively compare the performance of the original Swin-T, the compressed Swin-T and the data multiplexed Swin-T, two bar- curve graphs of accuracy and parameter quantity, accuracy and FLOPs are drawn. Figure 7 As shown in the accuracy and parameter quantity comparison of the network model, Figure 8 As shown in the accuracy and FLOPs comparison of the network model, Figure 7 Figure 8 As shown in the accuracy and parameter quantity comparison of the network model,

[0055] The specific embodiments of the present application are described above. It should be understood that the present application is not limited to the specific embodiments described above, and various modifications or changes can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application.​

Claims

1. A method for reusing Transformer network model data under a reconfigurable array, characterized in that, By employing a weight-sharing method based on reverse loops, an L-layer MSA module using L / 2 layer parameters is constructed. The first L / 2 layers use different parameters, while the subsequent L / 2 layers reuse the parameters of the first L / 2 layers, thereby achieving weight sharing. The implementation of the weight sharing method is shown in Algorithm 1. CreateNewLayer is a function that creates the MSA module. First, the first L / 2 layers are stacked, and the parameters of these layers are independent of each other. The last L / 2 layers use a reverse loop weight sharing method to reuse the parameters of the first L / 2 layers. The method for reusing Transformer network model data under a reconfigurable array includes the following steps: S1. Divide the PE array into four PE groups, each PE group containing 4×4 PEs, forming a cluster; S2. The PEs in the first group read image features and positional features from the block-based input image data; the PEs in the second group read the feature information from the PEs in the first group to perform data processing and storage for various operations of the encoder module; the PEs in the third group read the feature information from the PEs in the second group to perform data processing and storage for various operations of the decoder module; the PEs in the fourth group prepare for various operations of the downstream tasks of the Transformer network; different PE groups read image features and weights from different input channels, and the output results are stored in the PE groups of the entire array.

2. The method for reusing Transformer network model data under a reconfigurable array according to claim 1, characterized in that, Data processing in the encoder module is achieved through collaborative processing of PEs within the second group. Each input vector is mapped into a PE, and after extracting image features, the next step of computation is performed.

3. The method for reusing Transformer network model data under a reconfigurable array according to claim 1, characterized in that, The computation process of the core unit MSA module of the encoder module is decomposed into parallel vector / matrix operations. Each operation uses one row of image features and one row of image weights, and generates one row of output results. The outputs are accumulated together to generate feature maps of different sizes after mapping.

4. The method for reusing Transformer network model data under a reconfigurable array according to claim 2, characterized in that, The input vector of the one-dimensional image comes from the storage structure of the first set of PE arrays.

5. The method for reusing Transformer network model data under a reconfigurable array according to claim 1, characterized in that, After the features of the image patch are extracted in parallel to the PE of the first row, image weights are generated. Using the window shift partitioning method and image weights, vector transformation and matrix operations of scaling dot product attention are implemented. Finally, the output data is accumulated to form the final output result.

Images