Information processing device running large language model

Abstract

The invention discloses an information processing device for running a large language model. In order to reduce high calculation cost brought to an artificial intelligence system by matrix multiplication, the information processing device comprises a plurality of data processing sub-modules, at least one data processing sub-module comprises a Token mixer which comprises an RNN structure, and a weight value in a weight matrix in the RNN structure is selected from-1, 0 and 1 or is selected from-1 and 1; the channel mixer is a gating linear unit, and a weight value in a weight matrix in the gating linear unit is selected from-1, 0 and 1 or is selected from-1 and 1; the input and the output of the Token mixer are added to obtain a first result, the input and the output of the channel mixer are added to obtain a second result, and the first result is used for obtaining the input of the channel mixer. According to the method, matrix multiplication can be eliminated, the calculation cost is reduced, considerable performance is maintained, and a steeper scale law is shown. The method is suitable for the field of artificial intelligence.

Description

[0001] This patent application claims priority to the invention patent application number 202410702214.7 (invention name: neural network and large language model) submitted to the State Intellectual Property Office of China on June 1, 2024, and introduces and merges all the contents recorded therein into this patent application document. Technical Field

[0002] The present invention relates to the field of artificial intelligence, and in particular to an information processing device for running a large language model. Background Art

[0003] The current development of artificial intelligence has made significant progress, especially Transformer, which has achieved excellent performance in the fields of computer vision and large language models (LLMs). Transformer has become the core of current advanced artificial intelligence technologies such as chatGPT and Sora. Artificial intelligence technologies based on neural networks and LLMs are bound to be widely used in all aspects of society in the future.

[0004] As the scale of neural network parameters in large models rapidly expands, the energy efficiency of neural network operation has gradually become a bottleneck for the widespread application of artificial intelligence. Energy consumption, latency, computing costs, etc. have become unfavorable factors that restrict the widespread application of artificial intelligence.

[0005] Matrix multiplication in neural networks dominates the amount of computation in neural network operations. Matrix multiplication is a basic and abundant operation in neural network operations. It is an operation with high computational cost, low efficiency and consumes a lot of computing resources. In the fully connected dense layer, matrix multiplication is used to assign weights to simulate the synaptic operation of the brain; in the self-attention layer, matrix multiplication is used to simulate the attention mechanism of the brain.

[0006] Regarding how to make matrix multiplication in neural networks perform efficiently, in addition to using dedicated neural network acceleration chips that optimize matrix operations through unique hardware architectures (such as those based on Ohm's law and Kirchhoff's laws), there are mainly two types of technical paths.

[0007] One is to use quantization technology to simplify matrix multiplication into pure addition and accumulation operations by quantizing weights or activation functions into binary or ternary. Quantized activation functions have led to the research of pulse neural networks, while quantized weights have given rise to binary neural networks, such as SpikeGPT and BitNet.

[0008] (BitNet) Hongyu Wang, Shuming Ma, etal. Bitnet: Scaling 1-bittransformers for large language models. arXiv preprint arXiv:2310.11453,2023. https: / / github.com / Beomi / BitNet-Transformers /

[0009] (BitNet) Shuming Ma, Hongyu Wang, etal. The era of 1-bit llms: Alllarge language models are in 1.58 bits. arXiv preprint arXiv:2402.17764,2024. https: / / github.com / kyegomez / BitNet

[0010] The other category is dedicated to replacing matrix multiplication with more basic operations. For example, AdderNet abandons matrix multiplication in both forward propagation and backpropagation, but its construction through addition or subtraction is limited to convolutional networks and is difficult to apply to other types of neural networks, especially large language models.

[0011] Although matrix multiplications in fully connected dense layers can be eliminated, matrix multiplications still exist in the self-attention mechanism. If the attention matrix is ​​quantized by three-value weights to eliminate matrix multiplications in the attention mechanism, the performance of neural networks or large language models will be greatly reduced, or even unable to converge.

[0012] Developing neural networks without matrix multiplication is becoming increasingly important for developing artificial intelligence with high energy efficiency, low training cost, and larger parameter scale. However, how to ensure that the performance of the neural network is maintained without degradation or almost without degradation after eliminating / reducing multiplication / matrix multiplication is a very challenging technical problem. Summary of the invention

[0013] In order to alleviate or partially alleviate the above technical problems, the solution of the present invention is as follows:

[0014] An information processing device includes several data processing submodules, at least one of which includes: a Token mixer, including an RNN structure, wherein the weight values ​​in the weight matrix in the RNN structure are taken from -1, 0, 1, or from -1, 1; a channel mixer, which is a gated linear unit, and the weight values ​​in the weight matrix in the gated linear unit are taken from -1, 0, 1, or from -1, 1; and the input and output of the Token mixer are added to obtain a first result, the input and output of the channel mixer are added to obtain a second result, and the first result is used to obtain the input of the channel mixer.

[0015] In a certain class of embodiments, the first result is used to obtain the input of the channel mixer, specifically: the first result is used as the input of the channel mixer after being normalized by a first-layer normalization module.

[0016] In certain embodiments, the second result is normalized by a second-layer normalization module and then used as the output of the at least one data processing sub-module.

[0017] In some embodiments, the output of at least one data processing sub-module serves as the input of another data processing sub-module.

[0018] In certain embodiments, the first data processing submodule among the plurality of data processing submodules receives a representation vector as an input vector; the representation vector is the output of the input sequence after being processed by the embedding and position encoding module.

[0019] In a certain class of embodiments, the input sequence is derived from one or more of the following types of information: images, texts, sound waves; the input sequence is converted into an input vector after passing through an embedding and position encoding module; the input vector is sent to a first information processing submodule as the input of the first information processing submodule, and the output of the first information processing submodule is used as the input of the next information processing submodule among the plurality of data processing submodules.

[0020] In certain embodiments, the information processing device generates at least one of the following information based on the input sequence: text, image, and sound wave.

[0021] In certain embodiments, the information generated by the information processing device is transmitted to the user device via a wired and / or wireless network, and the generated information is output via an output device of the user device.

[0022] In certain embodiments, the RNN structure is a network structure having different internal states that vary with time steps.

[0023] In certain embodiments, multiplication is not used when processing an input vector using a weight matrix in an RNN structure; or / and, multiplication is not used when processing an input vector using a weight matrix in a gated linear unit.

[0024] In certain embodiments, only addition and subtraction are used when processing an input vector using a weight matrix in an RNN structure; or / and, only addition and subtraction are used when processing an input vector using a weight matrix in a gated linear unit.

[0025] In certain embodiments, for the RNN structure, the hidden state h at the t-1 time step is t-1 , the input vector at time step t is x t ; Input vector x t After being processed by the first linear layer, the second linear layer, and the third linear layer, the first vector, the second vector, and the third vector are obtained respectively; at time t, the hidden state h t Equal to: the first vector and the hidden state h t-1 The Hadamard product between the two vectors is added to the Hadamard product between the second vector and the third vector; wherein the weight values ​​in the weight matrices of the first linear layer, the second linear layer, and the third linear layer are all taken from -1, 0, 1, or from -1, 1.

[0026] In certain embodiments, the weight matrix of the second linear layer is obtained by transforming the weight matrix of the first linear layer.

[0027] In some embodiments, the information processing device further includes a fourth linear layer and a fifth linear layer; the input vector x t After the fourth linear layer is processed, the fourth vector is obtained; the fourth vector and the hidden state h t The Hadamard product between them is processed by the fifth linear layer without an activation step to obtain the output vector of the t time step.

[0028] In a certain class of embodiments, through the training step, the weight values ​​in the weight matrix in the RNN structure are the weight values ​​after the training is completed; based on the weight values ​​after the training is completed, the information processing device is used to perform the reasoning process on the input sequence.

[0029] In certain embodiments, the information processing device further includes an encoder; and the plurality of data processing sub-modules serve as a decoder or a part of a decoder.

[0030] In certain embodiments, the plurality of data processing sub-modules all include the Token mixer and the channel mixer.

[0031] An information processing method includes several data processing sub-steps, at least one of which includes: processing a received input vector through a Token mixer, wherein the Token mixer includes an RNN structure, and the weight values ​​in the weight matrix in the RNN structure are taken from -1, 0, 1, or from -1, 1; adding the input and output of the Token mixer to obtain a first result; obtaining the input of a channel mixer according to the first result; adding the input and output of the channel mixer to obtain a second result; wherein the channel mixer is a gated linear unit, and the weight values ​​in the weight matrix in the gated linear unit are taken from -1, 0, 1, or from -1, 1.

[0032] In certain embodiments, the first result is normalized and then used as the input of the channel mixer.

[0033] In certain embodiments, the second result is normalized and used as the output result of the at least one data processing sub-step.

[0034] In certain embodiments, the output result of at least one data processing sub-step serves as input data to be processed by another data processing sub-step.

[0035] In a certain class of embodiments, the first data processing sub-step among the plurality of data processing sub-steps receives a representation vector as input data to be processed; the representation vector is the output of the input sequence after embedding and position encoding processing.

[0036] In a certain class of embodiments, the input sequence is derived from one or more of the following types of information: images, text, sound waves; the input vector serves as data to be processed in a first information processing sub-step, and the output result of the first information processing sub-step serves as input data to be processed in a next information processing sub-step among the plurality of data processing sub-steps.

[0037] In certain embodiments, the information processing method further includes the following steps: generating at least one of the following information based on the input sequence: text, image, sound wave.

[0038] In certain embodiments, the information processing method further includes the following step: transmitting the generated information to a user device via a wired and / or wireless network.

[0039] In certain embodiments, the RNN structure is a network structure having different internal states that vary with time steps.

[0040] In certain embodiments, multiplication is not used when processing an input vector using a weight matrix in an RNN structure; or / and, multiplication is not used when processing an input vector using a weight matrix in a gated linear unit.

[0041] In certain embodiments, only addition and subtraction are used when processing an input vector using a weight matrix in an RNN structure; or / and, only addition and subtraction are used when processing an input vector using a weight matrix in a gated linear unit.

[0042] In certain embodiments, for the RNN structure, the hidden state h at the t-1 time step is t-1 , the input vector at time step t is x t ; Input vector x t After being processed by the first linear layer, the second linear layer, and the third linear layer, the first vector, the second vector, and the third vector are obtained respectively; at time t, the hidden state h t Equal to: the first vector and the hidden state h t-1 The Hadamard product between the two vectors is added to the Hadamard product between the second vector and the third vector; wherein the weight values ​​in the weight matrices of the first linear layer, the second linear layer, and the third linear layer are all taken from -1, 0, 1, or from -1, 1.

[0043] In some embodiments, the weight matrix of the second linear layer is obtained by transforming the weight matrix of the first linear layer.

[0044] In certain embodiments, the input vector x t After the fourth linear layer is processed, the fourth vector is obtained; the fourth vector and the hidden state h t The Hadamard product between them is processed by the fifth linear layer without an activation step to obtain the output vector of the t time step.

[0045] In a certain class of embodiments, the information processing method further includes the following steps: obtaining, through a training step, weight values ​​in a weight matrix in the RNN structure after training; and executing the information processing method on an input sequence according to the weight values ​​after training.

[0046] A computer program product comprises a computer program or instructions, which, when executed or compiled by a processor, implements the steps of any one of the above information processing methods.

[0047] A computer-readable storage medium stores a computer program / instruction, which implements the steps of any one of the above information processing methods when executed or compiled by a processor.

[0048] The technical solution of the present invention has one or more of the following beneficial technical effects:

[0049] (1) No matrix multiplication, reducing various computing costs (power consumption, computing resources, etc.). More importantly, the present invention can not only converge, but also maintain or basically maintain accuracy, which makes it possible to have highly energy-efficient and widely applicable artificial intelligence.

[0050] (2) Follow the scaling law. This ensures that accuracy can be effectively improved by increasing computing power, model parameters, and data set size.

[0051] (3) It shows a steeper scaling law. The faster loss reduction rate indicates that the present invention can be more efficient when using additional computing resources to improve performance.

[0052] (4) It has a linear attention mechanism. The RNN structure replaces the traditional attention mechanism to improve the speed of hardware processing input sequences (obtained through transformations of text, images, speech, etc.) and reduce various costs of training and reasoning.

[0053] (5) Lower memory usage and latency.

[0054] In addition, other beneficial effects of the present invention will be mentioned in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a comparison chart of the loss performance of the present invention and the prior art improvement attempt without matrix multiplication;

[0056] Figure 2 is a schematic diagram of an information processing device of the present invention;

[0057] Figure 3 It is a schematic diagram of an embodiment proposed by the present invention;

[0058] Figure 4 It is an information processing flow chart of the RNN structure of the present invention;

[0059] Figure 5 It is a comparative example diagram of the present invention and the prior art in processing matrix multiplication;

[0060] Figure 6 It is the information processing flow chart of GRU without matrix multiplication;

[0061] Figure 7 It is the information processing flow chart of GLU without matrix multiplication;

[0062] Figure 8 It is a comparison chart of the scale law verification results;

[0063] Fig. 9 This is a performance comparison chart of different LLMs and configurations;

[0064] Fig.10 This is a schematic diagram of the implementation scheme for running a token without matrix multiplication;

[0065] Fig.11 This is a pseudocode description diagram that combines quantized RMSNorm and BitLinear operations to improve the hardware efficiency of the BitLinear layer. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] In the description of the present invention, unless otherwise specified, "plurality" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items.

[0068] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order.

[0069] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" and "for example" is intended to present related concepts in a specific way for easy understanding.

[0070] For some modules and steps disclosed in the present invention, those skilled in the art can usually implement them as specific embodiments of hardware, software or a combination of software and hardware according to actual needs. The present invention aims to disclose and protect the above different implementation methods.

[0071] In addition, some prior arts mentioned in the present invention are fully incorporated into this application by reference, and are taken as part of the technical content disclosed in this application and as the basis for subsequent modifications of this application.

[0072] In the field of natural language processing, such as translation tasks and generative question-answering tasks, both recurrent neural networks (RNNs) and convolutional neural networks (CNNs) have limitations when processing long sequences: RNNs are prone to gradient vanishing or gradient exploding problems when processing long sequences, and have difficulty capturing long-distance dependencies and cannot be computed in parallel. CNNs usually require inputs to have a fixed size, and when processing variable-length sequences, in order to make the sequences have the same length, shorter sequences need to be padded, which results in low computational efficiency.

[0073] Transformer abandons RNN and proposes a self-attention mechanism that can calculate the degree of association between each position in the sequence and other positions, and weight the information in the sequence according to these degrees of association. The multi-head attention mechanism is an extension of the self-attention mechanism. It focuses on different representation subspaces at the same time to better capture different relationships and features in the input sequence.

[0074] The complete architecture of Transformer includes encoder and decoder, and the QKV (Query, Key, Value) structure in the attention mechanism is the most basic and important unit. With the development of technology, GPT, which was born later, only retains the decoding part (decode-only). This architecture is very suitable for generative tasks.

[0075] (Transformer) Vaswani, Ashish, Noam M. Shazeer, etal. Attention isAll you Need. Neural Information Processing Systems (2017).

[0076] (GPT) Radford, Alec and Karthik Narasimhan. Improving LanguageUnderstanding by Generative Pre-Training. (2018).

[0077] A large number of expensive matrix operations result in high computational costs for LLMs to process input information. Especially for a new architecture like GPT4o that supports multimodal information processing such as real-time video / images, voice, and text signals, it is increasingly urgent to respond to massive requests within 200-300 milliseconds, improve inference efficiency, and reduce training / inference power consumption, latency, and computing resource consumption.

[0078] Some binary and ternary weight solutions (BitNet) have been disclosed in the prior art. However, for the Transformer based on the attention mechanism, in order to reduce power consumption and computing cost, the most intuitive approach is to binarize or ternary at least two matrices in the QKV architecture.

[0079] Figure 1 This is a comparison chart of the loss performance of the present invention and the prior art improvement scheme without matrix multiplication. To explore the feasibility of this method, based on the assumption that the weights of all dense layers become ternary values, the values ​​of Q and K are quantized to obtain a ternary attention map, which further eliminates the multiplication in the self-attention and obtains the Transformer++ without matrix multiplication. Unfortunately, this scheme cannot converge.

[0080] (Transformer++) Thapak Prakhar, Prodip Hore. Transformer++. arXivpreprint arXiv:2003.04974,2020.

[0081] In other words, if the matrix multiplication operation is removed from the Transformer based on the binarization and ternaryization techniques in the existing technology, the final processing performance will become very poor. The possible reason is that the activation contains outliers that are important to the network performance but difficult to effectively quantify.

[0082] Figure 2 Schematic diagram of the information processing device of the present invention. The data processing device proposed in the present invention includes a plurality of (N=12 for example) data processing submodules, and these unique data processing submodules are the core of the present invention, and the core lies in that the data processing submodule proposed in the present invention includes a token mixer and a channel mixer. For example, the data processing device can be used to translate the input text.

[0083] The information processing device may be a hardware entity including a dedicated integrated circuit, a GPU, an FPGA or a combination thereof, or a part of a hardware entity. These information processing devices may be configured as a part of a larger entity, for example, the information processing device may be configured as an information processing component of an entity such as a car, a robot, an unmanned aerial vehicle, or an intelligent terminal. The information processed by the information processing device may come from the environment in which the information processing device is located in real time, or may be obtained through Internet transmission, or even be stored in a local device after collection, transmission, and other links.

[0084] For ease of understanding and description, the present invention often uses mathematical language commonly used in the art to accurately describe or display certain technical details of the present invention. These mathematical symbols (such as input vectors, weight matrices) often imply or refer to corresponding data, hardware, information flow, information conversion processes, etc., and should never be interpreted as abstract pure mathematical symbols or formulas, and the subject matter of the present invention should not be understood as a pure mathematical formula or physical law.

[0085] After understanding the present invention, those skilled in the art can implement the technical solution of the present invention as hardware, software or a combination of software and hardware. After the disclosure of the present invention, for example, an efficient artificial intelligence entity can be obtained by using FPGA, GPU, ASIC, etc. as hardware carriers. The present invention configures or changes or converts the objective world's materials and configures or changes the information processing method to perceive, collect, process, and respond to information from the objective world, thereby promoting the benefits of human beings from the reasonable or optimal use of information.

[0086] Further, for at least one data processing submodule: the input and output of the Token mixer are added to obtain a first result, the input and output of the channel mixer are added to obtain a second result, and the first result is used to obtain the input of the channel mixer.

[0087] Obviously, when the present invention introduces each data processing submodule, it implies the existence of corresponding data processing substeps. As needed, these submodules and substeps generally involve one or more operations such as data reading, computing, sending, and storing. The present invention aims to introduce these key processes or steps. As to which operations are to be taken and how to implement these operations, those skilled in the art may adopt different technical implementation details and means, and the present invention is not limited to a specific implementation method.

[0088] On the basis of the data processing submodule of the present invention, those skilled in the art may add other functions or modules to fully utilize one or several technical advantages of the present invention.

[0089] For example, an encoder is added on the basis of a decoder-only architecture to obtain a performance advantage at least in the decoding stage.

[0090] Furthermore, the data processing submodule of the present invention can be implemented as an application-specific integrated circuit (ASIC) through a large number of logic gate circuits, or can be run on a hardware platform, such as an FPGA or a GPU, in the form of a software module. The ASIC embodiment can process the input sequence more efficiently but has a longer development cycle, while the GPU embodiment has a shorter development cycle and is easier to deploy and apply.

[0091] Furthermore, a linear layer and a softmax layer may be added after the decoding output to finally output probability information and / or an output sequence.

[0092] Figure 3 FIG. 1 is a schematic diagram of an embodiment of the present invention. First, the input sequence is embedded and the position encoding module is used to obtain a representation vector of the input sequence.

[0093] For example, for natural language processing related tasks, a word is used as input, the word is embedded, and then added to the position embedding to obtain a representation vector, which serves as an input vector of a data processing device.

[0094] In the present invention, the input sequence or representation vector / input vector may be derived from one or more of the following types of information: images (including videos), text (including speech-to-text), and sound wave information that is not converted into text. Furthermore, by way of example, these types of information may include biomedical data, personal health data, financial data, meteorological data, or various types of transaction data.

[0095] For example, the data processing device in the present invention can be a server or a battery-driven device. Among them, the server based on the present invention can provide various AI services, such as generating text (question and answer, text summarization, copywriting generation, etc.), generating images, generating videos, generating sounds, generating codes, as well as dubbing, chatting, health consultation, recommendation system, etc. Specifically, the above services can be provided to networked users through the Internet based on the server-client model.

[0096] Similar local AI services can also be provided for battery-driven devices based on the present invention, such as mobile phones, tablets, personal computers, robots, drones, cars, etc. Preferably, support for multimodal information input is the key to obtaining general artificial intelligence for autonomous systems such as robots. The energy efficiency advantage of the present invention has low requirements for computing resources and is particularly suitable for battery-driven devices that are sensitive to power consumption and cost.

[0097] For example, the information processing device generates at least one of the following information according to the input sequence: text, image, sound wave.

[0098] Furthermore, the information generated by the information processing device is transmitted to the user device via a wired or / and wireless network, and the generated information is outputted via an output device of the user device. For example, the generated information can be sent to the user device via the Internet. After receiving the generated information, the user device displays or plays the generated information via a speaker or / and a screen.

[0099] The above are examples of the application scope or manner of the information processing device and the information processing method of the present invention, and the present invention is not limited to them.

[0100] For the first data processing submodule, the input vector is used as the input of the Token mixer, and the output and input of the Token mixer are added as the input of the first layer normalization module. The output of the first layer normalization module is used as the input of the channel mixer, and the input and output of the channel mixer are added as the input of the second layer normalization module. This connection method of adding module input and module output is also called residual connection.

[0101] Sequentially, for the next data processing submodule, its input is the output of the previous data processing submodule.

[0102] As one of the improvements of the present invention, the Token mixer of the present invention is used to mix information in the time dimension. The present invention abandons the QKV structure at the core of the Transformer and replaces it with a Token mixer based on the RNN structure. On the one hand, the quadratic complexity of the attention mechanism in the Transformer requires a lot of computing and memory in tasks involving long sequences and limited resources. For this reason, some people even propose flashAttention, which draws on the idea of ​​hierarchical storage, to improve training performance; on the other hand, based on the inventor's research findings, it shows that the attention mechanism is not as important as generally believed in the field. By replacing the attention mechanism with an RNN structure, such as masked multi-head self-attention, reasoning and training can be performed with only linear computational complexity, which is of great significance and value in alleviating the high computational cost of artificial intelligence.

[0103] As a further improvement of the present invention, the RNN is a scalable RNN. The scaling law is extremely important for the performance of large models such as LLMs. This is also one of the important reasons why Transformer can achieve significant technical progress. It points out that for the amount of calculation C, the number of model parameters N and the size of the data set D, when not restricted by the other two factors, the accurate performance L of information processing has a power law relationship with each factor. Therefore, improvements to Transformer and a new information processing framework should meet or maintain the advantages of the scaling law, which is quite technically challenging.

[0104] For example, a scalable RNN unit can refer to the RWKV document disclosed by the inventor. The time mixer is RNN-like. The so-called RNN-like means that the output state at time t is used as the input state at time t+1 in the network.

[0105] RWKV explains how to apply RNN as a time mixer to Transformer to obtain a scalable, linearly complex (in terms of time and space) sequence processing task solution. The present invention hereby incorporates the entire content and features of the document into this application by reference and as part of the technical content disclosed in this application.

[0106] (RWKV) Bo Peng, Eric Alcaide, et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023.

[0107] Although RWKV reduces the original Transformer quadratic complexity to linear complexity, matrix multiplication still exists. As a further improvement of the present invention, the scalable RNN is an RNN without matrix multiplication. More importantly, this RNN without matrix multiplication can still maintain excellent network performance.

[0108] Generally, the Token mixer in the present invention has an RNN structure. The RNN structure is a basic unit of information processing, including a forget gate (first linear layer) and an input gate (second linear layer). The RNN structure involves a new information generation step and a hidden state update step, wherein the initial hidden state ,in represents an all-zero vector.

[0109] For the RNN structure of the present invention, there is a hidden state h at the t-1 time step t-1 , the input vector at time step t is x t ; Input vector x t After the first linear layer (W f 、b f ), the second linear layer (W i 、b i ) and the third linear layer (W c 、b c ) are processed, the first vector (f t ), the second vector (i t ) and the third vector (c t ); At time t, the hidden state h t Equal to: the first vector and the hidden state h t-1 The Hadamard product between the two vectors is added to the Hadamard product between the second vector and the third vector; wherein the weight values ​​in the weight matrices of the first linear layer, the second linear layer, and the third linear layer are all taken from -1, 0, 1, or from -1, 1.

[0110] Specifically, the forget gate is used to control the degree of forgetting of the hidden state at the previous moment. The information processing flow of the forget gate can be described in mathematical language as , where f t is the forget gate output at the current moment (i.e., time step t), x t is the input vector at the current moment, W f is the weight matrix of the forget gate, b f is the bias matrix of the forget gate, is the activation function.

[0111] The input gate is used to control the influence of the current input on the hidden state. That is, , where i t is the input gate output at the current moment, x t is the input vector at the current moment, W i is the weight matrix of the input gate, b i is the bias vector of the input gate, is the activation function.

[0112] As for the new information generation step, this step is used to generate new information at the current moment. That is, , where c t is the new information at the current moment, x t is the input vector at the current moment, W c is the weight matrix of new information, b c is the bias vector of the new information, is the activation function.

[0113] For the hidden state update step, this step is used to update the hidden state at the current moment. That is, , where h t is the hidden state at the current moment, f t is the forget gate output at the current moment, h t-1 is the hidden state of the previous moment (i.e., time step t-1), i t is the input gate output at the current moment, c t It is new information at the present moment.

[0114] refer to Figure 4 This is the information processing flow chart of the RNN structure of the present invention. The present invention uses a binary or ternary weight matrix W f , W i , W c , so when the input vector x tWhen "multiplied" by the binary or ternary weight matrix, the traditional matrix multiplication operation can be eliminated, leaving only addition and subtraction. Finally, the structure of the aforementioned RNN can be expressed as follows (here represents a binary accumulation operation or a ternary accumulation operation, and some examples below may only represent one of the operations):

[0115]

[0116] in are all non-linear activation functions, and may be partially or completely the same, for example, they are taken from Sigmoid, SiLU, ReLU, tanh, etc. For example, Figure 4 Shown is as well as This is just one embodiment, and the present invention is not limited to this.

[0117] In the above RNN information processing unit, h t or t-1 It is the internal state that does not change with different time steps (t, t-1).

[0118] Furthermore, the weight matrix of the second linear layer is obtained by transforming the weight matrix of the first linear layer. , that is, the weight matrix of the second linear layer is obtained by transforming the weight matrix of the first linear layer.

[0119] For the entire information processing device, the above RNN architecture is the basic means to efficiently process the input sequence. For the input vector x at time step t (current moment) t For example, through the above information processing unit and combined with the hidden state h at the t-1 time step (previous moment) t-1 , we can directly get the corresponding hidden state h at the current moment t .

[0120] The binarized or ternary weight matrix refers to a weight matrix that only includes binary (values) or ternary (values) of {-1, 1} or {-1, 0, 1}.

[0121] refer to Figure 5, which shows a comparative example diagram of the present invention and the prior art processing matrix multiplication. In this figure, not only the principle of the self-attention mechanism is shown, but also the information processing schematic diagram of the linear GRU without matrix multiplication, the GLU without matrix multiplication, and the traditional matrix multiplication, the scaled three-value accumulation schematic diagram, and the RMS normalization (Norm) method are shown. These are schematic diagrams of the information flow processed by each information processing module in the information processing device. For ease of understanding, it is described or displayed in mathematical language. After understanding the present invention, those skilled in the art can implement the technical solution of the present invention as hardware, software, or a combination of software and hardware. For example, by using FPGA and GPU as hardware carriers, an efficient artificial intelligence entity is obtained.

[0122] As a further embodiment of the RNN structure of the present invention, the present invention also discloses a gated recurrent unit (GRU) without matrix multiplication. Among the many architectures of RNN, GRU has the advantages of simplicity, high efficiency, and simple structure (less gating). The information processing process of GRU can be described by the following mathematical language:

[0123]

[0124] in, is the input vector at time step t, is the hidden state vector of the previous time step (i.e., time step t-1), is the reset gate vector, is the forget gate vector, is a candidate hidden state, is the final hidden state vector at time step t, is the output vector at time step t, and are the learnable weight matrix and bias vector, is the Sigmoid activation function, Denotes the Hadamard product (element-wise).

[0125] The input vectors, output vectors, hidden state vectors, etc. of the present invention are all names of some specific data sets stored in specific locations in the information processing device and information processing method of the present invention, and they are representations of information.

[0126] In one embodiment of the present invention, the weight matrix W associated with the hidden state is removed. cc , W hr , W hf The activation tanh(.) between the hidden state and the kernel state can be computed in parallel like the Transformer.

[0127] Figure 6is the information processing flow chart of the GRU without matrix multiplication. For the Token mixer, in one embodiment of the present invention, a GRU without matrix multiplication can be constructed in the following way, which is a basic information processing unit. For this information processing unit, x t is the input vector at time step t, is the final output vector at time step t. The mathematical description of the information processing flow of the GRU unit without matrix multiplication is as follows:

[0128]

[0129] Among them, W f , W c , W g , W o is the three-value (element) weight, b f 、b c 、b g 、b o is the bias vector, f t is the forget gate output, is the Sigmoid activation function, c t is the input vector, is the SiLU activation function, h t is the hidden state at time step t, h t-1 is the hidden state at time step t-1, is the output gate, is the intermediate output. In addition, the initial hidden state h0 is set to 0. represents the Hadamard product, Represents matrix multiplication with three-valued weights.

[0130] (HGRN)Zhen Qin, Songlin Yang, et al. Hierarchically gated recurrent neural network for sequence modeling. Advances in Neural InformationProcessing Systems, 36, 2024.

[0131] In addition, in the GRU of the present invention, the fourth linear layer (W g 、b g ), the input vector is processed by the fourth linear layer to obtain the fourth vector (g t ). The fourth vector is the same as the hidden state h at time step t. t The Hadamard product between ), after a fifth linear layer without activation step (Wo 、b o ) is processed to obtain the output vector O of the GUR at time step t t .

[0132] Optionally, similar to HGRN, the present invention further performs a cummax operation to make the forget gate value of a higher layer closer to 1.

[0133] The GRU information processing unit of the present invention can be regarded as a special case of the aforementioned RNN. t When processing, it is limited to 1-f t , rather than i, which has a higher degree of freedom t ; The hidden state h at the current moment t It is not directly used as the output vector, but is combined with g t Perform a linear transformation after performing the Hadamard product.

[0134] Figure 7 This is an information processing flow chart of a GLU without matrix multiplication. For the channel mixer, the present invention can use a gated linear unit (GLU), which is also a basic information processing unit and is used by many LLMs, such as the currently popular LLaMa and the aforementioned RWKV.

[0135] (LLaMa) Hugo Touvron, Thibaut Lavril, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.

[0136] Here, these documents are introduced into the present invention by reference and are regarded as a part of the technical content disclosed in the present invention.

[0137]

[0138] in represents the Hadamard product, Represents matrix multiplication with three-valued or two-valued weights.

[0139] GLU is mainly configured to perform the following steps:

[0140] (1) The input vector of time step t Through the weight matrix Upscaling, we get .

[0141] (2) Through g tElement-wise gating u t , and then through the nonlinear activation function Processing, obtain p t .

[0142] (3) t Through the weight matrix W d , downscaling is d t , restore to its original size.

[0143] In one embodiment of the present invention, as with LLaMa, the upsampling factor is set to 8d / 3, keeping the total number of parameters of GLU at 8d 2 .

[0144] The channel mixer only contains operations for dense layers, which can be completely replaced by a single-bit operation, making it an addition operation. By using three-value weights in the BitLinear module, the present invention can eliminate the need for expensive matrix multiplications, making channel mixing more computationally efficient while maintaining its effectiveness in mixing information across channels.

[0145] In a standard dense layer, the input and the weight matrix The matrix multiplication between can be expressed as:

[0146]

[0147] in, is the output. To avoid using standard multiplication-based dense layers, similar to BitNet, the BitLinear module can be used to replace dense layers containing matrix multiplication. The BitLinear module uses three-valued weights to convert matrix multiplication operations into pure addition operations with accumulation, that is, ternary / three-value accumulation. When using three-valued weights, the elements in the weight matrix W are restricted to values ​​in the set {-1, 0, +1}. Let Represents a three-value weight matrix. Matrix "multiplication" with three-value weights can be expressed as:

[0148]

[0149] in is the output, Represents matrix multiplication with three-value weights, which can be simplified to accumulation. It can only take values ​​in {-1, 0, +1}, so the "multiplication" operation in the matrix "multiplication" of the three-valued weight can be replaced by a simple addition or subtraction operation:

[0150]

[0151] Therefore, the three-value weight multiplication in the present invention can be described by the following mathematical expression:

[0152] .

[0153] For the specific methods of binarization and ternary conversion, reference may be made to two articles of BitNet, which are the background technology of the present invention, and which respectively introduce the binarization and ternary conversion technologies. These articles are incorporated into the present application by reference and serve as part of the technical content disclosed in the present invention.

[0154] For example, a specific implementation of ternary conversion can be as follows: during the training process, the weights are first quantized to {−1, 0, 1} using the absmean quantization function, which scales the weight matrix by the average absolute value and rounds each element to {−1, 0, +1}. The mathematical language description of the process is as follows:

[0155]

[0156] Where n and m are the number of rows and columns of W. After weight quantization, the activation value is also quantized to 8-bit precision, just like BitNet. You can use absmax quantization to make the activation value be placed In the range, given that b is the number of bits, and Q b =2 b-1 :

[0157]

[0158] in, is a minimum value used to prevent overflow when clipping.

[0159] In order to preserve variance and maintain numerical stability after quantization, you can use RMSNorm before activating quantization, which is also used in BitNet:

[0160]

[0161] Among them, Q b is the maximum activation value, is the weighted mean.

[0162] For the information processing device of the present invention, in order to achieve accurate processing of the input sequence, it is necessary to be trained with a large number of data sets. The training stage is the stage of learning various parameters such as weight matrices and bias vectors. With these trained parameters, the entire data processing device can perform intelligent information processing on the input sequence input by the application scenario, such as a personal health assistant.

[0163] In addition, the present invention also discloses a computer program product, including a computer program or instructions, which implements the steps of the method described in any of the preceding items when executed or compiled by a processor.

[0164] In addition, the present invention also discloses a computer-readable storage medium having a computer program / instruction stored thereon, characterized in that the computer program / instruction implements the steps of the method described in any of the preceding items when executed or compiled by a processor.

[0165] In the training phase, in order to handle non-differentiable functions such as Sign and Clip functions during back-propagation, a straight-through estimator (STE) can be used as a substitute function for the gradient. STE allows the gradient to flow through the network without being affected by these non-differentiable functions, thereby enabling the training of quantized information processing modules. This technology is widely used in binary neural networks (BNNs) and spiking neural networks (SNNs), where STE is used as one of the substitute functions.

[0166] Further, by implementing the present invention as an LLM device running without matrix multiplication, subsequent experiments will evaluate the performance of the LLM without matrix multiplication. When training an LLM with ternary weights, using the same learning rate as a conventional LLM may result in too small updates, but has no effect on the clipping operation. This prevents effective updates of weights and results in biased gradients and update estimates based on ternary weights.

[0167] To address this challenge, a common practice is to use a larger learning rate when training binary or ternary weight LLMs, as it helps converge faster. In the experiments, the same learning rate is maintained for 370M and 1.3B LLMs.

[0168] Specifically, for Transformer++, a learning rate of 3e-4 can be used, while for the LLM without matrix multiplication of the present invention, the experiments use learning rates of 4e-3, 2.5e-3, and 1.5e-3 for 370M, 1.5B, and 2.7B network scale LLMs, respectively. These learning rates are selected based on the most effective hyperparameter sweeps to achieve faster convergence during training.

[0169] When training a traditional Transformer, a cosine learning rate scheduler is usually used and a minimum learning rate is set, usually 0.1 times the initial learning rate. This approach is followed when training the full-precision Transformer++.

[0170] However, for the LLM without matrix multiplication of the present invention, its learning dynamics are different from the traditional Transformer and require a different learning strategy. First, keep the cosine learning rate scheduler, and then reduce the learning rate in the middle of the training phase, such as halving the learning rate. It is worth mentioning that the inventors observed that in the final stage of training, when the learning rate of the network is close to 0, the loss is significantly reduced, showing an S-shaped loss curve.

[0171] In the experimental part, the proposed LLM without matrix multiplication is mainly tested on medium-sized language tasks. Two variants of LLM without matrix multiplication are compared with a recurrent high-level Transformer (Transformer++ based on LLaMA-2), and the architecture has two versions: 370M and 1.3B parameters.

[0172] To ensure fair comparison, LLMs with 370M and 1.3B parameters were pre-trained on 1.5 billion and 100 billion tokens on the SlimPajama dataset, respectively. All experiments used the flash-linear-attention framework and used the LLaMA tokenizer (with a vocabulary of 32,000) and the optimized triton kernel.

[0173] Scaling laws for neural networks state that as the size of the training set and the size of the parameters increase, the error decreases exponentially, which provides the potential for guaranteed performance in deep learning and LLMs as training becomes increasingly expensive. In LLM training, a widely adopted best practice is to first test scalability with a smaller LLM, as scaling laws kick in. The technical report for GPT-4 revealed that an LLM with 1 / 10,000 the size of the final LLM parameters can still accurately estimate the performance of the final LLM.

[0174] Figure 8 It is a comparison chart of the scaling law verification results. The inventors evaluated the scaling laws of 370M, 1.3B and 2.7B parameter sizes in Transformer++ and the matrix-free LLM of the present invention. For conservative comparison, each operation between the LLM without matrix multiplication and Transformer++ is treated equally. But please note that all weights and activations in Transformer++ use BF16, while the BitLinear layer in the LLM with matrix multiplication uses ternary parameters and BF16 activation. Therefore, the average operation in MatMul-free LM is computationally cheaper than Transformer++.

[0175] To gain more insight into the scaling behavior, curves for both LLMs are fitted here. An important finding is that the scaling law of the matrix-free LLM of our invention exhibits a steeper decline compared to Transformer++. This faster reduction in loss suggests that the matrix-free LLM is more efficient in utilizing additional computational resources to improve performance. Therefore, the scaling law curve of the matrix-free LLM of our invention is expected to intersect the scaling law curve of Transformer++ at approximately 1023 FLOPs. This computational scale is roughly equivalent to the training FLOPs required to train LLaMA-3 8B (trained with 150 billion tokens) and LLaMA-2 70B (trained with 20 billion tokens), indicating that the matrix-free LLM can gain performance gains when scaled up.

[0176] Following the setting of BitNet, we also evaluated the zero-shot performance of these LLMs on a series of language tasks, including ARC-Easy, ARC-Challenge, Hellaswag, Winogrande, PIQA, and OpenbookQA, and the results are shown in Table 1. All evaluations are performed using the LLM evaluation toolkit.

[0177] Table 1: Zero-sample accuracy comparison table

[0178]

[0179] Fig. 9 Performance comparison graphs of different LLMs and configurations are shown. Among them, sub-graphs (a) and (b) show the training performance comparison between VanillaBitLinear and FusedBitLinear in terms of time and memory consumption as a function of batch size. Sub-graph (c) illustrates the effect of learning rate on the training loss of LLM without matrix multiplication. Sub-graph (d) compares the inference memory consumption and latency of LLM without matrix multiplication and Transformer++ between different model sizes.

[0180] Compared to the Transformer++ baseline, the matrix-free LLM achieves competitive performance on all tasks, demonstrating its effectiveness in zero-shot learning despite the absence of matrix multiplication operations and the lower memory required for ternary weights. Notably, the 2.7B matrix-free LLM outperforms Transformer++ on ARC-Challenge and OpenbookQA, while maintaining comparable performance on other tasks. The performance gap between the matrix-free LLM and Transformer++ narrows as the model size increases, i.e., it follows the scaling law. These results show that the matrix-free architecture is able to achieve strong zero-shot performance on a wide range of language tasks, from question answering, commonsense reasoning, to physical understanding.

[0181] The inventors evaluated the FusedBitLinear and VanillaBitLinear implementations proposed in the present invention in terms of training time and memory usage. Fig. 9 For each experiment, the input size and sequence length are set to 1024. All experiments are performed using NVIDIA A100 80GB GPU.

[0182] During training, the sequence length and batch dimensions are flattened, making the effective batch size the product of these dimensions. Experiments show that the fusion operator benefits from larger batch sizes, which can increase training speed and reduce memory consumption. 8 The training speed of the 1.3B parameter size LLM is improved from 1.52 seconds per iteration to 1.21 seconds, a 25.6% improvement over the vanilla implementation. In addition, memory consumption is reduced from 82GB to 32GB, a 61.0% reduction in memory usage. The performance achieved by the FusedBitLinear implementation improves significantly with increasing batch size, allowing more samples to be processed simultaneously and reducing the total number of iterations.

[0183] For example, in one embodiment, compared with a benchmark without the technical solution of the present invention, running the technical solution of the present invention on a GPU reduces the amount of data stored in the training phase by 61%, and reduces the amount of data stored in the inference phase by more than 10 times by using optimized kernels.

[0184] Fig. 9 Subfigure (d) of Figure 4 compares the GPU inference memory consumption and latency of the proposed matrix-free LLM and Transformer++ for various model sizes. In the matrix-free LLM, BitBLAS is used for acceleration in the experiment to further improve the efficiency. The evaluation is performed with a batch size of 1 and a sequence length of 2048.

[0185] The matrix-free LLM consistently exhibits lower memory usage and latency compared to Transformer++ across all model sizes. For a single layer, the matrix-free LLM requires only 0.12GB of GPU memory and has a latency of 3.79ms, while Transformer++ consumes 0.21GB of memory and has a latency of 13.87ms. The memory and latency advantages of the matrix-free LLM become more pronounced as the model size increases. It is worth noting that for model sizes greater than 2.7B, the results are simulated using randomly initialized weights. For the largest model size of 13B parameters, the matrix-free LLM uses only 4.19GB of GPU memory and has a latency of 695.48ms, while Transformer++ requires 48.50GB of memory and has a latency of 3183.10ms. These results highlight the efficiency gains achieved by the matrix-free LLM, making it a promising approach to LLM, especially during inference.

[0186] The learning rate is a critical hyperparameter in LLM training. The model becomes more sensitive to the learning rate in the ternary / binary weight system. To determine the optimal learning rate, the experiment uses a 370M model and a batch size of 50k tokens to search in the range of 1.5e-3 to 3e-2. The results of this search are shown in Figure 2. Fig. 9 As shown in sub-figure (c).

[0187] The results show that the final training loss decreases monotonically as the learning rate increases from 1.5e-3 to 1e-2. Only when the learning rate exceeds 2e-2 does the model exhibit instability. This finding suggests that previous work using ternary weights, such as BitNet using a learning rate of 1.5e-3, may not be optimal and higher learning rates may lead to better performance. These findings are consistent with observations from DeepseekLLM, which found that the optimal learning rate for traditional LLM is actually larger than the value typically reported in most LLM training settings. It was also observed that models trained with larger learning rates at the beginning of the training process exhibited a faster decrease in training loss in the later stages of training compared to those trained with smaller learning rates.

[0188] In addition, the present invention can also be implemented as a neural network accelerator. The neural network accelerator can be a GPU-based accelerator or an ASIC-based accelerator. The present invention does not limit the specific implementation of the neural network accelerator. As an example, Fig.10 Shows a schematic diagram of the implementation scheme for running a token without matrix multiplication.

[0189] By way of example, the neural network accelerator includes four functional units: a row-wise operation unit, a root mean square unit, an access unit, and a three-value matrix multiplication unit. They all allow simple out-of-order execution. A custom assembler can be written for the custom instruction set to convert the assembly file into an instruction ROM. Table 2 exemplarily shows the custom instruction set of the present invention.

[0190] Table 2: Custom instruction set

[0191]

[0192] Register routing delegates incoming instructions to available GPU thread groups. The register file consists of multiple registers, each stored in its own thread local memory. Each register can only be accessed by one thread block at a time. If an instruction requests access to a busy compute unit or register, the thread will be suspended until the unit or register is freed. If two instructions do not block each other, they can be executed in parallel by different thread blocks.

[0193] The RMS unit uses a specialized parallelization algorithm to improve accuracy and runs in three stages. The first stage loads the target vector into thread-local registers and uses a lookup table to perform a square operation on each element. The second stage uses the parallel computing power of the GPU to divide and conquer the mean of adjacent vector elements and generate an RMS result. The third stage normalizes each element by dividing it by the RMS result through the GPU thread. By leveraging the parallel capabilities of the GPU, the divide-and-conquer parallel mean calculation can significantly reduce calculation time and rounding errors.

[0194] The three-value matrix multiplication unit receives the global memory address of the three-value matrix, and then executes the TMATMUL instruction to the specified vector. In one embodiment of the present invention, the three-value matrix is ​​stored in the global memory of the GPU. When the TMATMUL instruction is run, the matrix in the global memory is loaded into the shared memory through the cache, and the ternary addition operation is performed in parallel by multiple threads to accelerate the operation. At present, the three required TMATMUL instructions occupy most of the total execution time. Further optimization can improve the execution efficiency of the TMATMUL instruction by introducing more thread parallelization and shared memory cache.

[0195] Furthermore, BitNet shows that a stable triplet layer requires adding an extra RMSNorm before the input of the BitLinear module. However, the vanilla implementation of BitNet is not efficient.

[0196] Modern GPUs have a memory hierarchy with a large global high-bandwidth memory (HBM) and smaller, faster shared memory (SRAM), and the implementation of BitNet introduces many I / O operations: reading the previous activation into SRAM for RMSNorm, writing it back for quantization, reading it again for quantization, storing it, and then reading it again for linear operations.

[0197] To solve this inefficiency problem, the present invention can achieve reading activation only once and applying RMSNorm and quantization as a fused operation in SRAM.

[0198] Fig.11 The method of improving the hardware efficiency of the BitLinear layer / module by fusing quantized RMSNorm and BitLinear operations is introduced to those skilled in the art in the form of pseudocode. Optimizing the use of SRAM to reduce the I / O cost of HBM can significantly speed up the calculation. Due to the number of ternary weights and element-by-element operations, the activations in this model have a larger memory footprint than the weights, so the optimization work in this embodiment is mainly focused on the activations.

[0199] In order to better illustrate the present invention, numerous specific details are provided in the above specific embodiments. It should be understood by those skilled in the art that the present invention can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present invention.

[0200] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. An information processing device, comprising a plurality of data processing submodules, characterized in that: At least one data processing submodule includes: A token mixer, comprising an RNN structure, wherein the weight values ​​in the weight matrix in the RNN structure are taken from -1, 0, 1, or from -1, 1; The channel mixer is a gated linear unit, and the weight values ​​in the weight matrix of the gated linear unit are taken from -1, 0, 1, or from -1, 1; and, The input and output of the Token mixer are added to obtain a first result, the input and output of the channel mixer are added to obtain a second result, and the first result is used to obtain the input of the channel mixer.

2. The information processing device according to claim 1, characterized in that: The first result is used to obtain the input of the channel mixer, specifically: the first result is used as the input of the channel mixer after being normalized by the first layer normalization module.

3. The information processing device according to claim 2, characterized in that: The second result is normalized by the second-layer normalization module and serves as the output of the at least one data processing sub-module.

4. The information processing device according to claim 3, characterized in that: The output of at least one data processing submodule is used as the input of another data processing submodule.

5. The information processing device according to claim 4, characterized in that: The first data processing submodule among the plurality of data processing submodules receives a representation vector as an input vector; the representation vector is an output of the input sequence after being processed by the embedding and position encoding module.

6. The information processing device according to claim 5, characterized in that: The input sequence is derived from one or more of the following types of information: images, text, sound waves; The input sequence is converted into an input vector after being embedded and encoded by the positional encoding module; The input vector is sent to the first information processing submodule as the input of the first information processing submodule, and the output of the first information processing submodule is used as the input of the next information processing submodule among the plurality of data processing submodules.

7. The information processing device according to claim 6, characterized in that: The information processing device generates at least one of the following information according to the input sequence: text, image, and sound wave.

8. An information processing method, comprising a plurality of data processing sub-steps, characterized in that: At least one data processing sub-step includes: Processing the received input vector through a Token mixer, wherein the Token mixer includes an RNN structure, and weight values ​​in a weight matrix in the RNN structure are taken from -1, 0, 1, or from -1, 1; Adding the input and output of the Token mixer to obtain a first result; Obtaining an input of a channel mixer according to the first result; Adding the input and output of the channel mixer to obtain a second result; wherein, The channel mixer is a gated linear unit, and the weight values ​​in the weight matrix of the gated linear unit are taken from -1, 0, 1, or from -1, 1.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor or compiled and executed, the steps of the method according to claim 8 are implemented.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed or compiled by a processor, the steps of the method according to claim 8 are implemented.

Images