Transformer compression method, device, equipment and medium

Abstract

The present application discloses a transformer compression method, device, equipment and medium, and relates to the field of neural network technology. The method includes: determining the key-value cache of the transformer model as an input sequence; the input sequence includes multiple multi-dimensional input elements; reducing the dimension of each input element in the input sequence to obtain a compressed sequence; inputting the compressed sequence into the transformer model to obtain an output sequence; multiplying the output sequence by a target matrix to obtain a target output sequence; the dimension of each output element in the target output sequence is the same as the dimension of the corresponding input element. Reducing the dimension of the key-value cache can reduce the storage space occupied, and inputting the reduced-dimensional key-value cache into the transformer model allows the transformer model to run in a low-dimensional space, compressing the computing resources required by the transformer model.

Description

Technical Field

[0001] The present application relates to the field of neural network technology, and in particular to a transformer compression method, device, equipment and medium. Background Art

[0002] The large language model built on the basis of transformer has promoted the transformation of various industries. With the vigorous development of large language models, the scale of models continues to expand, and the length of input sequences of models continues to increase, requiring more storage space and computing resources. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a transformer compression method, device, equipment and medium to reduce the storage space and computing resources occupied by the transformer model.

[0004] To achieve the above object, an embodiment of the present application provides a transformer compression method in one aspect, the method comprising the following steps:

[0005] Determine a key-value cache of a transformer model as a first input sequence; wherein the first input sequence includes a plurality of multi-dimensional input elements;

[0006] reducing the dimension of each of the input elements in the first input sequence to obtain a first compressed sequence;

[0007] Inputting the first compressed sequence into the transformer model to obtain a first output sequence output by the transformer model;

[0008] The first output sequence is left-multiplied by a first target matrix to obtain a first target output sequence; wherein the dimension of each output element in the first target output sequence is the same as the dimension of the corresponding input element.

[0009] In some embodiments, inputting the first compressed sequence into the transformer model to obtain a first output sequence output by the transformer model comprises the following steps:

[0010] The first compression sequence is input into a target compression layer in the transformer model to obtain the first output sequence output by the target compression layer.

[0011] In some embodiments, the method further comprises the following steps:

[0012] reducing the dimension of the first input sequence to obtain a second compressed sequence;

[0013] Inputting the second compressed sequence into the transformer model to obtain a second output sequence output by the transformer model;

[0014] The second output sequence is right-multiplied by a second target matrix to obtain a second target output sequence; wherein the dimension of the second target output sequence is the same as the dimension of the first input sequence.

[0015] In some embodiments, determining the key-value cache of the transformer model as the first input sequence comprises the following steps:

[0016] The key-value cache calculated by the self-attention layer in the transformer model is determined as the first input sequence.

[0017] In some embodiments, reducing the dimension of each of the input elements in the first input sequence to obtain a first compressed sequence comprises the following steps:

[0018] The first input sequence is left-multiplied by a third target matrix to reduce the dimension of each of the input elements in the first input sequence to obtain the first compressed sequence.

[0019] In some embodiments, before determining the key-value cache of the transformer model as the first input sequence, the method further includes the following steps:

[0020] The transformer model is used to calculate the vector data to obtain the key-value cache.

[0021] In some embodiments, before calculating the vector data using the transformer model to obtain the key-value cache, the method further includes the following steps:

[0022] Convert images, text or spatial data points into corresponding vector data.

[0023] To achieve the above object, another aspect of an embodiment of the present application provides a transformer compression device, the device comprising:

[0024] A sequence determination unit, configured to determine a key-value cache of a transformer model as a first input sequence; wherein the first input sequence includes a plurality of multi-dimensional input elements;

[0025] a sequence compression unit, configured to reduce the dimension of each of the input elements in the first input sequence to obtain a first compressed sequence;

[0026] A model compression unit, configured to input the first compressed sequence into the transformer model to obtain a first output sequence output by the transformer model;

[0027] An output determination unit is used to multiply the first output sequence by a first target matrix to obtain a first target output sequence; wherein the dimension of each output element in the first target output sequence is the same as the dimension of the corresponding input element.

[0028] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above method when executing the computer program.

[0029] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0030] The embodiments of the present application include at least the following beneficial effects:

[0031] The present application can determine the key-value cache of the transformer model as the first input sequence; wherein the first input sequence includes multiple multi-dimensional input elements; the dimension of each input element in the first input sequence is reduced to obtain a first compressed sequence; the first compressed sequence is input into the transformer model to obtain a first output sequence output by the transformer model; the first output sequence is multiplied by the first target matrix on the left to obtain a first target output sequence; wherein the dimension of each output element in the first target output sequence is the same as the dimension of the corresponding input element. The present application reduces the dimension of the key-value cache, which can reduce the storage space occupied by the key-value cache, and inputs the reduced-dimensional key-value cache into the transformer model, so that the transformer model runs in a low-dimensional space, compressing the computing resources required for the transformer model, that is, the present application can realize the simultaneous compression of the key-value cache and the transformer model, which can reduce the occupation of space resources and computing resources, and greatly improve the computing efficiency of the transformer model. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A flowchart of a transformer compression method provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the structure of a transformer compression device provided in an embodiment of the present application;

[0035] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0037] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0038] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0040] The embodiments of the present application provide a transformer compression method, device, equipment and medium. The technical solution of the present application includes: determining the key-value cache of the transformer model as a first input sequence; wherein the first input sequence includes multiple multi-dimensional input elements; reducing the dimension of each input element in the first input sequence to obtain a first compressed sequence; inputting the first compressed sequence into the transformer model to obtain a first output sequence output by the transformer model; multiplying the first output sequence by the first target matrix to obtain a first target output sequence; wherein the dimension of each output element in the first target output sequence is the same as the dimension of the corresponding input element. The present application reduces the dimension of the key-value cache, which can reduce the storage space occupied by the key-value cache, and inputs the reduced-dimensional key-value cache into the transformer model, so that the transformer model runs in a low-dimensional space, compressing the computing resources required by the transformer model, that is, the present application can realize the simultaneous compression of the key-value cache and the transformer model, which can reduce the occupation of space resources and computing resources, and greatly improve the computing efficiency of the transformer model.

[0041] The embodiment of the present application provides a transformer compression method, which relates to the field of neural network technology. The transformer compression method provided in the embodiment of the present application can be applied to a terminal, a server, or a software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements the transformer compression method, etc., but is not limited to the above forms.

[0042] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0043] Reference Figure 1 , the embodiment of the present application provides a transformer compression method, which may include but is not limited to S100 to S130, as follows:

[0044] S100: Determine a key-value cache of a transformer model as a first input sequence; wherein the first input sequence includes multiple multi-dimensional input elements.

[0045] It should be noted that the input sequence of the embodiment of the present application may include multiple tokens, each token serves as an element, and each element may be multi-dimensional data, including multiple data.

[0046] Furthermore, S100 may include:

[0047] The key-value cache calculated by the self-attention layer in the transformer model is determined as the first input sequence.

[0048] It can be understood that this embodiment can obtain the key-value cache generated by the self-attention layer as the first input sequence. It should be noted that the input sequences of other optional situations in the embodiment of the present application can also be determined based on the key-value cache generated by the self-attention layer.

[0049] As a further implementation, before S100, the embodiment of the present application may include:

[0050] The transformer model is used to calculate the vector data to obtain the key-value cache.

[0051] It can be understood that the key-value cache of this embodiment can be generated by the transformer model performing various calculations on any type of vector data.

[0052] Furthermore, the step of obtaining the vector data in this embodiment includes:

[0053] Convert images, text or spatial data points into corresponding vector data.

[0054] It can be understood that this embodiment can convert image features, text features, spatial data point features, polymer material features or gene sequence features into corresponding vector data, and then use the transformer model to calculate the vector data to obtain a key-value cache.

[0055] S110: Reduce the dimension of each of the input elements in the first input sequence to obtain a first compressed sequence.

[0056] Specifically, this embodiment may perform a dimensionality reduction operation on each element in the input sequence.

[0057] Furthermore, S100 may include:

[0058] The first input sequence is left-multiplied by a third target matrix to reduce the dimension of each of the input elements in the first input sequence to obtain the first compressed sequence.

[0059] S120: Input the first compressed sequence into the transformer model to obtain a first output sequence output by the transformer model.

[0060] Specifically, the compressed sequence is input into the transformer model, and the transformer model can run in a low-dimensional space, thereby achieving compression of the transformer model.

[0061] Furthermore, in order to reduce the performance loss caused by model compression, considering that the transformer model contains multiple layers, this embodiment can compress only some layers, specifically including:

[0062] The first compression sequence is input into a target compression layer in the transformer model to obtain the first output sequence output by the target compression layer.

[0063] S130: Multiply the first output sequence by a first target matrix on the left to obtain a first target output sequence; wherein the dimension of each output element in the first target output sequence is the same as the dimension of the corresponding input element.

[0064] It can be understood that, in this embodiment, after left-multiplying the first output sequence by the first target matrix, the output dimension of the transformer model is increased, that is, the dimension of each output element in the first target output sequence is the same as the dimension of the corresponding input element.

[0065] In addition, the embodiment of the present application can also reduce the dimension of the input sequence, which can specifically include the following steps:

[0066] reducing the dimension of the first input sequence to obtain a second compressed sequence;

[0067] Inputting the second compressed sequence into the transformer model to obtain a second output sequence output by the transformer model;

[0068] The second output sequence is right-multiplied by a second target matrix to obtain a second target output sequence; wherein the dimension of the second target output sequence is the same as the dimension of the first input sequence.

[0069] Next, the solution of the embodiment of the present application will be described in detail and explained in conjunction with specific application examples:

[0070] The uncompressed transformer model can be expressed as:

[0071] y = transformer(x);

[0072] The input x of the transformer model is a sequence of L tokens (i.e., elements, which can be determined based on the key-value cache generated by the self-attention layer). The dimension of each element in the sequence is d, so the input sequence can be represented as a real number with a dimension of d×l: x∈R d×l ; Output y of the transformer model: y∈R d×l .

[0073] This embodiment proposes a transformer compression method based on latent space. The compression expression can be expressed as:

[0074] y=latent_transformer(x)=B*transformer(Ax);

[0075] in, The compressed transformer model is represented as latenttransformer, whose input x is a sequence of length L tokens, and the dimension of each element in the sequence is d c , then the input x can be represented as a c ×l real number: Output of the transformer model: After left - multiplying it by matrix B, the output y of the latent transformer model: y ∈ R d×l .

[0076] It can be seen that, compared with the uncompressed transformer model, the latent transformer model runs in a compressed low - dimensional space. The dimension of the model is lower, requiring less computing and storage resources and having higher efficiency. Since matrices A and B are very small compared to the overall resources of the model and can be ignored. If d c < d, the compression ratio can be approximately expressed as: If d c > d, the model coefficients will become sparse, but the model scale will become times larger.

[0077] The beneficial effects of this embodiment include: It can achieve simultaneous compression of the model and the key - value cache, reducing the required computing and storage resources.

[0078] Furthermore, in order to reduce the performance loss caused by model compression, considering that the transformer model includes multiple layers, this embodiment can compress only some layers. Assuming that the n - th layer is to be compressed, it can be expressed as:

[0079] y = latent_transformer_n(x) = B * transformer_n(Ax);

[0080] Where latent_transformer_n represents the n - th layer of the latent transformer model; transformer_n represents the n - th layer of the uncompressed transformer model, and the dimensions of matrices A and B remain unchanged. x is the input of the n - th layer of the uncompressed transformer model, and y is the output of the n - th layer of the latent transformer model.

[0081] Furthermore, the method of compressing the element dimension can be used to compress the dimension l of the input sequence as well. The expression for compressing the input sequence dimension is:

[0082] y = latent_transformer(x) = transformer(xC)D;

[0083] Where

[0084] Next, a more specific embodiment will be described: Take the deployment of an open source model Yi with a network size of 34GB on a graphics processor as an example. The number of layers of model Yi is 60, the number of heads is 8, the model dimension is 128, the data precision is BF16, the HBM capacity of the graphics processor is 80GB, and the HBM bandwidth is 2T byte / s. Then the following conditions are met:

[0085] The storage space of the model is: 34*2=68GB;

[0086] Assume that the sequence length of the text is 16K, and the key-value (KV) cache space of a single user is:

[0087] 16000*60*8*128*2*2≈3.6GB;

[0088] The theoretical delay of decoding is:

[0089]

[0090] If a latent transformer is used, assuming If the training data and tokens remain unchanged, the dimension of the model is 128 / 4=32.

[0091] The storage space of the model is: 34*2 / 4=17GB;

[0092] The key-value cache space for a single user is:

[0093] 16000*60*8*128 / 4*2*2≈0.9GB;

[0094] The theoretical delay of decoding is:

[0095]

[0096] According to this embodiment, the decoding delay can be reduced from 10.7s to 2.7s.

[0097] Assume that the sequence length of the text is 256K, and the key-value cache space of a single user is:

[0098] 256000*60*8*128*2*2≈58GB;

[0099] Since model weights + KV cache = 68GB + 58GB > 80GB = HBM capacity, HBM capacity cannot meet the requirements. If the latent transformer model is used, assuming Then we have:

[0100] The storage space of the model is: 34*2 / 4=17GB;

[0101] The KV cache space for a single user is:

[0102] 256000*60*8*128 / 4*2*2≈14.6GB;

[0103] The theoretical delay of decoding is:

[0104]

[0105] Through this embodiment, it can be obtained that the HBM capacity can not only meet the requirements, but the decoding delay is only increased from 2.7s to 4.7s.

[0106] This embodiment can reduce the storage space occupied by the key-value cache, and input the reduced-dimensional key-value cache into the transformer model, so that the transformer model runs in a low-dimensional space, compressing the computing resources required by the transformer model and improving the operating efficiency.

[0107] Reference Figure 2 The embodiment of the present application further provides a transformer compression device, which can implement the above transformer compression method, and the device includes:

[0108] A sequence determination unit, configured to determine a key-value cache of a transformer model as a first input sequence; wherein the first input sequence includes a plurality of multi-dimensional input elements;

[0109] a sequence compression unit, configured to reduce the dimension of each of the input elements in the first input sequence to obtain a first compressed sequence;

[0110] A model compression unit, configured to input the first compressed sequence into the transformer model to obtain a first output sequence output by the transformer model;

[0111] An output determination unit is used to multiply the first output sequence by a first target matrix to obtain a first target output sequence; wherein the dimension of each output element in the first target output sequence is the same as the dimension of the corresponding input element.

[0112] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0113] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above transformer compression method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0114] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] See also Figure 3 , Figure 3 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0116] The processor 301 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0117] The memory 302 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 302, and the processor 301 calls and executes the compression method of the transformer in the embodiment of this application;

[0118] Input / output interface 303, used to implement information input and output;

[0119] The communication interface 304 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0120] A bus 305 that transmits information between the various components of the device (e.g., the processor 301, the memory 302, the input / output interface 303, and the communication interface 304);

[0121] The processor 301 , the memory 302 , the input / output interface 303 and the communication interface 304 are connected to each other in communication within the device via the bus 305 .

[0122] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the compression method of the above-mentioned transformer is implemented.

[0123] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0124] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0125] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0126] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0127] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0129] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0130] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0131] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0134] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0135] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A transformer compression method, characterized in that: The method comprises the following steps: Determine a key-value cache of a transformer model as a first input sequence; wherein the first input sequence includes a plurality of multi-dimensional input elements; reducing the dimension of each of the input elements in the first input sequence to obtain a first compressed sequence; Inputting the first compressed sequence into the transformer model to obtain a first output sequence output by the transformer model; Multiplying the first output sequence by the first target matrix on the left to obtain a first target output sequence; wherein the dimension of each output element in the first target output sequence is the same as the dimension of the corresponding input element; The method further comprises the following steps: reducing the dimension of the first input sequence to obtain a second compressed sequence; Inputting the second compressed sequence into the transformer model to obtain a second output sequence output by the transformer model; multiplying the second output sequence by a second target matrix to obtain a second target output sequence; wherein the dimension of the second target output sequence is the same as the dimension of the first input sequence; Before determining the key-value cache of the transformer model as the first input sequence, the method further includes the following steps: Calculating the vector data using the transformer model to obtain the key-value cache; Before calculating the vector data using the transformer model to obtain the key-value cache, the method further includes the following steps: Convert images, text or spatial data points into corresponding vector data.

2. A transformer compression method according to claim 1, characterized in that: The step of inputting the first compressed sequence into the transformer model to obtain a first output sequence output by the transformer model comprises the following steps: The first compression sequence is input into a target compression layer in the transformer model to obtain the first output sequence output by the target compression layer.

3. The transformer compression method according to claim 1, characterized in that: The step of determining the key-value cache of the transformer model as the first input sequence comprises the following steps: The key-value cache calculated by the self-attention layer in the transformer model is determined as the first input sequence.

4. The transformer compression method according to claim 1, characterized in that: The step of reducing the dimension of each input element in the first input sequence to obtain a first compressed sequence comprises the following steps: The first input sequence is left-multiplied by a third target matrix to reduce the dimension of each of the input elements in the first input sequence to obtain the first compressed sequence.

5. A transformer compression device, characterized in that: The device comprises: A sequence determination unit, configured to determine a key-value cache of a transformer model as a first input sequence; wherein the first input sequence includes a plurality of multi-dimensional input elements; a sequence compression unit, configured to reduce the dimension of each of the input elements in the first input sequence to obtain a first compressed sequence; A model compression unit, configured to input the first compressed sequence into the transformer model to obtain a first output sequence output by the transformer model; an output determination unit, configured to obtain a first target output sequence by left-multiplying the first output sequence by a first target matrix; wherein the dimension of each output element in the first target output sequence is the same as the dimension of the corresponding input element; The device is also used for: reducing the dimension of the first input sequence to obtain a second compressed sequence; Inputting the second compressed sequence into the transformer model to obtain a second output sequence output by the transformer model; multiplying the second output sequence by a second target matrix to obtain a second target output sequence; wherein the dimension of the second target output sequence is the same as the dimension of the first input sequence; Before determining the key-value cache of the transformer model as the first input sequence, the device is further used for: Calculating the vector data using the transformer model to obtain the key-value cache; Before calculating the vector data using the transformer model to obtain the key-value cache, the device is further used to: Convert images, text or spatial data points into corresponding vector data.

6. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images