Key value compression method in self-attention mechanism, large language model and electronic equipment

By performing multiple residual decomposition and cluster compression on the key-value matrix, and combining the query matrix for attention calculation, the limitations of the traditional self-attention mechanism in long-sequence tasks are solved, and efficient long-sequence processing and flexible sequence representation are achieved.

CN120106150AActive Publication Date: 2025-06-06BEIJING ACAD OF ARTIFICIAL INTELLLIGENCE
5 Cites 0 Cited by

Patent Information

Application Number
CN202510344978.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-03-21
Publication Date
2025-06-06
Estimated Expiration
2045-03-21

AI Technical Summary

Technical Problem

The traditional self-attention mechanism based on softmax has limitations in memory capacity and computing efficiency when processing long-sequence tasks, making it difficult to effectively process long-sequence data.

Method used

By performing multiple residual decompositions on the key matrix and the value matrix, clustering and compression of the key residual vector and the value residual vector after each decomposition, attention calculation is performed in combination with the query matrix, and all attention calculation results are accumulated to achieve approximate attention output.

Benefits of technology

Improves the efficiency of Transformer in long-sequence tasks, solves the problem that Linear Transformer cannot use standard Softmax Transformer parameters, and implements a more flexible and robust sequence representation to adapt to various attention mechanisms.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a key value compression method in a self-attention mechanism, a large language model and electronic equipment, and relates to the technical field of computers. The compression method comprises the following steps: respectively carrying out multiple times of residual decomposition on a key matrix and a value matrix to obtain a key residual vector and a value residual vector after each time of decomposition; respectively carrying out clustering compression on the key residual vector and the value residual vector after each decomposition, and carrying out attention calculation on the query matrix, the compressed key residual vector and the compressed value residual vector; and accumulating all attention calculation results. According to the method and the device, the problems that standard Softmax Transform parameters cannot be used by the Linear Transform, and the difference between the standard Softmax Transform parameters and the standard Softmax Transform parameters is large are solved.
Need to check novelty before this filing date? Find Prior Art