Sparse processing method and apparatus for sparse attention network, and electronic device
Through the sparse processing method of the sparse attention network and the use of constraints to construct masked sparse representation, the problem of high memory consumption in large model training is solved, and memory consumption is reduced and training efficiency is improved.
WO2025194553A1 Publication Date: 2025-09-25BEIJING BAIDU NETCOM SCI & TECH CO LTD
4 Cites 8 Cited by
Patent Information
- Application Number
- PCT/CN2024/089610
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-03-19
- Filing Date
- 2024-04-24
- Publication Date
- 2025-09-25
AI Technical Summary
Technical Problem
In the training of large transformer-based models, the memory consumption of the attention module is quadratically proportional to the sequence length. Especially when using custom masks, it is difficult to further reduce the memory consumption with existing technologies.
Method used
The sparse processing method of the sparse attention network is adopted to construct a masked sparse representation through constraints, reducing memory consumption from the square of the sequence length to the linear level. The sparse attention mechanism is combined with the fine-tuning training of the large language model.
Benefits of technology
This significantly reduces the memory requirements for training large models, improves training efficiency, and enhances computing performance.
✦ Generated by Eureka AI based on patent content.
Abstract
A sparse processing method and apparatus for a sparse attention network, and an electronic device. The method specifically comprises: in a memory cell, using constraint conditions to construct a masked sparse representation (S301), wherein the constraint conditions are: the dimension of a mask matrix is [B,a,S], B represents the batch size, a represents the number of heads, S represents the sequence length, and each element in the S dimension represents a masked starting row of each column in the mask matrix; and a calculation unit acquiring the masked sparse representation from the memory cell, using the masked sparse representation to perform sparse processing on input data, and storing the sparse processing result into the memory cell (S302). According to the method, the constraint conditions are used to construct the masked sparse representation, so that memory consumption can be reduced from a quadratic order of the sequence length to a linear order of the sequence length, thereby remarkably reducing memory requirements during large model training, and improving the training efficiency.
Need to check novelty before this filing date? Find Prior Art