Novel multi-head attention mechanism

Abstract

The invention discloses a novel multi-head attention mechanism. According to a method employing local attention, compared with a global attention method adopted by a traditional multi-head attention mechanism, the complexity of the model is reduced, the sizes of all matrixes in the operation process are only in direct proportion to the length of the sequence, and compared with a matrix in direct proportion to the square of the sequence in the traditional attention mechanism, the storage space consumption of the model is reduced to a large extent. In calculation process, compared with a solution in Transformer-XL, blocking processing is not carried out on the sequence, the sequence characteristics of the original sequence are reserved to a great extent, softmax is used for establishing global semantics, and compared with a cross-block connection mode used in Transformer-XL, the parallelism degree of the model is improved.

Description

technical field [0001] The invention relates to the technical fields of artificial intelligence, machine learning and data mining, in particular to a novel multi-head attention mechanism. Background technique [0002] With the continuous integration of artificial intelligence technology and machine learning technology in the field of natural language processing, more and more deep learning technologies have been applied in the field of natural language processing. Among them, GPT, BERT, RoBERTa, ALBERT, XL-Net and other methods based on Transformer based on multi-head attention mechanism have won praise from the industry, and are increasingly being applied in natural language processing and other fields. [0003] However, the original multi-head attention mechanism has its inherent disadvantages: first, the space occupation of the multi-head attention mechanism is proportional to the square of the length of the processed sequence, and the space complexity is high, which will...

AI Technical Summary

Problems solved by technology

[0003] However, the original multi-head attention mechanism has its inherent disadvantages: first, the space occupation of the multi-head attention mechanism is proportional to the square of the length of the processed sequence, and the space complexity is high, which will take up a lot of time when processing longer sequences. Computational space; secondly, the attention mechanism establishes the relationship between all elements in the sequence, but in the field of actual language processing, it is not necessary to model all the elements in all sequences, that is, the traditional multi-head attention matrix There is a lot of waste in the calculations in , slowing down the speed of sequence processing

[0004] The traditional way to solve the space complexity of the multi-head attention mechanism, such as Transformer-XL, divides the sequence into blocks, and the segmentation of the sequence destroys the original continuous structure of the sequence and the characteristics of the original data.

And in the connection structure part between blocks, the attention mechanism similar to Decoder in Transformer is used for processing, and the loop structure is added in the process of model processing, which reduces the degree of parallelization of the model and reduces the overall performance of the model.

Images