CNN text classification method combined with multi-head self-attention mechanism

Abstract

The invention discloses a CNN text classification method combined with a multi-head self-attention mechanism, and the method comprises the steps: obtaining a word segmentation sequence of a to-be-classified text, and carrying out the filtering of special symbols and stop words; querying a preset or randomly initialized word embedding model to obtain an embedding matrix of the word segmentation sequence, wherein each line of the matrix is an embedding vector of each segmented word; for each row vector of the obtained embedded matrix, superposing the position coding vector of the word segmentation corresponding to the vector; generating a self-attention matrix for the embedded matrix after superposition position coding through a self-attention mechanism; repeating the generation of the self-attention matrix for a plurality of times, and splicing the plurality of generated matrixes on the dimension of the column; multiplying the self-attention matrix after word sequence splicing by the weighting matrix to realize dimensionality reduction and fusion; and inputting the dimensionality-reduced and fused self-attention matrix into a CNN for training or prediction.

Description

【Technical field】 [0001] The present invention relates to a CNN text classification method combined with a multi-head self-attention mechanism, and in particular, one or more embodiments relate to the technical field of natural language processing (NLP). 【Background technique】 [0002] Text classification is one of the common downstream tasks in NLP. Deep learning algorithms are widely used in text classification and have excellent performance. At present, text classification based on deep learning mostly uses RNN, CNN and Transformer models. Among them, RNN and Transformer can learn the global semantic information of text, but due to the large amount of calculation, the application is limited. CNN has a small amount of calculation and is convenient for parallel acceleration. It has incomparable advantages in industrial applications. However, due to the limitation of the width of the receptive field, it can only learn local semantic information of the text, and its classific...

AI Technical Summary

Problems solved by technology

At present, text classification based on deep learning mostly uses RNN, CNN and Transformer models, among which RNN and Transformer can learn the global semantic information of text, but due to the large amount of calculation, the application is limited

CNN has a small amount of calculation and is convenient for parallel acceleration. It has incomparable advantages in industrial applications. However, due to the limitation of the width of the receptive field, it can only learn local semantic information of the text, and its classification performance is not as good as that of RNN and Transformer models.

Images