A dialogue summary and key intention extraction method, system and device based on a large model and multi-dimensional acoustic features

Abstract

The application discloses a dialogue summary and key intention extraction method, system and equipment based on a large model and multi-dimensional acoustic characteristics, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining dialogue audio and identifying to obtain a text Token sequence with a timestamp; a multi-dimensional acoustic feature vector is extracted, and a multi-modal feature matrix is constructed; the matrix is input into a large language model, the acoustic features are mapped into Key and Value matrices, the attention bias term is calculated and superimposed into the model self-attention weight matrix, and the dynamic adjustment of the attention of the text Token sequence is realized; and the dialogue summary and the key intention list are decoded and output based on the adjusted weight. The application proposes an attention bias injection mechanism, realizes the underlying fusion of acoustic features and text semantics through the attention layer, effectively solves the feature loss problem in the voice dialogue, and significantly improves the accuracy and robustness of intention recognition.

Images