A dialogue summary and key intention extraction method, system and device based on a large model and multi-dimensional acoustic features

Abstract

The application discloses a dialogue summary and key intention extraction method, system and equipment based on a large model and multi-dimensional acoustic characteristics, and belongs to the technical field of artificial intelligence. The method comprises the following steps: obtaining dialogue audio and identifying to obtain a text Token sequence with a timestamp; a multi-dimensional acoustic feature vector is extracted, and a multi-modal feature matrix is constructed; the matrix is input into a large language model, the acoustic features are mapped into Key and Value matrices, the attention bias term is calculated and superimposed into the model self-attention weight matrix, and the dynamic adjustment of the attention of the text Token sequence is realized; and the dialogue summary and the key intention list are decoded and output based on the adjusted weight. The application proposes an attention bias injection mechanism, realizes the underlying fusion of acoustic features and text semantics through the attention layer, effectively solves the feature loss problem in the voice dialogue, and significantly improves the accuracy and robustness of intention recognition.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction, specifically to a dialogue processing technology that combines multidimensional acoustic feature extraction with the attention mechanism of a large language model. Background Technology

[0002] Retrieval-enhanced generation (RAG) techniques and large language models excel in dialogue summarization and intent extraction tasks. However, existing technologies typically process textual semantics alone, neglecting the rich emotional and state information inherent in speech signals. In actual voice dialogue interactions, acoustic features such as speech rate, intonation, and energy fluctuations directly reflect the core intent and contextual semantics of the dialogue. Existing methods, lacking multimodal fusion mechanisms, struggle to achieve high-precision extraction and real-time error correction of key information in voice dialogues. Summary of the Invention

[0003] This invention aims to provide a method, system, and device for dialogue summarization and key intent extraction based on a large model and multidimensional acoustic features. By constructing a multimodal feature matrix and dynamically adjusting the attention weights of the large model, the accuracy of dialogue summarization and intent recognition is improved.

[0004] The first aspect of the present invention provides a dialogue processing method based on a large model and multidimensional acoustic features, comprising: acquiring and recognizing dialogue audio data to obtain a text token sequence with timestamps; extracting multidimensional acoustic feature vectors, aligning the text token sequence with the acoustic feature vector modality to construct a multimodal feature matrix; inputting the multimodal feature matrix into the attention mechanism layer of a large language model, and dynamically adjusting the attention weight distribution of the text token sequence using the acoustic feature vectors; decoding based on the adjusted weight distribution to output a dialogue summary and a list of key intents.

[0005] A second aspect of the present invention provides a dialogue processing system based on a large model and multidimensional acoustic features, comprising: a speech processing module, an acoustic perception module, a multimodal attention module, and an information extraction module.

[0006] The beneficial effects of this invention are as follows: by injecting multidimensional acoustic features into the attention mechanism of a large model, the model's ability to capture key semantic information in speech dialogue is enhanced, effectively improving the accuracy and robustness of dialogue summarization and intent recognition.

[0007] This invention differs from existing technologies that rely on simple feature concatenation or graph neural network-based reconstruction. Instead, it proposes an attention bias injection scheme targeting the internal attention mechanism of Large Language Models (LLMs). Specifically, this invention maps multidimensional acoustic features into Key and Value matrices, calculates an attention bias term B based on the similarity between the Query and Key matrices, and directly superimposes this bias term B onto the original self-attention weight matrix S of the LLM, ensuring that the modified weight matrix P satisfies P = Softmax(S + B). This scheme enables the LLM to directly perceive real-time fluctuations in acoustic features (such as prosody and speech rate) during decoding, thereby achieving low-level modality fusion at the attention computation level. Compared to traditional intent recognition models, this mechanism effectively solves the feature loss problem in complex speech contexts, improving the accuracy of capturing key intents in long texts and enhancing semantic alignment capabilities. Attached Figure Description

[0008] Figure 1 This is an overall flowchart of the method provided in the embodiments of the present invention; Figure 2 This is a schematic diagram illustrating the principle of acoustic emotion feature extraction in this embodiment of the invention; Figure 3 This is a closed-loop architecture diagram of reinforcement learning and knowledge base evolution in an embodiment of the present invention; Figure 4 This is a block diagram of the hardware structure of a computer device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the network structure and data flow of the Actor-Critic reinforcement learning agent in an embodiment of the present invention. Detailed Implementation

[0009] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. The solution provided by the present invention achieves high-precision and highly robust dialogue processing through a combination of hardware and software.

[0010] Example 1: System Architecture. This system is deployed based on a server cluster. The system includes a processor, memory, communication interface, and bus. The processor, by calling computer programs in memory, realizes real-time processing of dialogue audio streams, acoustic feature calculation, dynamic adjustment of multimodal attention matrix, and summarization and intent generation.

[0011] Example 2: Method Flow. The method in this example includes the following steps: Multidimensional acoustic feature extraction. In this embodiment, the audio sampling rate is set to 16kHz. A Hamming window is used for windowing, followed by a Fast Fourier Transform (FFT) to generate a log-Mel spectrum. A 512-dimensional deep spectral feature vector is extracted using a ResNet network; a 256-dimensional global prosodic feature vector (including fundamental frequency, energy, and zero-crossing rate information) is extracted using a Bi-LSTM network. These features are concatenated, and based on the timestamps output by the ASR, millisecond-level alignment between the acoustic feature vector and the text token sequence is achieved. A modal alignment compensation algorithm is then used to optimize the alignment accuracy, thereby constructing a multimodal feature matrix.

[0012] Attention weights are dynamically weighted. This embodiment adjusts the weights by introducing an attention bias term. The embedding vectors of text tokens are mapped to a Query matrix, and the aligned multidimensional acoustic feature vectors are mapped to a Key matrix and a Value matrix. By calculating the similarity between the Query matrix and the Key matrix, an attention bias term B that integrates acoustic information is generated. This bias term B is then superimposed on the original self-attention weight matrix S of the large language model to obtain the corrected weight matrix P, calculated as: P = Softmax(S + B). In the differentiated adjustment mechanism, a differentiated attention weighting baseline is preset based on the dialogue role (such as customer or agent) identified by voiceprint recognition. The acoustic bias term is adaptively adjusted using dynamic normalization technology to eliminate interference caused by excessive fluctuations in the baselines of different roles during the dialogue. When the dialogue fluctuation variance is lower than a preset threshold, the bias weight of acoustic features is automatically reduced, and the semantic weight of pure text is increased. In the attention saturation suppression mechanism, the fluctuation of local acoustic features relative to the baseline is monitored in real time. When the fluctuation amplitude or duration exceeds the preset saturation threshold, the weight decay function is triggered to perform a weight reduction penalty. The expression of the weight decay function is: f(t)=exp(-λ*t), where λ is the decay coefficient and t is the duration.

[0013] Dialogue summarization and key intent generation. The large model uses injected acoustic bias weights to forcibly recall core intents such as to-do items and business requests from long text sequences and generate summaries. If the acoustic features represent high emotional intensity and the text confidence is low, error correction logic is triggered, combining contextual semantic cues with acoustic features to complete the intent.

[0014] Example 3: Computer Device. The computer device provided in this example includes a processor and a memory. The processor executes the computer program stored in the memory to implement the above method, ensuring that the algorithm executes efficiently in environments with limited computing resources.

Claims

1. A method for dialogue summarization and key intent extraction based on large models and multidimensional acoustic features, characterized in that, Includes the following steps: Acquire dialogue audio data and perform speech recognition to obtain a text token sequence with timestamps; Extract the multidimensional acoustic feature vector from the audio data, and modally align the text token sequence with the multidimensional acoustic feature vector based on the timestamp to construct a multimodal feature matrix; The multimodal feature matrix is ​​input into the attention mechanism layer of the large language model. The embedding vector of the text token is mapped to a query matrix, and the aligned multidimensional acoustic feature vector is mapped to a key matrix and a value matrix. By calculating the similarity between the query matrix and the key matrix, a self-attention bias term B that integrates acoustic information is generated. Before performing Softmax calculation, the self-attention bias term B is superimposed on the original self-attention input matrix S of the large language model. The corrected self-attention weight matrix P is obtained by calculating the Softmax function. The calculation logic formula is: P=Softmax(S+B), where P is the corrected self-attention weight matrix, S is the original self-attention input matrix of the large language model, and B is the self-attention bias term. The large language model uses the modified self-attention weight matrix P for decoding, and outputs a dialogue summary and a list of key intents.

2. The method according to claim 1, characterized in that, Extracting the multidimensional acoustic feature vector of the audio data includes: extracting spectral features and prosodic features from the audio data; the spectral features include a log-Mel spectrum, and the prosodic features include at least one of fundamental frequency, energy, and rate of change of speech.

3. The method according to claim 1, characterized in that, It also includes a differentiated processing mechanism based on dialogue roles: Dialogue roles are identified based on voiceprint features, and different attention weighting baselines are preset for different roles. When the variance of the current dialogue segment is detected to be lower than a preset threshold, the bias weight of multidimensional acoustic features is automatically reduced and the weight of pure text semantics is increased.

4. The method according to claim 1, characterized in that, It also includes attention saturation inhibition mechanisms: Real-time monitoring of the fluctuation of local acoustic features relative to the baseline; when the fluctuation amplitude or duration exceeds the preset saturation threshold, a time decay operation is performed on the attention bias term of the current window to guide the model's attention to shift to the semantic dimension.

5. The method according to claim 1, characterized in that, It also includes a voice error correction mechanism: When the multidimensional acoustic features represent high emotional intensity and the recognition confidence of the corresponding text is lower than a preset threshold, a modal conflict identifier is generated. Based on the modal conflict identifiers and contextual semantic cues, the large language model performs implicit error correction or intent reasoning on the text token sequence.

6. A dialogue summarization and key intent extraction system based on large models and multidimensional acoustic features, characterized in that, include: The speech perception module is used to acquire dialogue audio and perform speech recognition to obtain a text token sequence with timestamps; An acoustic perception module is used to extract multidimensional acoustic feature vectors from the audio data and modally align the text token sequence with the multidimensional acoustic feature vectors according to the timestamp to construct a multimodal feature matrix. A multimodal attention module is used to input the multimodal feature matrix into the attention mechanism layer of the large language model, map the embedding vector of the text token to a query matrix, map the aligned multidimensional acoustic feature vector to a key matrix and a value matrix, generate a self-attention bias term B that integrates acoustic information by calculating the similarity between the query matrix and the key matrix, and before performing Softmax calculation, superimpose the self-attention bias term B onto the original self-attention input matrix S of the large language model, and calculate the corrected self-attention weight matrix P through the Softmax function. The calculation logic formula is: P=Softmax(S+B), where P is the corrected self-attention weight matrix, S is the original self-attention input matrix of the large language model, and B is the self-attention bias term. The information extraction module is used by the large language model to perform decoding processing using the modified self-attention weight matrix P, and output dialogue summary and key intent list.

7. A computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the method steps as described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the method steps as described in any one of claims 1 to 5.

Images