Voice interaction optimization method and system based on multimodal large model

Abstract

The present invention discloses a voice interaction optimization method and system based on a multimodal large model, which relates to the field of artificial intelligence and voice interaction technology, including: performing voice recognition in response to real-time user voice to obtain voice text information; using the voice text information in combination with the voice waveform of the user's real-time voice as input of a multimodal recognition model to determine whether the voice conversation is interrupted and identify the interruption intention, and obtain a voice interruption result; obtaining the user's new intention based on the voice interruption result, and dynamically adjusting the system response strategy to achieve interaction optimization; improving the accuracy and comprehensiveness of interruption detection through multimodal fusion, thereby accurately identifying the interruption intention and responding to the user's intention in real time, significantly improving the system dialogue interaction efficiency and reliability, and overcoming the problems of low detection accuracy and inability to achieve dynamic response to user interruption behavior in existing voice interruption detection, resulting in low interaction efficiency and poor reliability of the voice dialogue system.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and voice interaction technology, specifically to a voice interaction optimization method and system based on a multimodal large model. Background Art

[0002] As artificial intelligence (AI) has made significant progress in fields such as natural language processing (NLP), machine learning (ML), and deep learning (DL), voice interaction technology, as an important branch of AI, relies on technologies such as speech recognition, natural language understanding, and speech synthesis to achieve natural dialogue between people and devices. In voice dialogue systems, handling user interruptions is an important part of ensuring the smoothness and naturalness of interactions.

[0003] Currently, the methods for detecting and responding to users' interruption intentions include: keyword-based interruption detection, which uses predefined interruption words (such as "stop," "wait a moment," and "interrupt") to identify whether the user intends to interrupt the system's voice output or conversation flow; acoustic feature-based interruption detection, which determines interruption intention by analyzing the acoustic features of the user's voice, mainly relying on changes in parameters such as volume, speaking rate, and pitch; timer-based interruption detection, which detects user interruption behavior by setting a time window; and conversation state-based interruption detection, which relies on the state tracking capability of the conversation management system and determines interruption behavior by analyzing the current conversation context and user input.

[0004] While the aforementioned technical solutions have achieved some success in speech interruption detection, they still suffer from some significant shortcomings. First, the single, limited detection format results in insufficient accuracy. For example, keywords are limited to a predefined vocabulary. If a user uses non-keyword expressions, the system cannot recognize them, resulting in missed detections. Acoustic features are susceptible to interference from individual user differences or environmental noise, resulting in a high misjudgment rate. While the conversation state takes context into account, its reliance on natural language understanding (NLU) renders it ineffective when semantic understanding is inaccurate, especially in complex, multi-intent scenarios. Each of the aforementioned detection methods is single-dimensional and lacks multimodal fusion, resulting in low detection accuracy and poor applicability. Second, the existing solutions' dialogue management strategies are relatively rigid. Even if an interruption is detected, they cannot intelligently adjust the subsequent dialogue process based on the specific content of the interruption, resulting in poor interactive effects in the voice dialogue system. Summary of the Invention

[0005] The purpose of this application is to address the problems of low detection accuracy and inability to dynamically respond to user interruption behaviors in existing voice interruption detection, which leads to low interaction efficiency and poor reliability of the voice dialogue system; a voice interaction optimization method and system based on a multimodal large model are proposed. By fusing voice text and acoustic features as the input of the multimodal recognition model, it determines whether the system's voice dialogue is interrupted and identifies the interruption intention at the same time, so as to dynamically adjust the system response strategy according to the user's interruption intention. This application can accurately identify user interruption behaviors in a variety of scenarios, and improve the accuracy and comprehensiveness of interruption detection through multimodal fusion, thereby accurately identifying interruption intentions and responding to user intentions in real time, significantly improving the system dialogue interaction efficiency and reliability.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of the present application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for optimizing voice interaction based on a multimodal large model, the method comprising:

[0008] Perform speech recognition in response to the user's real-time speech to obtain speech text information; use the speech text information combined with the speech waveform of the user's real-time speech as input to the multimodal recognition model to determine whether the voice conversation has been interrupted and identify the interruption intention, and obtain the speech interruption result; obtain the user's new intention based on the speech interruption result, and dynamically adjust the system response strategy to achieve interaction optimization.

[0009] In this solution, through real-time speech recognition, features can be directly extracted from the original waveform and decoded into text to obtain speech text information, providing higher quality and more timely text input for subsequent accurate interruption detection. At the same time, it avoids the analysis process of too many redundant data features and improves detection efficiency. Through a large multimodal model (here is a multimodal recognition model), the acoustic features of speech text information and speech waveforms are processed simultaneously, solving the problem of poor detection accuracy caused by a single keyword or acoustic detection, reducing the missed judgment and false judgment rate of a single dimension, and achieving more comprehensive and intelligent interruption judgment. When judging the interruption behavior, the user's interruption intention is simultaneously identified to dynamically generate a response based on the interruption intention, thereby accurately and effectively managing the dialogue process, improving the flexibility, timeliness and reliability of the interaction, and further improving the naturalness and intelligence of the dialogue.

[0010] Preferably, the speech recognition is performed in response to the user's real-time speech to obtain speech text information, including: real-time collection of user speech signals, dynamic adjustment of signal gain according to the direction of the sound source, and lateral interference suppression of the speech signal to obtain a target audio stream; generating a spectrogram according to the target audio stream to calculate cepstral coefficients and their differences to obtain time-frequency domain features; performing fine-grained acoustic information capture on the original waveform data of the target audio stream to obtain fine-grained waveform features; generating an initial text sequence based on the time-frequency domain features and the fine-grained waveform features, and correcting the initial text sequence to obtain final speech text information.

[0011] Preferably, generating a spectrogram based on the target audio stream to calculate cepstral coefficients and their differences to obtain time-frequency domain features includes: performing frame processing on the target audio stream and then generating a spectrogram through Fourier transform; calculating the cepstral coefficients of the target audio stream based on the spectrogram to obtain basic acoustic features of the target audio stream; calculating the first-order difference and second-order difference of the cepstral coefficients on the time axis based on the cepstral coefficients, the first-order difference and the second-order difference respectively characterizing the change features of the target audio stream on the time axis; and combining the basic acoustic features with the dynamic change features as the time-frequency domain features of the target audio stream.

[0012] Preferably, the method generates an initial text sequence based on the time-frequency domain features and the fine-grained waveform features, corrects the initial text sequence, and obtains the final speech text information, including: fusing the time-frequency domain features and the fine-grained waveform features as the input of the speech recognition module in the multimodal recognition model, using an encoder to context-encode the input information, and extracting a high-dimensional acoustic feature sequence; establishing a global mapping relationship between the speech sequence and the text sequence in the high-dimensional acoustic feature sequence through a decoder, and generating a coherent initial text sequence based on a hybrid loss function and a constrained search algorithm; and verifying the initial text sequence according to the confidence score of each text segment to obtain the final speech text information.

[0013] Preferably, the voice waveform of the user's real-time voice in combination with the voice text information is used as the input of the multimodal recognition model to determine whether the voice conversation is interrupted and identify the interruption intention, and obtain the voice interruption result, including: dividing the voice text information into word sequences based on the model-based word segmenter and converting them into word embedding vectors; extracting semantic features according to the self-attention mechanism through a text encoder based on the word embedding vectors; aligning the voice waveform with the voice text information according to the timestamp, and extracting time series features through an acoustic encoder; based on the semantic features and the time series features, calculating the feature correlation between text and acoustics through a cross-modal fusion layer, and generating a multimodal fusion feature containing text semantic information, acoustic features and text-acoustic correlation; analyzing the multimodal fusion feature based on multi-task branches, determining whether the voice conversation is interrupted and identifying the interruption intention, and obtaining the voice interruption result.

[0014] Preferably, the multi-task branch-based analysis of the multimodal fusion features, determination of whether the voice conversation is interrupted and identification of the interruption intention, and acquisition of the voice interruption result includes: pooling the multimodal fusion features based on the voice interruption task branch, and calculating the interruption probability based on the pooled features; comparing the interruption probability with a preset probability threshold to determine whether a voice interruption occurs; in the event of a voice interruption, extracting context-dependent multimodal fusion features based on the intention recognition task branch, and mapping them to the intention label space to determine the intention type of the interruption content; and filling the intention-question slot of the intention type to obtain a voice interruption result including the interruption type and the interruption intention.

[0015] Preferably, the interruption probability is compared with a preset probability threshold to determine whether voice interruption occurs, including: when the interruption probability is less than the minimum probability threshold, the system continues to output; when the interruption probability is greater than the maximum probability threshold, it is determined that voice interruption occurs; when the interruption probability is between the minimum probability threshold and the maximum probability threshold, verifying whether voice interruption occurs based on the context information of the multimodal fusion feature.

[0016] Preferably, the method of obtaining the user's new intention based on the voice interruption result and dynamically adjusting the system response strategy to achieve interaction optimization includes: extracting the interruption type and interruption intention based on the voice interruption result, saving the context of the current voice text information to a key-value pair database according to the interruption type, and dynamically switching the system task state; generating a response text according to the interruption intention, and converting it into natural speech to output to the user; recording the user's behavioral feedback on the response, and optimizing the response strategy based on the reinforcement learning mechanism of the multimodal recognition model, including adjusting the length of the response output text, to achieve dialogue interaction optimization between the system and the user.

[0017] Preferably, the method further comprises:

[0018] The multimodal recognition model is trained and optimized based on the voice and text information of the user's interruption behavior, specifically including: cross-modal data annotation of the conversation voice data and text data of the user's interruption behavior, including the time information of the interruption behavior, the interruption type, the interruption intention, the acoustic features, and the context information of the interrupted text; based on the annotated time information, interruption type, interruption intention, acoustic features, and the context information of the interrupted text, a semantic-acoustic-intention modal association label is established; and the annotated data and the modal association label are used as the optimized training parameters of the multimodal recognition model to train and optimize the multi-task branches.

[0019] In a second aspect, an embodiment of the present application provides a voice interaction optimization system based on a multimodal large model, comprising:

[0020] The acquisition module is used to perform speech recognition in response to the user's real-time speech and obtain speech text information; the analysis module is used to combine the speech text information with the speech waveform of the user's real-time speech as the input of the multimodal recognition model to determine whether the voice conversation has been interrupted and identify the interruption intention, and obtain the speech interruption result; the response module is used to obtain the user's new intention based on the speech interruption result and dynamically adjust the system response strategy to achieve interaction optimization.

[0021] Beneficial effects of this application:

[0022] 1. Through a cross-modal attention mechanism, text semantics and acoustic features are processed simultaneously to achieve end-to-end interruption probability and intent recognition. It also supports the recognition of non-explicit interruptions and complex intents, significantly reducing missed and false positive rates and improving the system's ability to more comprehensively and intelligently identify interruptions.

[0023] 2. This application leverages the natural language generation and context modeling capabilities of the large model to dynamically switch system task states based on the interruption type and dynamically generate responses based on the interruption intent. This overcomes the problem of fixed-strategy dialogue management, which is unable to intelligently adjust responses based on the content of the interruption (such as new questions, corrections), resulting in stiff and inflexible interactions, and significantly improves the naturalness and intelligence of the dialogue.

[0024] 3. This application optimizes the multimodal recognition model through multi-task learning strategy training, while optimizing interruption types, intent recognition, and acoustic anomaly detection, sharing underlying feature representations, and combining data enhancement (such as noise superposition and speech rate changes) in training to improve generalization capabilities, so as to achieve continuous optimization of the recognition model, thereby ensuring the model's accurate detection and intent recognition of user interruption behaviors, improving the system's adaptability to diverse scenarios, and further enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Other features, objects, and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are provided for illustration purposes only and are not to be construed as limiting the present application. Like reference numerals are used throughout the drawings to denote like parts.

[0026] Figure 1 A flowchart of a voice interaction optimization method based on a multimodal large model provided in an embodiment of the present application.

[0027] Figure 2 A schematic diagram of the functional modules of a multimodal recognition model provided in an embodiment of the present application.

[0028] Figure 3 A schematic diagram of state transition in dynamic dialogue management provided in an embodiment of the present application.

[0029] Figure 4 A flowchart of speech interaction training optimization based on a multimodal large model is provided in an embodiment of the present application.

[0030] Figure 5 Schematic diagram of the voice interaction optimization system module based on a multimodal large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application more clear, the application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific implementation method described here is only an optimal embodiment of this application, which is only used to explain this application and does not limit the scope of protection of this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0032] Some nouns or terms that appear in the description of the embodiments of this application are subject to the following interpretations:

[0033] Transformer framework: The architecture consists of an encoder and a decoder. Each layer incorporates multi-head attention and a feed-forward network (FFN). Training is stabilized through residual connections and layer normalization. The self-attention mechanism computes the association weights of all positions in the sequence in parallel, addressing the problem of long-distance dependencies.

[0034] LayerNorm (layer normalization): is a standardization operation used to eliminate the amplitude differences of the input data on each neuron and ensure that the data distribution of each layer output is similar;

[0035] Linear transformation: refers to the linear mapping operation, which realizes the linear transformation from input features to output features through the parameter matrix W. It is mainly used to adjust the dimension of the feature space or integrate information.

[0036] BiLSTM+CRF architecture: This is a sequence labeling architecture that combines deep learning and statistical models. It is used for sequence labeling of natural language processing tasks. It uses BERT to obtain global semantic vectors, BiLSTM to capture contextual features, and CRF to optimize label sequence prediction.

[0037] Example 1: Figure 1 As shown, a voice interaction optimization method based on a multimodal large model includes steps S101-S103, wherein:

[0038] S101: Perform voice recognition in response to the user's real-time voice to obtain voice text information.

[0039] In this embodiment, real-time user voice signals are collected using a high-performance microphone array. Automatic speech recognition (ASR) technology, based on a large model, converts the audio waveform into a text sequence in real time. The acquisition process employs multi-channel signal processing to enhance sound source directionality, and a deep learning noise reduction model removes interference. The ASR model, pre-trained using self-supervised learning methods, can directly extract speech features from the raw waveform and decode it into text. It supports recognition in multiple languages ​​(such as Chinese, English, and Japanese), with varying accents, and in complex noise environments.

[0040] As an optional implementation, the above step S101 specifically includes:

[0041] S1011. Collect user voice signals in real time, dynamically adjust signal gain according to the direction of the sound source, and suppress lateral interference of the voice signals to obtain the target audio stream;

[0042] S1012: Generate a spectrogram according to the target audio stream to calculate cepstral coefficients and their differences to obtain time-frequency domain features;

[0043] S1013. Capturing fine-grained acoustic information of the original waveform data of the target audio stream to obtain fine-grained waveform features;

[0044] S1014: Generate an initial text sequence based on the time-frequency domain features and the fine-grained waveform features, and modify the initial text sequence to obtain final speech text information.

[0045] In some possible embodiments, user voice signals are collected using a high-performance 4-channel microphone array with a sampling rate of 16kHz and a bit depth of 16 bits, ensuring high audio fidelity. The acquisition process utilizes adaptive beamforming technology, dynamically adjusting gain based on the direction of the sound source to enhance the target user's voice while suppressing lateral interference. To cope with complex environments, the system integrates a deep learning-based noise reduction model that uses a convolutional neural network to analyze spectral characteristics in real time and remove background noise such as wind, vehicle horns, and crowd noise. Automatic gain control and echo cancellation technologies ensure stable volume and the absence of echo interference, outputting a high-quality audio stream.

[0046] In some embodiments, the captured audio stream is framed, with each frame length of 25 milliseconds and a frame shift of 10 milliseconds. Hamming windows are used to reduce spectral leakage. Subsequently, a spectrogram is generated through a short-time Fourier transform (SFT). 40-dimensional Mel-frequency cepstral coefficients, first-order differences, and second-order differences (a total of 120 dimensions) are calculated as basic time-frequency domain features. Simultaneously, the original waveform data (sampling rate of 16kHz) is retained as direct input to a larger model (i.e., a multimodal recognition model), leveraging its self-supervised learning capabilities to capture fine-grained acoustic information, such as pitch fine-tuning and energy distribution, to obtain fine-grained waveform features.

[0047] In this embodiment, automatic speech recognition technology is used to extract time-frequency domain features and fine-grained waveform features, thereby comprehensively capturing the spectral characteristics and dynamic change patterns of the speech signal. For example, the spectral envelope of the speech reflects the resonance characteristics of the vocal tract and the dynamic changes of the speech over time, such as phoneme transitions and changes in speaking rate, providing comprehensive and accurate data support for subsequent interruption detection and intention recognition.

[0048] As an optional implementation, the above step S1012 specifically includes:

[0049] After framing the target audio stream, generate a spectrogram through Fourier transform;

[0050] Calculating cepstral coefficients of the target audio stream according to the spectrogram to obtain basic acoustic features of the target audio stream;

[0051] Calculating a first-order difference and a second-order difference of the cepstral coefficients on a time axis based on the cepstral coefficients, wherein the first-order difference and the second-order difference respectively represent change characteristics of the target audio stream on the time axis;

[0052] The basic acoustic features and the dynamic change features are combined as time-frequency domain features of the target audio stream.

[0053] In some possible embodiments, the spectrogram is converted to a Mel spectrum using a Mel filter bank, the Mel spectrum is logarithmized, and then the dimensionality is reduced using a discrete cosine transform (DCT) to obtain the Cepstral Coefficients (MFCCs). MFCCs typically use the first 12-13 dimensions, but in this embodiment, MFCCs use 40 dimensions, including higher-order high-frequency coefficients, to ensure more significant acoustic features are captured.

[0054] The speech information corresponding to MFCCs includes the spectral envelope, timbre, and static characteristics of phonemes, which represent the speech signal. The spectral envelope, the overall shape of the spectrum, is used to distinguish vowels from consonants; timbre distinguishes male, female, or specific voices; and the static characteristics of phonemes distinguish between voiceless and voiced consonants.

[0055] Furthermore, first-order differences are the rate of change of MFCCs over time, that is, the difference between MFCCs in adjacent frames. This is used to supplement the dynamic characteristics of speech, including trends in phonemes, speech rate and rhythm, and intonation. For example, speech is continuous, and transitions between phonemes, such as the transition from "b" to "a" in "ba," cause the spectral envelope to change over time. First-order differences can capture the "speed" of this transition. For example, when the pitch rises, the high-frequency components of the MFCCs increase over time, and the first-order differences exhibit a positive characteristic. Second-order differences are the rate of change of first-order differences, that is, the difference between the first-order differences of adjacent frames. This is used to capture faster and more dramatic dynamic changes in speech, including sudden speech events, strong fluctuations in emotion, and the dynamic characteristics of noise or interference. For example, plosives ("p," "t," and "k"), as well as accents (emphasis on a syllable), cause the spectral envelope to change dramatically in a short period of time, significantly increasing the second-order differences. For example, when pronouncing "p," the sudden burst of airflow causes a sudden surge in spectral energy, resulting in a peak in the second-order difference coefficient. Although the second-order difference mainly targets the speech itself, it can also indirectly reflect the sudden changes in background noise, thereby improving the robustness of noise capture.

[0056] As an optional implementation, the above step S1014 specifically includes:

[0057] The time-frequency domain features and the fine-grained waveform features are integrated as input to the speech recognition module in the multimodal recognition model, and an encoder is used to contextually encode the input information to extract a high-dimensional acoustic feature sequence;

[0058] Establishing a global mapping relationship between the speech sequence and the text sequence in the high-dimensional acoustic feature sequence through a decoder, and generating a coherent initial text sequence based on a hybrid loss function and a constrained search algorithm;

[0059] The initial text sequence is verified according to the confidence score of each text segment to obtain the final speech text information.

[0060] In some possible embodiments, the speech recognition module in the multimodal recognition model integrates a Transformer framework, consisting of a 24-layer encoder and a 12-layer decoder, and employs a self-attention mechanism to process temporal information. The encoder extracts high-dimensional feature representations (1024 dimensions) from the original waveform, while the decoder generates a text sequence using a concatenated temporal classification (CTC) loss and a constrained search algorithm (beam size = 15). The encoder's core objective is to convert the original speech waveform (time-domain signal) into a high-dimensional acoustic feature sequence containing contextual dependencies, providing the decoder with an "understandable" speech representation. The decoder's core objective is to generate a text sequence consistent with the speech content based on the high-dimensional acoustic features output by the encoder.

[0061] In some embodiments, time-frequency domain features and fine-grained waveform features are low-dimensional time-frequency feature sequences, which are mapped to high-dimensional spatial inputs such as 512 or 1024 dimensions through a linear layer. These are then input into stacked Transformer encoder layers, where a multi-head self-attention mechanism is used to capture long-range contextual dependencies within the speech signal, such as the association between initials and finals in speech and the relationship between stress and preceding and following syllables. Specifically, each attention head calculates the association weights between different frames (such as the similarity between the second and seventh frames) through a matrix operation of "query, key, and value," and then aggregates these weighted features to form features containing contextual information. The multi-head design can capture dependencies of different scales in parallel.

[0062] Furthermore, the features output by the self-attention algorithm are nonlinearly transformed to enhance their nonlinear expressiveness and capture fine local acoustic details. The outputs of each encoder layer are connected via residuals to obtain high-dimensional acoustic features. The decoder uses the correspondence between speech frames and text characters. To address the ambiguity in the temporal alignment of speech and text, a hybrid loss function is used to align the high-dimensional acoustic features using a blank character mechanism, ignoring redundant alignment. This establishes a global mapping between speech and text sequences within the high-dimensional acoustic features and encodes the entire context sequence.

[0063] Furthermore, for the encoded context sequence, a constrained search algorithm is used, starting from an empty sequence or start symbol. Combining the statistical patterns of subsequent words in a large-scale text corpus, the probability distribution of the next candidate word in the preceding word sequence is generated. For each time step, multiple candidate sequences are generated, each containing a candidate word and a predicted probability. The candidate words belonging to high-frequency domain vocabulary are searched among these multiple candidate sequences, and their probabilities are weighted and increased. The candidate sequence with the highest weighted probability is selected as the output, thus obtaining a coherent initial text sequence. The conditional probability of the entire text is the product of the conditional probabilities of all words.

[0064] Specifically, the Transformer encoder outputs the probability that the acoustic feature corresponding to each time step belongs to a certain phoneme / word, which is used as the acoustic probability. For the entire text, the acoustic average probability of the corresponding words at all time steps can be calculated, and the acoustic average probability is weighted and fused with the text conditional probability, and normalized to obtain the final confidence score; the confidence score is compared with the preset confidence threshold. If the confidence score is lower than the confidence threshold, resampling is triggered or the user is prompted to repeat the input through voice.

[0065] In some examples, the acoustic average probability and the text conditional probability are weighted and fused, with each weighting 50%. The scores are normalized to the range of 0-1 using the Sigmoid function to obtain the final confidence score. When the score is lower than a threshold, such as 0.85, the text is judged to be unreliable, triggering resampling (re-extraction of acoustic features) or prompting the user to repeat the input ("Please say it again") to reduce the risk of error propagation.

[0066] In this embodiment, the context prediction word probability provides a basis for semantic rationality, high-frequency domain vocabulary enhances scene adaptability, and the confidence score provides the system with a basis for text sequence verification. The three together improve the practicality and accuracy of the system.

[0067] S102: Using the voice text information in combination with the voice waveform of the user's real-time voice as input to a multimodal recognition model to determine whether the voice conversation is interrupted and identify the interruption intention, thereby obtaining a voice interruption result.

[0068] As an optional implementation, the above step S102 specifically includes:

[0069] S1021. A model-based word segmenter segments the speech text information into word sequences and converts them into word embedding vectors;

[0070] S1022. Based on the word embedding vector, semantic features are extracted through the text encoder according to the self-attention mechanism;

[0071] S1023, aligning the speech waveform with the speech text information according to the timestamp, and extracting time series features through an acoustic encoder;

[0072] S1024. Based on the semantic features and the time series features, calculating the feature correlation between text and acoustics through a cross-modal fusion layer, and generating a multimodal fusion feature including text semantic information, acoustic features, and text-acoustic correlation;

[0073] S1025. Analyze the multimodal fusion features based on the multi-task branches, determine whether the voice conversation is interrupted, identify the interruption intention, and obtain the voice interruption result.

[0074] In some possible embodiments, the speech text information and the original speech waveform obtained in step S101 are used as inputs of a multimodal recognition model and are preprocessed separately, wherein the text is segmented into word sequences by a word segmenter (such as Chinese jieba, English NLTK, Japanese MeCab) and converted into a 768-dimensional word embedding vector (such as a pre-trained representation based on BERT); the speech waveform retains a 16kHz sampling rate and is synchronized with the text through timestamp alignment to ensure temporal consistency of cross-modal data.

[0075] Further, combined with Figure 2 As shown in the figure, the multimodal recognition model includes a text encoder (16 layers), an acoustic encoder (16 layers) and a cross-modal fusion layer (4 layers). The text encoder extracts semantic features (such as sentence vectors, dimension 1024) through the self-attention mechanism and analyzes the semantic structure and vocabulary distribution of the input; the acoustic encoder processes the waveform sequence and extracts time series features (such as volume change trend, fundamental frequency fluctuation, dimension 1024); the cross-modal fusion layer uses a multi-head attention mechanism to calculate the correlation between text and acoustics (such as the matching degree between "wait a moment" and the sudden increase in volume), generates a unified cross-modal representation (dimension 1536), and obtains multimodal fusion features.

[0076] In some examples, the semantic feature sequence output by the text encoder is , where T is the length of the text; the time series features output by the acoustic encoder are , where S is the length of the text. Before cross-modal fusion, A and X need to be aligned based on the time dimension and feature projection, as follows:

[0077] Through linear interpolation or attention pooling, A is converted into a sequence of the same length as X, expressed as ;

[0078] X and Projecting them into the query (Q), key (K), and value (V) spaces respectively, we can get:

[0079] 、 、 、 、 、 ;

[0080] in, 、 and is the learnable parameter matrix with dimension , ;

[0081] The interaction between text and acoustics is calculated using a multi-head attention mechanism to capture the correlation between text and acoustics. The attention calculation method is expressed as follows:

[0082] Self-attention within text: ;

[0083] Text Focus on Acoustics: ;

[0084] Acoustic Intra-Self-Attention: ;

[0085] Acoustic Focus Text: ;

[0086] Then we can calculate the attention weight: ; Among them, the attention weight matrix It directly reflects the feature relevance between text and acoustics. For example, the feature vector of "wait a moment" in the text may have a high attention weight with the frame with a sudden increase in volume in the acoustics.

[0087] Furthermore, based on multi-head attention, four outputs of multimodal feature interaction and aggregation can be obtained: text self-attention , Acoustic Self-Attention , text→acoustic cross attention , acoustic→text cross attention , based on the above four attention outputs splicing or weighted aggregation to obtain the attention of multimodal features: ; Project the aggregated features to the target dimension (1536 dimensions) to obtain a unified representation of the features:

[0088] , This representation simultaneously includes textual semantic information, such as the intent of "wait a moment," acoustic features, such as sudden volume increases and fundamental frequency changes, and text-acoustic correlations, such as the degree of matching between "wait a moment" and pauses in the acoustics. In this embodiment, the aggregated features projected onto the target dimension are used as multimodal fusion features.

[0089] In other embodiments, to optimize computational efficiency, the system may segment the speech waveform into segments, such as 2 seconds per segment, and mark key time points (such as volume peaks) to facilitate large-model focused analysis.

[0090] In other embodiments, the model supports dynamic adjustment of attention weights to prioritize high-confidence modalities. For example, when the text confidence is high (e.g., 0.95) and the confidence of noisy speech is 0.7, the weights of text self-attention and text→acoustic cross-attention are increased; when the speech is clear but the text may be misrecognized, the weights of acoustic self-attention and acoustic→text cross-attention are increased. By dynamically adjusting attention weights based on confidence, the model prioritizes processing acoustic or text features, avoiding simultaneous analysis of all text and acoustic features, reducing the number of features to be processed and improving feature analysis efficiency.

[0091] In this embodiment, by stacking multiple cross-modal fusion layers, the interaction between text and acoustics can be gradually refined, ultimately outputting a unified multimodal fusion feature representation. This achieves a precise correspondence between text vocabulary and speech features, and can complement and enhance text information and acoustic features. For example, the grammatical structure of the text can compensate for ambiguous pronunciation in speech, while the prosodic information of speech can be used to assist in understanding the text's sentiment, such as matching interrogative tone with question words. By integrating multimodal fusion features that strongly correlate text and acoustics, the accuracy and reliability of voice conversation interruption detection are ensured, providing decision support for accurately identifying new user intent and making dynamic responses.

[0092] As an optional implementation, combined with Figure 3 As shown, the above step S1025 specifically includes:

[0093] Pooling the multimodal fusion features based on the voice interruption task branch, and calculating the interruption probability based on the pooled features;

[0094] Comparing the interruption probability with a preset probability threshold to determine whether voice interruption occurs;

[0095] In the event of speech interruption, context-dependent multimodal fusion features are extracted based on the intent recognition task branch and mapped to the intent label space to determine the intent type of the interruption content;

[0096] The intent-question slot of the intent type is filled to obtain a voice interruption result including the interruption type and the interruption intent.

[0097] Specifically, comparing the interruption probability with a preset probability threshold to determine whether voice interruption occurs includes:

[0098] When the interruption probability is less than the minimum probability threshold, the system continues to output;

[0099] When the interruption probability is greater than a maximum probability threshold, determining that a voice interruption occurs;

[0100] When the interruption probability is between a minimum probability threshold and a maximum probability threshold, verifying whether speech interruption occurs based on context information of the multimodal fusion feature.

[0101] In some possible embodiments, the voice interruption task branch outputs the interruption probability based on multimodal fusion features through a two-layer fully connected network + Softmax, as follows:

[0102] Multimodal fusion features Pooling is performed to obtain a global representation; pooling can be performed using average pooling or attention pooling, and the average pooling is expressed as ; Attention pooling through learnable weight vector The multimodal fusion features are weighted and expressed as ; Features after pooling Contains global associations between text semantics and acoustic features;

[0103] The pooled features are classified through a two-layer fully connected network to output the interruption probability, which is classified as follows:

[0104] ;

[0105] ;

[0106] ;

[0107] in, The input layer is responsible for receiving pooling features ; and is the weight matrix, , ; and is the bias term; is the output layer, which outputs the classification result p; p is the final interruption probability; ReLU represents the activation function of the fully connected layer, which is used to introduce nonlinear characteristics and enhance the expressiveness of the model. It maps the input signal to the output through nonlinear transformation, outputs 0 when the input is negative, and directly outputs the input feature when it is non-negative.

[0108] In some examples, a dual threshold mechanism is used to verify the interruption probability output by the model. For example, the highest threshold is 0.9 and the lowest threshold is 0.7. If the probability exceeds 0.9, the interruption is confirmed; if the probability is lower than 0.7, the current output continues; if it is between 0.7-0.9, the system calls the model's context analysis module (based on the Transformer memory network) and combines the conversation history, such as the context of the previous 5 seconds, to determine whether it is a false trigger to avoid misjudgment. For example, if the user says "no" and the context is an error correction scenario, the interruption is confirmed.

[0109] In some possible embodiments, the intent recognition task branch performs fine-grained classification of interruption content based on the BiLSTM+CRF architecture, as follows:

[0110] Fusion of multimodal features As the input of BiLSTM, the feature sequence is processed forward and backward to capture context-dependent features;

[0111] The context-dependent features are mapped to the label space based on the linear layer, and the labels represent different intent types;

[0112] Based on the transition probability between labels learned by the CRF layer, the optimal label sequence is determined by the Viterbi algorithm. The optimal label sequence is expressed as: , where y represents the label sequence, P represents the transition probability between labels, and s represents the label space;

[0113] The intention type of the interruption content is determined based on the optimal label sequence, and the intent-question slot is filled according to the intent type to obtain the specific content of the user's new intention to interrupt the voice conversation.

[0114] Among them, the intent types include stop, continue, new question, correction, and confirmation; specifically, the intent type can be determined according to the actual scenario; according to the intent type, the intent-question slot can be filled through entity recognition or template matching. For example, if the interruption probability is greater than 0.9 and the interruption type is "new question", the specific task can be further identified, such as checking the weather, and the intent-question is filled in as "new question: check the weather"; that is, if the user inputs "check the weather later", it is classified as "new question: check the weather"; if the user inputs "No, I want to go to Shanghai", it is classified as "correction: destination → Shanghai".

[0115] In this embodiment, multiple task branches share fusion features, enabling the model to simultaneously use the interruption type and interruption content for accurate intent recognition. When the interruption probability is greater than the highest probability threshold, intent recognition is initiated. This multi-task collaboration improves the generalization ability of the model, accurately identifies the user's interruption intention, and achieves more natural and smooth human-computer dialogue interaction.

[0116] S103: Obtain the user's new intention based on the voice interruption result, and dynamically adjust the system response strategy to achieve interaction optimization.

[0117] As an optional implementation, the above step S103 specifically includes:

[0118] S1031, extracting the interruption type and interruption intention based on the voice interruption result, saving the context of the current voice text information to a key-value pair database according to the interruption type, and dynamically switching the system task state;

[0119] S1032: Generate a response text according to the interruption intention, and convert it into natural speech and output it to the user;

[0120] S1033. Record the user's behavioral feedback on the response, and optimize the response strategy based on the reinforcement learning mechanism of the multimodal recognition model, including adjusting the length of the response output text, to achieve optimization of the dialogue interaction between the system and the user.

[0121] In some possible embodiments, combined with Figure 4 As shown, the system task status includes "system output", "waiting for input", "new task processing", etc. If the interruption is a new question, the model saves the current context to the key-value database and switches the task status; in this embodiment, context modeling supports up to 20 rounds of historical tracking, focusing on relevant information through the attention mechanism to ensure the consistency of multi-task switching.

[0122] Specifically, the natural language generation module of the multimodal recognition model generates response text based on the interruption intention; if the intention is "stop", a concise reply is generated, such as "OK, it has been paused, please continue"; if it is a "new question", a detailed answer is generated after calling an external service (such as the flight API), such as "Please tell me the departure and destination, and I will check for you"; the response text is synthesized into natural speech through the text-to-speech module, with adjustable timbre and support for multi-language output.

[0123] Furthermore, the system records user behavioral feedback on responses, such as whether they reply immediately or repeatedly interrupt, and optimizes response strategies through the model's reinforcement learning module. For example, if a user frequently interrupts a long answer, the model shortens the output length, for example, from 20 words to 10. In addition, the system supports voice sentiment analysis. If the user is detected to be impatient, such as speaking faster, concise responses are prioritized to improve interaction efficiency. Among other things, a reinforcement learning module can be built based on the PPO algorithm, with user satisfaction used as the reward function, allowing the system to adjust strategies in a targeted manner to improve the user experience.

[0124] This embodiment leverages the natural language generation module and context modeling capabilities of a multimodal recognition model to dynamically generate responses based on interruption intent. A state machine is used to manage the conversation flow, supporting multi-task switching and context recovery. This addresses the existing problem of fixed conversation management strategies, inability to intelligently adjust responses based on the content of the interruption, and resulting in stiff and inflexible interactions. By dynamically switching system task states based on interruption type and intent, and optimizing response strategies based on user behavioral feedback, this approach seamlessly adapts to complex interaction scenarios while taking into account user experience, significantly improving the naturalness and intelligence of conversations.

[0125] As an optional implementation, combined with Figure 3 As shown, the method further includes:

[0126] S104: Training and optimizing the multimodal recognition model based on the voice and text information of the user's interruption behavior; specifically including:

[0127] Cross-modal data annotation of conversational voice and text data of user interruptions, including the time information of the interruption, interruption type, interruption intention, acoustic features, and contextual information of the interrupted text;

[0128] Establishing a semantic-acoustic-intent modal association label based on the annotated time information, interruption type, interruption intention, acoustic features, and context information of the interrupted text;

[0129] The labeled data and the modality association labels are used as optimized training parameters of the multimodal recognition model to perform training optimization on the multi-task branches.

[0130] In some possible embodiments, cross-modal data annotation includes manual annotation and automatic annotation, wherein automatic annotation includes: time information, interruption type, interruption intention and contextual information of the interrupted text; specifically: recording the start and end time points of the interruption behavior, determining the time range of the interruption corresponding to the time axis of the original speech waveform, and annotating the time range of the interrupted content; interruption types include explicit interruptions and implicit interruptions; wherein, explicit interruptions represent interruptions triggered by words or phrases, and implicit interruptions represent interruptions reflected by acoustic features; interruption intentions include annotating the specific intention of the interruption based on semantic and acoustic features; annotating the contextual information when the interruption occurs includes the content of the interrupted text and the logical relationship between the interruption and the context. Manual annotation includes: annotating the key acoustic properties of the interrupted speech, including at least volume features, fundamental frequency features, speech rate features and background noise.

[0131] It should be noted that data annotation provides task-related supervisory signals for the model. Massive unlabeled audio-text pairs allow the model to learn general cross-modal associations, but they cannot learn task rules for specific scenarios such as "interruptions." The judgment of interruptions relies on the synergy between text semantics and acoustic features, and unlabeled data cannot provide supervision for "semantic-acoustic matching." Therefore, by annotating data, we can establish a precise "semantic-acoustic-intent" association for the model through information such as timestamps and acoustic features. This allows us to use associated labels to optimize the loss function and adjust model parameters through backpropagation to accurately improve model performance.

[0132] In some embodiments, pre-training of a multimodal recognition model involves unsupervised pre-training of the model using a large number of unlabeled audio-text pairs, allowing the model to grasp the general association between "speech waveform" and "text semantics" and provide basic feature extraction capabilities for subsequent fine-tuning. The pre-training task involves having the text encoder predict the mask content based on the context and speech waveform, and having the acoustic encoder predict the acoustic features of the masked segment based on the unmasked waveform and text content. The data input is the unlabeled speech waveform and the corresponding text, the optimization goal is to minimize the loss of the pre-training task, and the output is the pre-trained text encoder, acoustic encoder, and cross-modal fusion layer.

[0133] Furthermore, we leveraged the labeled data to adjust parameters based on the pre-trained model and optimize multi-task objectives, allowing the model to focus on learning the judgment rules and intent recognition capabilities of "interruption behavior." The optimized multi-task objectives include interruption classification, intent recognition, and acoustic anomaly detection; the details are as follows:

[0134] Input the labeled "speech waveform + corresponding text + interruption label" data and perform acoustic transformation on the labeled data. For example, add random noise, adjust the speaking speed by ±20%, and simulate different accents to improve the model's robustness to noise and speaking speed changes.

[0135] The cross-modal fusion layer calculates the correlation between the two through multi-head attention and generates fusion features;

[0136] The fused features are processed through multi-task branches to output the interruption probability and intent category;

[0137] Based on the multi-objective loss, the model parameters are adjusted to simultaneously optimize multiple task objectives until the harmonic mean of the interruption classification accuracy, intent recognition precision, and recall on the labeled dataset no longer increases. The multi-objective loss is the weighted sum of the losses of each task branch. For the speech interruption task branch, the cross-entropy loss is used to calculate the difference between the predicted interruption probability and the true label. For the intent recognition task branch, the BiLSTM+CRF loss function is used. For the cross-modal fusion layer, the cross-entropy loss is used to optimize the calculation accuracy of the semantic-acoustic correlation. The loss weights of each task branch can be 0.5:0.3:0.2, and the AdamW optimizer is used for training and optimization.

[0138] In some embodiments, a "high confidence modality first" mechanism can be added to fine-tuning: for example, if the input text confidence is higher than the speech noise confidence, the attention weight of the text feature is increased during cross-modal fusion to reduce the interference of speech noise.

[0139] In this embodiment, robustness is improved through data enhancement such as Gaussian noise, speaking speed ±20%, and accent synthesis. The labeled data includes the time information of the interruption behavior, the interruption type, the interruption intention, acoustic features, and the context information of the interrupted text. During the training process, the model will use these annotations as supervision signals to reversely adjust the cross-modal fusion layer and multi-task branches, thereby improving the accuracy of interruption judgment and the robustness of interruption type and intention recognition.

[0140] Example 2, based on the same inventive concept, the present application also provides a voice interaction optimization system based on a multimodal large model corresponding to the voice interaction optimization method based on a multimodal large model, such as Figure 5 As shown, the system includes:

[0141] An acquisition module 501 is configured to perform speech recognition in response to a user's real-time speech and acquire speech text information;

[0142] An analysis module 502 is configured to use the voice text information in combination with the user's real-time voice waveform as input to a multimodal recognition model to determine whether the voice conversation has been interrupted, identify the interruption intention, and obtain a voice interruption result;

[0143] The response module 503 is used to obtain the user's new intention according to the voice interruption result and dynamically adjust the system response strategy to achieve interaction optimization.

[0144] As an optional implementation, the acquisition module 501 specifically includes: a first acquisition submodule, which is used to collect user voice signals in real time, dynamically adjust the signal gain according to the direction of the sound source, and suppress lateral interference of the voice signal to obtain the target audio stream; a second acquisition submodule, which is used to generate a spectrum diagram based on the target audio stream to calculate the cepstral coefficients and their differences, and obtain time-frequency domain features; a third acquisition submodule, which is used to capture fine-grained acoustic information of the original waveform data of the target audio stream to obtain fine-grained waveform features; a fourth acquisition submodule, which is used to generate an initial text sequence based on the time-frequency domain features and the fine-grained waveform features, and to correct the initial text sequence to obtain the final speech text information.

[0145] Specifically, the second acquisition submodule is specifically used to: generate a spectrogram through Fourier transform after framing the target audio stream; calculate the cepstral coefficients of the target audio stream according to the spectrogram to obtain the basic acoustic features of the target audio stream; calculate the first-order difference and second-order difference on the time axis based on the cepstral coefficients, and the first-order difference and the second-order difference respectively characterize the dynamic change characteristics of the target audio stream on the time axis; combine the basic acoustic features with the dynamic change features as the time-frequency domain features of the target audio stream.

[0146] Specifically, the fourth acquisition submodule is specifically used to: fuse the time-frequency domain features and the fine-grained waveform features as the input of the speech recognition module in the multimodal recognition model, use the encoder to context-encode the input information, and extract the high-dimensional acoustic feature sequence; establish a global mapping relationship between the speech sequence and the text sequence in the high-dimensional acoustic feature sequence through the decoder, and generate a coherent initial text sequence based on the hybrid loss function and the constrained search algorithm; verify the initial text sequence according to the confidence score of each paragraph of text to obtain the final speech text information.

[0147] As an optional implementation, the analysis module 502 specifically includes: a first analysis submodule, which is used to segment the speech text information into word sequences based on a model-based word segmenter and convert them into word embedding vectors; a second analysis submodule, which is used to extract semantic features based on the word embedding vectors through a text encoder according to a self-attention mechanism; a third analysis submodule, which is used to align the speech waveform with the speech text information according to a timestamp and extract time series features through an acoustic encoder; a fourth analysis submodule, which is used to calculate the feature correlation between text and acoustics through a cross-modal fusion layer based on the semantic features and the time series features, and generate a multimodal fusion feature containing text semantic information, acoustic features, and text-acoustic correlation;

[0148] The fifth analysis submodule is used to analyze the multimodal fusion features based on the multi-task branches, determine whether the voice conversation is interrupted and identify the interruption intention, and obtain the voice interruption result.

[0149] Specifically, the fifth analysis submodule is specifically used to: pool the multimodal fusion features based on the voice interruption task branch, and calculate the interruption probability based on the pooled features; compare the interruption probability with the preset probability threshold to determine whether voice interruption occurs, wherein: when the interruption probability is less than the minimum probability threshold, the system continues to output; when the interruption probability is greater than the maximum probability threshold, it is determined that voice interruption occurs; when the interruption probability is between the minimum probability threshold and the maximum probability threshold, verify whether voice interruption occurs based on the context information of the multimodal fusion features; in the case of voice interruption, extract context-dependent multimodal fusion features based on the intention recognition task branch, and map them to the intention label space to determine the intention type of the interruption content; fill the intention-question slot of the intention type to obtain a voice interruption result including the interruption type and the interruption intention.

[0150] As an optional implementation, the response module 503 is specifically used to: extract the interruption type and interruption intention based on the voice interruption result, save the context of the current voice text information to the key-value pair database according to the interruption type, and dynamically switch the system task state; generate a response text according to the interruption intention, and convert it into natural speech to output to the user; record the user's behavioral feedback on the response, and optimize the response strategy based on the reinforcement learning mechanism of the multimodal recognition model, including adjusting the length of the response output text, to achieve optimization of the dialogue interaction between the system and the user.

[0151] As an optional implementation, the response module 503 is also specifically used to: train and optimize the multimodal recognition model based on the voice and text information of the user's interruption behavior, specifically including: cross-modal data annotation of the conversation voice data and text data of the user's interruption behavior, including the time information of the interruption behavior, the interruption type, the interruption intention, the acoustic features and the context information of the interrupted text; establish a semantic-acoustic-intention modal association label based on the annotated time information, interruption type, interruption intention, acoustic features and the context information of the interrupted text; use the annotated data and the modal association label as the optimized training parameters of the multimodal recognition model to train and optimize the multi-task branches.

[0152] The above specific implementation methods are preferred implementation methods of the present application, and are not intended to limit the specific implementation scope of the present application. The scope of the present application includes but is not limited to the specific implementation methods. Any equivalent changes made in accordance with the shape, structure, and method of the present application are within the scope of protection of the present application.

Claims

1. A voice interaction optimization method based on a multimodal large model, characterized by: The steps include: Performing voice recognition in response to real-time user voice to obtain voice text information; The voice text information is combined with the voice waveform of the user's real-time voice as input to the multimodal recognition model to determine whether the voice conversation is interrupted and identify the interruption intention, thereby obtaining a voice interruption result; Acquire the user's new intention based on the voice interruption result, and dynamically adjust the system response strategy to achieve interaction optimization; The method further comprises: The multimodal recognition model is trained and optimized based on the voice and text information of the user's interruption behavior, specifically including: Cross-modal data annotation of conversational voice and text data of user interruptions, including the time information of the interruption, interruption type, interruption intention, acoustic features, and contextual information of the interrupted text; Establishing a semantic-acoustic-intent modal association label based on the annotated time information, interruption type, interruption intention, acoustic features, and context information of the interrupted text; The labeled data and the modality association labels are used as optimized training parameters of the multimodal recognition model to perform training optimization on the multi-task branches.

2. The method for optimizing voice interaction based on a multimodal large model according to claim 1, characterized in that: The step of performing speech recognition in response to the user's real-time speech and obtaining speech text information includes: Collect user voice signals in real time, dynamically adjust signal gain according to the direction of the sound source, and suppress lateral interference of the voice signal to obtain the target audio stream; Generating a spectrogram according to the target audio stream to calculate cepstral coefficients and their differences to obtain time-frequency domain features; Capturing fine-grained acoustic information of the original waveform data of the target audio stream to obtain fine-grained waveform features; An initial text sequence is generated based on the time-frequency domain features and the fine-grained waveform features, and the initial text sequence is modified to obtain final speech text information.

3. The method for optimizing speech interaction based on a multimodal large model according to claim 2, characterized in that: Generating a spectrogram according to the target audio stream to calculate cepstral coefficients and their differences to obtain time-frequency domain features includes: After framing the target audio stream, generate a spectrogram through Fourier transform; Calculating cepstral coefficients of the target audio stream according to the spectrogram to obtain basic acoustic features of the target audio stream; Calculating a first-order difference and a second-order difference of the cepstral coefficients on a time axis based on the cepstral coefficients, wherein the first-order difference and the second-order difference respectively represent dynamic change characteristics of the target audio stream on the time axis; The basic acoustic features and the dynamic change features are combined as time-frequency domain features of the target audio stream.

4. The method for optimizing speech interaction based on a multimodal large model according to claim 2, characterized in that: Generating an initial text sequence based on the time-frequency domain features and the fine-grained waveform features, and correcting the initial text sequence to obtain final speech text information includes: The time-frequency domain features and the fine-grained waveform features are integrated as input to the speech recognition module in the multimodal recognition model, and an encoder is used to contextually encode the input information to extract a high-dimensional acoustic feature sequence; Establishing a global mapping relationship between the speech sequence and the text sequence in the high-dimensional acoustic feature sequence through a decoder, and generating a coherent initial text sequence based on a hybrid loss function and a constrained search algorithm; The initial text sequence is verified according to the confidence score of each text segment to obtain the final speech text information.

5. The method for optimizing voice interaction based on a multimodal large model according to claim 2, characterized in that: The method of combining the voice text information with the voice waveform of the user's real-time voice as input to the multimodal recognition model to determine whether the voice conversation is interrupted and identify the interruption intention, and obtain the voice interruption result, includes: The model-based word segmenter divides the speech text information into word sequences and converts them into word embedding vectors; Based on the word embedding vector, semantic features are extracted through the text encoder according to the self-attention mechanism; Aligning the speech waveform with the speech text information according to the timestamp, and extracting time series features through an acoustic encoder; Based on the semantic features and the time series features, the feature correlation between the text and the acoustics is calculated through a cross-modal fusion layer to generate a multimodal fusion feature containing text semantic information, acoustic features, and text-acoustic correlation; The multimodal fusion features are analyzed based on the multi-task branches to determine whether the voice conversation is interrupted and identify the interruption intention, and obtain the voice interruption result.

6. The method for optimizing voice interaction based on a multimodal large model according to claim 5, characterized in that: The multi-task branch-based analysis of the multimodal fusion features, determining whether the voice conversation is interrupted and identifying the interruption intention, and obtaining the voice interruption result includes: Pooling the multimodal fusion features based on the voice interruption task branch, and calculating the interruption probability based on the pooled features; Comparing the interruption probability with a preset probability threshold to determine whether voice interruption occurs; In the event of speech interruption, context-dependent multimodal fusion features are extracted based on the intent recognition task branch and mapped to the intent label space to determine the intent type of the interruption content; The intent-question slot of the intent type is filled to obtain a voice interruption result including the interruption type and the interruption intent.

7. The method for optimizing voice interaction based on a multimodal large model according to claim 6, characterized in that: Comparing the interruption probability with a preset probability threshold to determine whether voice interruption occurs includes: When the interruption probability is less than the minimum probability threshold, the system continues to output; When the interruption probability is greater than a maximum probability threshold, determining that a voice interruption occurs; When the interruption probability is between a minimum probability threshold and a maximum probability threshold, verifying whether speech interruption occurs based on context information of the multimodal fusion feature.

8. The method for optimizing voice interaction based on a multimodal large model according to claim 6, characterized in that: The acquiring of the user's new intention according to the voice interruption result and the dynamic adjustment of the system response strategy to achieve interaction optimization include: Extracting the interruption type and interruption intention based on the voice interruption result, saving the context of the current voice text information to a key-value pair database according to the interruption type, and dynamically switching the system task state; Generate response text according to the interruption intention, and convert it into natural speech and output it to the user; Record the user's behavioral feedback on the response, and optimize the response strategy based on the reinforcement learning mechanism of the multimodal recognition model, including adjusting the length of the response output text, to achieve optimized dialogue interaction between the system and the user.

9. A voice interaction optimization system based on a multimodal large model, characterized by: The method for optimizing speech interaction based on a multimodal large model according to any one of claims 1 to 8, wherein the system comprises: An acquisition module, configured to perform speech recognition in response to the user's real-time speech and acquire speech text information; An analysis module is configured to use the voice text information in combination with the voice waveform of the user's real-time voice as input to a multimodal recognition model to determine whether the voice conversation has been interrupted, identify the interruption intention, and obtain a voice interruption result; The response module is used to obtain the user's new intention based on the voice interruption result and dynamically adjust the system response strategy to achieve interaction optimization.

Images