Cross-language voice interaction method and system based on multi-modal semantic understanding, and storage medium

Abstract

The invention discloses a cross-language voice interaction method and system based on multi-modal semantic understanding and a storage medium, and relates to the technical field of artificial intelligence and natural language processing. The method comprises the following steps: acquiring a source language speech stream, performing parallel double-flow feature extraction, extracting text semantic features through a mixed language code recognition model based on a unified phoneme space, and extracting acoustic features containing rhythm information at the same time; performing alignment fusion on the text and the acoustic features by using a cross-modal attention mechanism to generate multi-modal semantic representation; analyzing the explicit intention and the implicit emotion based on the representation, and generating a reply strategy and an emotion control label of the target language; and finally, synthesizing a target voice with a corresponding emotion color. According to the invention, the bottleneck that the traditional cascade architecture loses side language information is broken through, the precise understanding and strategic feedback of complex contexts such as Chinese-English mixed language codes (Code-mixing), anti-quiescence, hesitation and the like are realized, and the method is particularly suitable for transnational business negotiation and international customer service scenes.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, specifically to a cross-language voice interaction method, system, and storage medium capable of processing code-mixing, recognizing paralinguistics, and providing emotional feedback. This invention is particularly suitable for highly complex interaction scenarios such as cross-border business negotiations, international intelligent customer service, and multilingual conference assistance. Background Technology

[0002] With the deepening of global economic integration, the demand for cross-language communication is growing exponentially. Traditional cross-language voice interaction systems typically employ a cascade architecture, a serial processing mode of "Automatic Speech Recognition (ASR) - Machine Translation (MT) - Text-to-Speech (TTS)". While this mode performs adequately in standardized tourism or everyday conversation scenarios, it exhibits significant technical shortcomings in complex business negotiations and specific language environments such as Southeast Asia. 1. Significant Loss of Emotional and Paralinguistic Information: Existing ASR systems often filter out acoustic features such as pitch, stress, and pauses when converting speech to text. In business negotiations, the speaker's attitude (such as hesitation, sarcasm, or eagerness) is often hidden in these acoustic features, rather than in the text itself. For example, when a user says sarcastically, "You guys shipped really fast," a traditional system can only translate the literal meaning, leading the recipient to misinterpret it as praise, causing serious interaction problems.

[0003] 2. Insufficient code-mixing processing capabilities. Existing technologies typically require pre-specifying a single language. When faced with mixed codes, frequent language ID switching often leads to truncated recognition, high latency, or garbled characters, failing to meet real-time interaction requirements.

[0004] 3. Data privacy and compliance challenges: In scenarios involving cross-border services, directly uploading users' raw voice data to the cloud for model training faces increasingly stringent data protection regulations (such as the EU GDPR, Singapore PDPA, and China's Measures for Security Assessment of Cross-border Data Transfer).

[0005] Therefore, there is an urgent need for a cross-language voice interaction method that can integrate acoustic and textual multimodal information, support zero-latency mixed code recognition, and have data privacy protection capabilities. Summary of the Invention

[0006] The purpose of this invention is to provide a cross-language voice interaction method, system, and storage medium based on multimodal semantic understanding, so as to solve the problems of emotion loss, difficulty in mixed code recognition, and poor data compliance in the prior art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a cross-language voice interaction method based on multimodal semantic understanding, comprising: S1: Obtain the original speech stream of the source language, and use the preprocessing module to perform noise reduction, pre-emphasis and frame segmentation on the speech stream; S2: Construct parallel dual-stream feature extraction channels to extract text semantic feature vectors and acoustic prosodic feature vectors respectively; the text channel is constructed based on a unified phoneme space, supporting seamless recognition of mixed codes; S3: Employ a multi-head cross-modal attention mechanism to align and fuse dual-stream features, generating a multimodal semantic representation vector containing sentiment weights; S4: Input the multimodal semantic representation vector into the strategy generation network, and generate the response text in the target language and sentiment style control tags through intent slot filling and sentiment polarity discrimination; S5: Input the target language response text and the emotional style control label into the emotional speech synthesis model to generate a target language speech stream with corresponding emotional coloring.

[0008] The beneficial effects of this invention are as follows: 1. High-precision emotion understanding: By introducing acoustic prosodic features, it effectively identifies implicit emotions such as "irony" and "hesitation", improving the success rate of interactions in business negotiation scenarios.

[0009] 2. Zero-latency mixed language recognition: Through unified phoneme space technology, it solves the recognition problem in mixed language environments such as Singlish in Southeast Asia, without the need to switch engines.

[0010] 3. Data security and compliance: By introducing federated learning and differential privacy technologies, the model can be adaptively calibrated without uploading the original recordings, which complies with international data compliance requirements. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the cross-language voice interaction method provided in Embodiment 1 of the present invention.

[0012] Figure 2 This is a network architecture diagram of the unified phoneme space hybrid code recognition model in this invention.

[0013] Figure 3This is a schematic diagram illustrating the principle of the Cross-Modal Attention mechanism in this invention.

[0014] Figure 4 This is a schematic diagram of the privacy protection architecture based on federated learning in this invention.

[0015] Figure 5 This is a hardware structure block diagram of the electronic device provided in the embodiments of the present invention. Detailed Implementation

[0016] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0017] Example 1: System Architecture and Hardware Environment The method provided in this invention can run on a server, cloud platform, or edge computing device. The electronic device includes, but is not limited to, at least one processor, memory, communication interface, and bus. The processor can be a graphics processing unit (GPU) or a neural network processing unit (NPU) for performing inference computations on deep learning models. The memory is used to store weight files for the hybrid code-sound model, multimodal fusion model, and policy generation model.

[0018] Example 2: Hybrid Code Recognition (Unified Phoneme Space Technology) When dealing with Singlish from Southeast Asia, it is common to see a mix of Chinese and English, such as "Can you help me da bao?".

[0019] Existing technologies typically employ a "Language Detection (LID) + Switching Model" approach, with latency usually exceeding 500ms. This invention proposes a Unified Phoneme Space technology: 1. Phoneme Mapping: Define a universal phoneme table P = {p_en, p_cn, p_dialects}. Map English phonetic symbols (such as / d / ) and Chinese pinyin initials (such as d) to a shared embedding space.

[0020] 2. Model Training: A Transformer-Transducer (RNN-T) architecture is used. In the encoder layer, the Mel-spectral features of the mixed speech are input; in the predictor layer, the probability distribution of the next phoneme is predicted directly, regardless of which language the phoneme belongs to.

[0021] 3. Results: When "da bao" is recognized, the model automatically selects the Chinese phoneme path based on acoustic features and context, without misrecognizing the English word "double".

[0022] Example 3: Cross-modal emotion fusion and irony recognition This embodiment describes in detail how to identify "sarcasm". Suppose the input voice is: "Your prices are really too 'affordable'." 1. Text feature extraction: The semantic vector V_text of "discount" is extracted using the BERT model, and its sentiment polarity score is +0.8 (positive).

[0023] 2. Acoustic Feature Extraction: The acoustic feature V_audio of this segment is extracted using a CNN. The system detected: The base frequency (F0) exhibits abnormal "low-high-low" jitter; Duration: The pronunciation of this word lasted 40% longer than the average. Energy: Abnormal accent placement.

[0024] The overall acoustic emotion score is -0.6 (negative, indicating dissatisfaction).

[0025] 3. Attention Fusion: To fuse the features from both streams, this embodiment employs a scaled dot product attention mechanism, calculated as follows: Attention(Q, K, V) = softmax( (Q × K^T) / √d_k ) × V In this model, the text vector serves as the query vector Q, and the acoustic vector serves as the key vector K and value vector V. Due to the conflicting sentiment polarities of Q and K (+0.8 vs -0.6), the fusion layer automatically assigns a higher weight α_audio to the acoustic features.

[0026] 4. Judgment result: The final multimodal semantic representation vector is biased towards the negative, and the system judges the intent as "price complaint / irony".

[0027] Example 4: Intent-based policy generation (Intent Reshaping) After recognizing the aforementioned "ironic" intent, the system proceeds to the strategy generation stage: 1. Intent slot resolution: User_Intent = COMPLAIN_PRICE (Price Complaint).

[0028] 2. Strategy Retrieval: Search the preset negotiation strategy library and match the strategy STRATEGY_EMPATHY_AND_DISCOUNT (empathy + discount guidance).

[0029] 3. Response generation: Common translation: "Your price is very favorable." (Incorrect, it adds fuel to the fire.)

[0030] This invention generates: "I understand your concern about the cost. Let's see if we can adjust the package."

[0031] 4. Emotion Synthesis: The TTS module loads the [SINCERE] emotion tag to generate a slow-paced, gentle voice to reduce the other party's defensiveness.

[0032] Example 5: Data Privacy Protection Based on Federated Learning To address restrictions on cross-border data transfer (such as China's "Measures for Security Assessment of Cross-border Data Transfer" and Singapore's PDPA), this invention designs a federated learning update mechanism: 1. Local computation: The gradient ∇θ of the model parameters is calculated on a local edge server in Singapore using the user's correction data (e.g., the user manually modified the recognition results).

[0033] 2. Differential Privacy Noise Addition: Laplace noise is added to the gradient ∇θ, calculated as follows: M(D) = f(D) + Laplace( Δf / ε ) Where D is the dataset, f is the query function, Δf is the sensitivity, and ε is the privacy budget. This formula adds random noise to ensure that even if gradients are leaked, attackers cannot reverse engineer the user's original speech.

[0034] 3. Encrypted upload: Only the encrypted gradients are uploaded to the cloud at the Hangzhou headquarters.

[0035] 4. Global Aggregation: The Hangzhou headquarters uses the FedAvg algorithm to aggregate gradients from around the world, update the general model, and then distribute it to subsidiaries in various regions.

Claims

1. A cross-language voice interaction method based on multimodal semantic understanding, characterized in that, Includes the following steps: S1: Obtain the original speech stream of the source language, and use the preprocessing module to perform noise reduction and frame segmentation on the original speech stream; S2: Perform parallel dual-stream feature extraction on the framed speech stream to obtain text semantic feature vector and acoustic prosodic feature vector respectively; S3: Utilize the Multi-Head Cross-Modal Attention mechanism to align and weight the text semantic feature vector with the acoustic prosodic feature vector at time steps to generate a multimodal semantic representation vector; S4: Input the multimodal semantic representation vector into the strategy generation network, parse the user's explicit intent and implicit emotional state, and generate the response text in the target language and the corresponding sentiment style control tag. S5: Based on the target language response text and sentiment style control tags, generate and output the target language speech stream using the sentiment speech synthesis model.

2. The method according to claim 1, characterized in that, The extraction of text semantic feature vectors in step S2 is achieved through a hybrid code recognition model based on the Unified Phoneme Space. The method for constructing the hybrid code recognition model includes: constructing a shared high-dimensional vector mapping space to map the first phoneme set of the source language (including the Chinese Pinyin scheme), the second phoneme set of the target language (including the International Phonetic Alphabet IPA), and the phoneme set of a specific dialect to the same vector coordinate system; During the decoding process, an end-to-end architecture is adopted, which allows the decoder to predict the posterior probabilities of phonemes from different languages ​​within the same time step, so as to achieve continuous recognition of code-mixing sentences without the need for prior language classification (LID).

3. The method according to claim 1, characterized in that, The extraction of acoustic prosodic feature vectors in step S2 specifically includes: Mel-frequency cepstral coefficients (MFCC) and filter bank features (Fbank) of speech frames are extracted using a convolutional neural network (CNN). The fundamental frequency (F0) trajectory, energy profile, and phoneme duration information of the speech stream are extracted using a Long Short-Term Memory (LSTM) network. The above features are concatenated to form an acoustic prosodic feature vector that includes paralinguistic information.

4. The method according to claim 1, characterized in that, Step S3 specifically includes irony recognition logic: Calculate the sentiment polarity score of the text semantic feature vectors respectively. Stext Emotional polarity score with acoustic prosodic feature vector Saudio ; When detected Stext It is positive polarity. Saudio It is of negative polarity, and the absolute value of the difference between the two is | Stext - Saudio | When the threshold δ is exceeded, the current interaction context is determined to be ironic (Sarcasm). When generating the multimodal semantic representation vector, the weight ratio of the acoustic prosodic feature vector is automatically increased, while the weight ratio of the text semantic feature vector is decreased.

5. The method according to claim 1, characterized in that, Step S4 involves generating a response text in the target language and executing an intent reshaping strategy, including: Identify and filter non-semantic pauses, fillers, and slips of the tongue in the source language; Based on the identified implicit emotional states, when generating response text in the target language, preset business negotiation templates or reassuring words are automatically inserted, rather than a literal translation of the source language text.

6. The method according to claim 1, characterized in that, This method also includes a model calibration step based on federated learning: On the local terminal side, the gradient deviation between the current user's voice features and the general model is calculated based on the user's interaction feedback data. The gradient bias is processed using differential privacy, which generates an encrypted gradient by superimposing random noise from a Laplace distribution or a Gaussian distribution. The encrypted gradient is uploaded to a cloud server for global model parameter aggregation, but the original audio stream is not uploaded.

7. A cross-language voice interaction system based on multimodal semantic understanding, characterized in that, include: The data acquisition module is used to acquire the raw speech stream of the source language; The feature extraction module is configured to run in parallel a hybrid code recognition model based on a unified phoneme space and an acoustic prosodic analysis model to extract text semantic features and acoustic prosodic features, respectively. The multimodal fusion module is used to generate multimodal semantic representation vectors using a cross-modal attention mechanism and to perform irony and sentiment conflict detection. The strategy generation module is used to generate target language response text containing intent strategies and sentiment style control tags based on multimodal semantic representation vectors. The speech synthesis module is used to output the target speech based on the reply text and tags.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 6.

Images