Self-adaptive interactive language learning system capable of multimodal emotion calculation and matching method of self-adaptive interactive language learning system

Abstract

The invention discloses a multi-modal emotion calculation enabling adaptive interactive language learning system and a matching method thereof, and belongs to the technical field of artificial intelligence, and the system comprises a multi-modal emotion real-time perception and quantification module MHAE-Net, a personalized adaptive dialogue and intervention strategy module ADIS-RL, and a virtual dialogue partner interactive interface module EC-NLG. The multi-modal emotion real-time perception and quantification module MHAE-Net is composed of a voice signal acquisition and processing unit, a voice emotion analysis unit, a text emotion analysis unit, a facial expression emotion analysis unit, a physiological signal emotion analysis unit and a multi-modal emotion fusion and decision-making unit. The interactive strategy is dynamically adjusted, the oral anxiety of the language is effectively relieved, the oral confidence is improved, the system adopts the multi-mode emotion calculation technology, the accuracy and robustness of emotion recognition are improved, the targeted interactive strategy is designed for different emotion states, and personalized emotion support and language practice are provided for learners.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to an adaptive interactive language learning system enabled by multimodal affective computing and a supporting method thereof. Background Art

[0002] Foreign language speaking anxiety (FLSA) is a common psychological barrier in language learning. It not only affects learners' motivation but also significantly reduces their oral expression ability. Current language learning tools and platforms often focus on imparting language knowledge and correcting pronunciation, while ignoring the importance of learners' emotional state. These tools and platforms lack a deep understanding of learners' emotional needs and lack effective intervention measures, leading to the following problems:

[0003] Lack of learning motivation: Due to lack of emotional support and encouragement, learners will find the learning process boring and tedious, which will reduce their learning motivation;

[0004] Oral expression is limited. Oral anxiety causes learners to be afraid of making mistakes and dare not practice speaking, thus limiting the improvement of oral expression ability.

[0005] Social interaction disorder: in oral communication in a foreign language, anxiety hinders learners from performing normally, affecting their social interaction and cross-cultural communication abilities;

[0006] Low learning efficiency. An unstable emotional state directly affects learning efficiency. Learners will find it difficult to concentrate under anxiety, resulting in poor learning results.

[0007] The deep integration of affective computing technology into language learning AI tutors aims to build a system that can monitor learners' emotional states in real time and make corresponding adjustments. However, research and application in this field are still in their infancy and have the following shortcomings:

[0008] Emotion recognition technology is immature. Although significant progress has been made, it still needs to be improved in terms of accuracy and real-time performance, especially in complex and changing language learning environments.

[0009] Adaptive strategies are limited. Existing adaptive interaction strategies are often simple and difficult to meet the individual needs of different learners.

[0010] The system's interactivity is insufficient, and existing AI tutors still need to be improved in providing emotional support and interactive experience to better simulate real interpersonal communication.

[0011] Based on this, the present invention designs an adaptive interactive language learning system enabled by multimodal affective computing and its supporting method to solve the above problems. Summary of the Invention

[0012] The purpose of the present invention is to propose an adaptive interactive language learning system enabled by multimodal affective computing and its supporting method in order to solve the problem that current language learning tools and platforms often focus on imparting language knowledge and correcting pronunciation, but ignore the importance of learners' emotional state. These tools and platforms lack an in-depth understanding of learners' emotional needs and effective intervention measures.

[0013] In order to achieve the above object, the present invention adopts the following technical solutions:

[0014] An adaptive interactive language learning system powered by multimodal affective computing and its supporting methods, including the multimodal emotion real-time perception and quantification module MHAE-Net, the personalized adaptive dialogue and intervention strategy module ADIS-RL, and the virtual dialogue partner interaction interface module EC-NLG;

[0015] The multimodal emotion real-time perception and quantification module MHAE-Net consists of a speech signal acquisition and processing unit, a speech emotion analysis unit, a text emotion analysis unit, a facial expression emotion analysis unit, a physiological signal emotion analysis unit, and a multimodal emotion fusion and decision-making unit;

[0016] The personalized adaptive dialogue and intervention strategy module ADIS-RL consists of a dialogue management unit, an emotion adaptation strategy engine, and a pronunciation and fluency assessment unit;

[0017] The virtual conversation partner interaction interface module EC-NLG consists of an emotion consistency natural language generation system, an emotion expression speech synthesis system, an avatar expression and action generation system, and user interface design and personalization.

[0018] As a further description of the above technical solution:

[0019] In the speech signal acquisition and processing unit, the system collects learners' speech signals in real time and converts speech to text using advanced end-to-end models, such as Conformer or Whisper OpenAI. For spoken language scenarios, the model needs to be fine-tuned on a spoken language dataset containing a large amount of non-fluency, pauses, and repetitions. Its objective function can be expressed as:

[0020]

[0021] Among them, x i Represents the characteristics of the speech signal, y i Represents the corresponding text label, θ represents the model parameter, and N represents the number of samples;

[0022] At the same time, the system extracts the acoustic features of the speech, including the following categories:

[0023] Time domain features, energy mean, zero crossing rate (ZCR), and voice segment ratio;

[0024] Frequency domain features, MFCC, formant frequency and bandwidth;

[0025] Sound quality characteristics, fundamental frequency jitter, amplitude shimmer, harmonic-to-noise ratio (HNR);

[0026] Prosodic features, pitch contour, speaking rate, pause distribution and intensity variation;

[0027] These features are extracted using the OpenSMILE toolkit in the following steps:

[0028] Audio is framed, with a frame length of 25ms and a frame shift of 10ms;

[0029] Apply a Hamming window to each frame

[0030] Calculate various acoustic features, such as MFCC:

[0031]

[0032] Where S(j) is the output of the Mel filter bank and M is the number of filters;

[0033] Finally, an 88-dimensional feature vector based on the eGeMAPS standard is formed;

[0034] The speech emotion analysis unit uses a deep learning model to analyze the extracted acoustic features and designs a hybrid model based on the combination of Transformer encoder and BiLSTM;

[0035] The unit consists of the following core components:

[0036] Multi-head self-attention mechanism to capture long-range dependencies in feature sequences;

[0037]

[0038] MultiHead(Q,K,V)=Concat(head1,...,head h )W O

[0039] in

[0040] BiLSTM layer, further captures the temporal information of features;

[0041]

[0042]

[0043]

[0044] Multi-task output layer, which performs emotion classification and emotion dimension value regression simultaneously;

[0045] P(y class |x)=softmax(W class h+b class )

[0046] y dimensions =σ(W dim h+b dim )

[0047] The model training adopts a multi-task learning framework, and the loss function is:

[0048]

[0049] in is the cross entropy loss for classification tasks, is the mean square error loss of the regression task, α is the weight parameter to balance the two tasks, is the L2 regularization term.

[0050] As a further description of the above technical solution:

[0051] The text sentiment analysis unit performs a series of processing on the ASR converted text:

[0052] Text cleaning, removing catchphrases, correcting ASR errors, and applying regular expression filtering;

[0053] Token segmentation and standardization, using NLTK or spaCy for token segmentation;

[0054] Feature extraction, using pre-trained BERT, RoBERTa, or DeBERTa models to extract text representations;

[0055] The mathematical expression of the model is:

[0056] h CLS =TransformerEncoder(Embedding(TokenIDs)) CLS

[0057] y emotion =Classifier(Dropout(h CLS ))

[0058] In order to improve the accuracy of text sentiment analysis, we integrate sentiment dictionary and syntactic structure analysis to build a sentiment feature enhancer:

[0059]

[0060] where f lex (T) is the feature based on the sentiment dictionary, f syn (T) is based on the characteristics of syntactic structure, Represents feature concatenation operation;

[0061] The facial expression emotion analysis unit,

[0062] If the user authorizes the use of the camera, the system captures the learner's facial image in real time and processes it according to the following steps:

[0063] Face detection and alignment: Use the MTCNN or RetinaFace algorithm to detect faces and extract 68 key points:

[0064]

[0065] Among them B i are the bounding box coordinates, S i is the detection confidence, L i is the set of key point coordinates;

[0066] Facial action unit AUs extraction, based on facial key points, uses OpenFace tools to extract AUs intensity and existence:

[0067]

[0068] It contains 17 main AUs, each of which has an intensity value (0-5) and existence (0 / 1);

[0069] Graph Attention Network modeling, using GAT to model the topological relationship between facial key points or AUs:

[0070]

[0071]

[0072] Temporal modeling, using 3D-CNN or Transformer to process continuous frame sequences;

[0073] For 3D-CNN: F t+1 =Conv3D(F t-k：t );

[0074] For Transformer, {F1',...,F T '}=TransformerEncoder({F1,...,F T});

[0075] Emotional representation extraction, and finally output visual embedding vector E visual .

[0076] As a further description of the above technical solution:

[0077] In the physiological signal emotion analysis unit, if the user wears a compatible wearable device, the system processes the following physiological signals:

[0078] Heart rate variability (HRV) analysis:

[0079] Time domain indicators, SDNN standard deviation, RMSSD root mean square of the difference between adjacent RR intervals;

[0080]

[0081]

[0082] Frequency domain indicators, using fast Fourier transform (FFT) to obtain LF0.04-0.15Hz, HF0.15-0.4Hz power and LF / HF ratio;

[0083] Skin conductivity GSR / EDA analysis;

[0084] Skin conductance level SCL, low-frequency baseline changes, reflects the overall level of arousal;

[0085] Skin conductance response (SCR), peak feature extraction, including number, amplitude and rise time;

[0086] SCR amplitude =max(GSR peak )-GSR onset

[0087] SCR risetime =t peak -t onset

[0088] Signal processing:

[0089] Low-pass filtering removes high-frequency noise,

[0090] Missing value interpolation, based on spline interpolation or forward filling;

[0091] Timestamp alignment, aligning physiological signals of different sampling rates with frame-level speech / visual features;

[0092] Temporal coding, using temporal convolutional networks (TCN) or LSTM networks to encode physiological signal sequences:

[0093] h t =LSTM(x t ,ht-1 )

[0094] or

[0095]

[0096] Where d is the dilation factor, and the final output is the physiological embedding vector E physio ;

[0097] The multimodal emotion fusion and decision-making unit fuses the emotion analysis results of each modality and adopts the following strategy:

[0098] Cross-modal Attention Fusion:

[0099]

[0100]

[0101] Among them E i represents the embedding vector of modality i, and f is the similarity measurement function;

[0102] Gated multimodal fusion:

[0103]

[0104] g i =σ(W g [h1,h2,...,h M ]+b g )

[0105] where g i is the gating weight of modality i, ⊙ denotes element-wise multiplication;

[0106] Tensor fusion, capturing high-order interactions through low-rank multimodal decomposition:

[0107]

[0108] Where ×1 represents the modulo-1 product of the tensor, represents the outer product operation;

[0109] Through the above fusion strategy, the system outputs the standardized emotional state vector E emotion ,Include:

[0110] Emotion classification results and confidence

[0111] Emotional dimension values: anxiety a∈[0,1], confidence c∈[0,1];

[0112] The selection of fusion strategy is based on a dynamic weighting algorithm, which is adaptively adjusted according to the reliability and consistency of each modality;

[0113]

[0114] where λ i is the reliability measure of mode i, and T is the temperature parameter.

[0115] As a further description of the above technical solution:

[0116] The dialogue management unit uses advanced natural language processing technology to analyze the learner's intentions and generate coherent and natural dialogue responses. Its architecture includes:

[0117] Intent recognition and slot filling use the joint BERT model to simultaneously complete intent recognition and slot filling tasks:

[0118] P(I,S|X)=P(I|X)·P(S|X)

[0119] P(I|X)=softmax(W I ·h [CLS] +b I )

[0120] P(s j |x j )=softmax(W S ·h j +b S )

[0121] Among them, X represents the input user speech, I represents the intent category, S represents the slot sequence, and h [CLS] is the [CLS] tag representation of BERT, h j is the representation of the jth word;

[0122] Recognized intent types include:

[0123] Information query Information_Request;

[0124] Clarification_Request;

[0125] Opinion_Expression;

[0126] Emotion_Expression;

[0127] Help_Request;

[0128] Feedback_Response;

[0129] Dialogue state tracking maintains a structured dialogue state DST, which contains the following information:

[0130] Current Topic T curr , Active slot set S active 、Dialogue History H t-k：t , user emotional state E user and the system's previous response R t-1 ;

[0131] State update function:

[0132] DST t =f update (DST t-1 ,I t ,S t ,E user,t )

[0133] Among them, f update It is a hybrid update function based on rules and learning, implemented as follows:

[0134] Topic tracking, updating the current topic based on the latest intent and slot;

[0135]

[0136] Slot update, updating or clearing slots based on new information;

[0137] S active,t =(S active,t-1 \S clear )∪S new

[0138] History compression, using recursive attention networks to compress conversation history;

[0139] H compressed =RecurrentAttention(H t-k：t )

[0140] The dialogue strategy decision is based on the output of the emotion adaptation strategy engine to determine the dialogue strategy:

[0141] a t =π(DST t ,E user,t ,g ADIS )

[0142] where g ADIS is the policy instruction of the emotion adaptive policy engine, and π is the policy function, which is implemented based on the deep Q network (DQN):

[0143]

[0144] The policy decision optimization goal is to maximize the weighted sum of user satisfaction and sentiment improvement:

[0145]

[0146] The knowledge enhancement mechanism enhances the conversational capabilities through the knowledge graph:

[0147] KG=(E,R,T)

[0148] Where E is the entity set, R is the relationship set, is a set of triples;

[0149] For knowledge queries in the field of language learning, the graph attention network GAT is used to extract relevant knowledge:

[0150] z e =GAT(e,N(e);Θ)

[0151] where w is the query entity, N(e) is its neighbor entities, and Θ is the model parameter.

[0152] As a further description of the above technical solution:

[0153] The emotion adaptation strategy engine is the core innovation unit of the system. It is based on hierarchical reinforcement learning (HRL). The system architecture includes:

[0154] Hierarchical reinforcement learning framework, high-level controller Meta-Controller:

[0155] State space, S H ={E user ,U profile ,T context ,G learning};

[0156] E user Emotional state vectors such as anxiety and confidence, U profile User profile language level, learning history, preferences, T context Theme context and G learning Completion of learning objectives;

[0157] Action space A H ={g1,g2,...,g K}, including the following macro strategies:

[0158] g1 represents Soothe & Encourage, g2 represents Difficulty Adjustment, g3 represents Relaxation Guidance, g4 represents Focus Redirection, g5 represents Achievement Reinforcement, g6 represents Gentle Correction, and state transition represents S H,t+1 =f H (S H,t ,g t ,A L,t );

[0159] Reward function:

[0160] R H (S H,t ,g t ,S H,t+1 )=w1ΔE anxiety +w2ΔE confidence +w3P engagement +w4F user

[0161] Where ΔE anxiety is the decrease in anxiety, ΔE confidence is the confidence improvement value, P engagement is the participation measure, F user It is explicit feedback from users;

[0162] Low-level controller:

[0163] State space S L ={D context ,E micro ,T complexity ,g H}、D context Current conversation micro-context, E micro Fine-grained emotional fluctuations, T complexity Current task complexity, g H High-level policy instructions and action space are based on g H Determine, if g H =G2 difficulty adjustment:

[0164] a1 means switching to a simpler topic, a2 means providing keyword prompts, a3 means providing sentence templates, and a4 means simplifying the question;

[0165] Reward function, combining internal and external rewards:

[0166] R L (SL,t ,a t ,S L,t+1 )=R internal +γI goal ·R H

[0167] where R internal It is the immediate feedback of a specific action. goal is the goal completion indicator, γ is the discount factor

[0168] Learning Algorithms

[0169] The high-level controller uses the PPO algorithm:

[0170]

[0171] in is the advantage estimate;

[0172] The low-level controller uses the SAC (Soft Actor-Critic) algorithm:

[0173]

[0174] in is the policy entropy, α is the temperature parameter;

[0175] Strategy library implementation, anxiety relief strategy sub-library:

[0176] Active empathy and encouragement, identifying anxiety triggers: E anxiety >θ anxiety Generate parameterized templates for encouraging words:

[0177] T encourage (p1,p2)={t1(p1,p2),t2(p1,p2),...,t n (p1,p2)}

[0178] Among them, p1 is the anxiety level parameter, and p2 is the personalized style parameter;

[0179] The specific implementation is through the adaptive discourse selection function:

[0180]

[0181] Difficulty is dynamically adjusted, and task complexity evaluation function is:

[0182] C task =f complexity (L sentence ,R grammar ,V vocabulary )

[0183] Among them L sentence is the sentence length, R grammar is the grammatical complexity, V vocabulary is the vocabulary difficulty;

[0184] Dynamic adjustment algorithm:

[0185] if E_anxiety>θ_high:

[0186] C_target=C_current*(1-α_high)

[0187] elseifE_anxiety>θ_medium:

[0188] C_target=C_current*(1-α_medium

[0189] else:

[0190] C_target=min(C_current*(1+β),C_max)

[0191] Then select the task closest to the target complexity from the task library:

[0192]

[0193] Guided relaxation intervention: Trigger condition: E anxiety >θ relax ∧tt last_relax >τ interval ;

[0194] A progressive strategy for relaxation coaching:

[0195] Level 1, simple breathing prompts ("Take a deep breath...")

[0196] Level 2, guided breathing exercises (3 seconds inhale - 4 seconds hold - 5 seconds exhale)

[0197] Level 3, brief mindfulness guidance (focusing on the present moment)

[0198] Decision tree algorithm for selecting a strategy:

[0199]

[0200] Non-judgmental error correction feedback, error correction sensitivity adjustment function:

[0201] S correction =S base ·(1-β·E anxiety )

[0202] Among them S base is the basic error correction sensitivity, β is the adjustment coefficient;

[0203] Expression adjustment, with expression selected along a continuous spectrum from mild to direct;

[0204] Style correction =lerp(Style gentle ,Style direct ,1-E anxiety ) is implemented as a parameterized error correction template:

[0205] M correction (error,severity,anxiety)

[0206] =template select (error_type,severity,anxiety)

[0207] Confidence Building Strategy Sublibrary:

[0208] Progressive challenge, implemented as an adaptive difficulty curve:

[0209] C next =C current +ΔC(E confidence ,P success )

[0210] Where ΔC is the difficulty increment function:

[0211] ΔC=α·E confidence +β·P success -γ·E anxiety

[0212] P success is the recent success rate, α, β, γ are weight coefficients;

[0213] The challenge sequence is designed as a Markov decision process (MDP), and the optimal path is calculated through value iteration;

[0214] Achievement enhancement and improved detection algorithm:

[0215]

[0216] When Progress>θ progress Praise is triggered when the progress is made, and the intensity of the praise is proportional to the degree of progress:

[0217] Intensity praise =min(1,α·Progress)

[0218] A virtual reward system that accumulates points based on progress;

[0219] Points t =Points t-1 +Base points (1+β·Progress)

[0220] Achievement unlocking conditions, when points or specific abilities reach a threshold;

[0221]

[0222] As a further description of the above technical solution:

[0223] The pronunciation and fluency assessment unit objectively evaluates learners' oral performance and provides a basis for dialogue strategies;

[0224] Pronunciation assessment module, phoneme-level scoring, using acoustic models to calculate phoneme deviations:

[0225]

[0226] Where d is the distance metric based on MFCC, P user and P native is the feature vector of the user and standard pronunciation; word-level score, combined with phoneme score and prosodic features:

[0227]

[0228] Sentence-level scoring, taking into account multiple factors such as linking and intonation:

[0229] Score sentence =f combine (Score words ,Rhythm,Intonation sentence ) Fluency assessment module, pause analysis, calculation of unnatural pause rate:

[0230]

[0231] Speech speed analysis, calculation of effective speech speed, effective words per minute:

[0232]

[0233] Repetition and self-correction, detection of repeated words and phrase correction:

[0234]

[0235]

[0236] Overall Fluency Score:

[0237]

[0238] Where norm() is a normalization function that maps the speaking rate to the interval [0,1];

[0239] Feedback control dynamically adjusts the presentation of evaluation results based on the instructions of the emotional adaptive strategy engine:

[0240] Feedback=f adjust (Scores,E anxiety ,Strategy ADIS )

[0241] The specific implementation is to adjust the score threshold to weaken or strengthen the feedback:

[0242] if Strategy=="Gentle_Correction":

[0243] Threshold_display=Threshold_base+α*E_anxiety

[0244] Intensity_negative=Intensity_base*(1-β*E_anxiety)

[0245] Intensity_positive=Intensity_base*(1+γ*E_confidence)

[0246] The evaluation results are filtered through the following functions to ensure that they meet the sentiment strategy requirements:

[0247] Feedback final =Filter(Raw feedback ,Strategy ADIS )

[0248] The filter function is implemented as a neural network classifier to perform emotional style conversion on the original feedback.

[0249] As a further description of the above technical solution:

[0250] The emotionally consistent natural language generation system and the virtual conversation partner interaction interface module provide a friendly and patient AI virtual partner as an interaction interface. Its core is the emotionally consistent natural language generation (EC-NLG) system. The architecture of the EC-NLG system is as follows:

[0251] The encoder-decoder architecture uses a Transformer-based encoder-decoder architecture with enhanced emotion control:

[0252] H enc =Encoder(X dialogue ,S emotion )

[0253]

[0254] Among them, X dialogue is the history of dialogue, S emotion are the sentiment strategy parameters, and Y is the generated response;

[0255] The encoder uses a multi-layer Transformer structure:

[0256] h l =TransformerLayer(h l-1 )

[0257] h0=WordEmbedding(X)+PositionEmbedding(X)+EmotionEmbedding(S emotion )

[0258] The decoder also uses a multi-layer Transformer, but adds an emotion control mechanism:

[0259] s l =TransformerDecoderLayer(s l-1 ,h L )

[0260] P(y t |y <t ,H enc ,S emotion )=softmax(W·s L +b)

[0261] Emotional control mechanisms;

[0262] To achieve controllable generation of emotions, the following technical solutions are adopted:

[0263] Prefix-Tuning;

[0264] Add a learnable prefix vector before the self-attention layers of the encoder and decoder:

[0265]

[0266] in is the sentiment-specific prefix parameter, |P| is the prefix length;.

[0267] These prefix parameters are generated by the sentiment mapping network:

[0268]

[0269] Conditional training, adding emotion labels to the training data, the training objectives are:

[0270]

[0271] Training data construction method, extracted from language learning dialogue data (X i ,Y i ) and mark the emotional label S emotion,i ;

[0272] Emotion intensity control, adjust the intensity of emotional expression by inserting special tags:

[0273] X augmented =Insert(X,"[EMOTION="+e+",INTENSITY="+i+"]")

[0274] Where e is the emotion type, such as encouragement and comfort, and i is the intensity value 1-5;

[0275] Decoding stage control, when decoding, the weighted context integration (WACI) method is used:

[0276]

[0277] where α t is the dynamically calculated weight, h emotion is the sentiment representation vector;

[0278] Weight calculation function:

[0279] α t =σ(w T [h t ;h emotion ]+b)

[0280] Response Diversity Enhancement,To increase the diversity of generated responses, we use the following,techniques:

[0281] Nucleus Sampling (Top-p) does not use greedy decoding, but instead samples from a subset of the vocabulary whose cumulative probability reaches a threshold p:

[0282]

[0283]

[0284] Diversity promotes loss, adding the maximum mutual information (MMI) objective during training:

[0285]

[0286] Response template library, maintains an emotion-specific response template library, containing templates for different emotion categories and intensities:

[0287]

[0288] where t i is the template text, e i is the emotion category, s i It’s the emotional intensity;

[0289] According to the emotion control requirements, the system retrieves the appropriate template from the template library and dynamically fills it:

[0290] R=Fill(t selected ,X context )

[0291] The template selection function uses cosine similarity matching:

[0292]

[0293] Sentiment consistency check: The generated candidate responses are evaluated by the sentiment consistency scorer to ensure that they are consistent with the target sentiment:

[0294] Score emotion (R)=Classifier emotion (R)

[0295] If the consistency score falls below a threshold, the system regenerates or adjusts the response:

[0296] while Score_emotion(R)<θ_consistency:

[0297] R=RegenerateResponse(X,S_emotion,previous_R).

[0298] As a further description of the above technical solution:

[0299] The emotional expression speech synthesis system uses advanced text-to-speech (TTS) technology to achieve emotion-controllable speech synthesis. The architecture includes:

[0300] The core TTS architecture adopts a two-stage TTS architecture, including an acoustic model and a vocoder:

[0301] FastSpeech 2 acoustic model, based on non-autoregressive Transformer, realizes the mapping from text to mel spectrogram:

[0302] D=PhonemeExtractor(text)

[0303] H D =PhonemeEncoder(D)

[0304] H L =H D +VarianceAdaptor(H D ,p,e,d)

[0305] M=MelDecoder(H L )

[0306] Among them, p is pitch information, e is energy information, d is duration information, and M is Mel spectrogram;

[0307] The training objectives of the FastSpeech 2 acoustic model are:

[0308]

[0309] HiFi-GAN vocoder, converts mel-spectrograms into waveforms, using a generative adversarial network architecture:

[0310]

[0311] The training objective combines multiple loss functions:

[0312]

[0313]

[0314] in It's about fighting losses. is the feature matching loss, is the Mel-spectrogram reconstruction loss;

[0315] Emotional control mechanism. To achieve emotion-controllable speech synthesis, the system uses the following technologies:

[0316] Global sentiment embedding, adding sentiment embedding to the acoustic model:

[0317] H D '=H D +EmotionEmbedding(e emotion )

[0318] Sentiment embedding is obtained through a lookup table:

[0319] e emotion =LookupTable(emotion_id)

[0320] Or generated by continuous parameters:

[0321] e emotion =f emotion (valence, arousal, dominance)

[0322] f emotion (v,a,d)=W e ·[v,a,d]+b e

[0323] Fine-grained prosody control, controlling the acoustic parameters of speech to express emotions, pitch adjustment: anxiety: high pitch; encouragement: large fluctuations;

[0324] p'=p·(1+α pitch ·f pitch (emotion))

[0325] Adjust the speaking speed to reduce anxiety: soothing attitude:

[0326] d'=d·(1+α duration ·f duration (emotion))

[0327] Energy adjustment, encouragement, and energy enhancement:

[0328] e'=e·(1+α energy ·f energy (emotion))

[0329] Reference encoder, extracting style features from emotional speech reference:

[0330] e ref =ReferenceEncoder(x ref )

[0331] H D ″=H D +StyleAdapter(e ref )

[0332] Prosody model,To enhance the naturalness of emotional speech, we establish a prosody prediction model:

[0333] p,e,d=VariancePredictor(H D ,emotion)

[0334] The rhythm prediction model consists of three sub-networks, each of which has the following architecture:

[0335] Conv1D(kernel_size=3,filters=256)->ReLU->LayerNorm->

[0336] Conv1D(kernel_size=3,filters=256)->ReLU->LayerNorm->

[0337] Conv1D(kernel_size=3,filters=1)

[0338] The relationship between prosodic features and emotions is realized through conditional input:

[0339] h in =[h phoneme ;h emotion ]

[0340] The statistical characteristics of each prosodic parameter are also adjusted according to the emotion type:

[0341] μ p (e),σ p (e),μ d (e),σ d (e),μ e (e),σ e (e).

[0342] As a further description of the above technical solution:

[0343] The virtual image expression and action generation system provides expression and action generation functions for scenarios where a 3D virtual human image is used. The system architecture includes:

[0344] Facial expression generation, BlendShape parameter control, based on the standard facial BlendShape system, control the expression of the virtual image:

[0345] BS={bs1,bs2,...,bs K}

[0346] Each bs i ∈[0,1] controls an expression parameter, such as raising eyebrows and raising the corners of the mouth;

[0347] Emotion to BlendShape mapping function:

[0348] BS=f emotion2bs (emotion,intensity)

[0349] Implemented as a neural network:

[0350] BS=σ(W bs ·[e emotion ;intensity]+b bs )

[0351] Dynamic expression control to generate coherent expression changes:

[0352] BS t =α·BS target +(1-α)·BS t-1

[0353] α=min(1,Δt / τ)

[0354] Where τ is the expression transition time constant, which depends on the rapidity of the expression change;

[0355] Emotional empathy expression: Generate empathic responses based on the user's recognized expressions:

[0356] BS empathy =f empathy (BS user ,emotion strategy )

[0357] When users express anxiety, the AI ​​displays an expression of understanding and concern;

[0358] Body action generation, based on a rule-based action library, with a series of preset actions related to emotions and dialogue functions:

[0359]

[0360] where a i is the action sequence, e i is the associated emotion, f i It is a conversational function, such as emphasis and questioning;

[0361] Action selection function:

[0362]

[0363] Based on sequence-to-sequence action generation, train sequence-to-sequence models to generate action sequences from text and emotions:

[0364] A=Seq2SeqModel(Text,Emotion)

[0365] The specific implementation is a Transformer model, and the output is a sequence of action parameters;

[0366] Action fusion and transition, smooth fusion of multiple actions:

[0367]

[0368]

[0369] where s i is the action selection score;

[0370] Audio and video synchronization, ensuring synchronization of voice and animation:

[0371] P viseme,t =f viseme (phoneme t-δ )

[0372] Among them, P viseme is the lip shape parameter, δ is the time offset, taking into account the processing delay;

[0373] Lip sync uses a phoneme-based mapping table:

[0374]

[0375] The user interface adopts a simple and friendly design and provides the following functions:

[0376] Interface layout optimization, using attention heat map analysis to optimize the layout of interface elements:

[0377]

[0378] Importance scoring function of interface elements:

[0379] f importance (element)=w1·frequency(element)+w2·criticality(element)

[0380] Personalized customization: users can customize multiple aspects of their AI companions:

[0381] Visual image, choose a different avatar or adjust the existing one;

[0382] Sound characteristics, select different sounds or adjust timbre parameters;

[0383] Interaction style, set the preferred interaction method and rhythm;

[0384] Personalization parameters are stored in the User model:

[0385] U preference ={voice,avatar,style,feedback_level,...}

[0386] Progress tracking and incentive system;

[0387] Visualization of learning progress:

[0388]

[0389] Design an achievement system, including: fixed achievements based on milestones, such as completing 10 conversations;

[0390] Achievements based on skill improvement, such as a 20% increase in fluency;

[0391] Consistency-based achievements, such as studying for 7 consecutive days;

[0392] The system uses an achievement unlocking algorithm based on a directed acyclic graph (DAG) to achieve more flexible achievement dependency processing:

[0393] mermaidgraph TD A[achievement network DAG]-->B[topological sorting]B-->C[user status evaluation]C-->D[condition scoring]D-->E[threshold determination]E-->F[cascade unlocking]F-->G[achievement update];

[0394] Progress data is regularly synchronized to the backend and used for model optimization:

[0395] U progress,t =f update (U progress,t-1 ,Performance t )

[0396] Adaptive interface elements dynamically adjust according to the learner's emotional state:

[0397] Anxiety state, simplify the interface and reduce distracting elements;

[0398] Confidence state, showing more challenge options and advanced content;

[0399] Interface element visibility control function:

[0400] Visibility(element)=f visibility (element,E user )

[0401] Overall interface complexity adjustment function:

[0402] Complexity UI =C base ·(1-α·E anxiety ).

[0403] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0404] 1. In this invention, by perceiving the learner's emotional state in real time and dynamically adjusting the interaction strategy, it can effectively alleviate language speaking anxiety and enhance oral confidence. The system adopts multimodal emotion computing technology to improve the accuracy and robustness of emotion recognition, and designs targeted interaction strategies for different emotional states, providing learners with personalized emotional support and language practice.

[0405] 2. In the present invention, real-time perception and dynamic adjustment are implemented. The system can perceive the learner's emotional state in real time and dynamically adjust the interaction strategy according to the emotional changes. This real-time and dynamic nature enables the system to more accurately meet the learner's emotional needs and provide a more personalized learning experience. When the learner shows anxiety, the system will immediately adjust the content and method of the conversation and use more gentle and encouraging language to help the learner relieve anxiety and improve learning efficiency and interest.

[0406] 3. In the present invention, language speaking anxiety is alleviated. By sensing the learner's emotional state in real time, the system can promptly identify and alleviate the learner's language speaking anxiety. When the learner shows tension or anxiety, the system will adopt active empathy and encouragement, dynamic difficulty adjustment, and guided relaxation intervention strategies to help the learner relax and reduce anxiety levels. The system will provide some simple oral exercises to help learners build confidence and gradually overcome anxiety.

[0407] 4. In the present invention, oral confidence is enhanced. Through personalized emotional support and language practice, the system can effectively enhance learners' oral confidence. The system will provide positive feedback and encouragement in a timely manner according to the learners' emotional state and oral performance, helping learners build self-confidence and thus participate in language learning more actively. When learners answer questions correctly, the system will give praise and encouragement in a timely manner to help learners build self-confidence and stimulate learning motivation.

[0408] 5. In the present invention, multimodal emotion computing technology adopts multimodal emotion computing technology to improve the accuracy and robustness of emotion recognition by fusing multi-source data such as speech, text, facial expressions and physiological signals. This multimodal fusion strategy can overcome the defect of a single modality being susceptible to interference and provide more comprehensive and accurate emotion recognition results. By fusing speech and facial expression data, the system can more accurately judge the learner's emotional state, thereby providing more personalized emotional support and language practice.

[0409] 6. In the present invention, targeted interactive strategies are designed for different emotional states, which can better meet the learners' personalized needs. When the learners show anxiety, the system will adopt anxiety relief strategies, active empathy and encouragement, dynamic difficulty adjustment, and guided relaxation intervention. When the learners show lack of self-confidence, the system will adopt self-confidence building strategies, such as progressive challenges and achievement reinforcement. When the learners show anxiety, the system will provide some simple oral exercises to help learners build confidence and gradually overcome anxiety.

[0410] 7. In the present invention, personalized emotional support and language practice are provided to learners, which can better meet the needs of different learners. The system will dynamically adjust the content and method of the conversation according to the learner's emotional state and oral performance to provide a more personalized learning experience. When the learner shows anxiety, the system will immediately adjust the content and method of the conversation and use more gentle and encouraging language to help learners relieve anxiety and improve learning efficiency and interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0411] Figure 1 A schematic diagram of the overall system control flow of the adaptive interactive language learning system enabled by multimodal affective computing and its supporting method proposed in the present invention;

[0412] Figure 2 A schematic diagram of the architecture of the hybrid model in the adaptive interactive language learning system enabled by multimodal affective computing and its supporting method proposed in the present invention;

[0413] Figure 3 A schematic diagram of the pre-trained language model architecture in the adaptive interactive language learning system enabled by multimodal affective computing and its supporting method proposed in the present invention;

[0414] Figure 4 This is a schematic diagram of the architecture of the dialogue management unit in the adaptive interactive language learning system enabled by multimodal affective computing and its supporting method proposed in the present invention;

[0415] Figure 5 This is an architectural diagram of the adaptive interactive language learning system and its supporting method enabled by multimodal affective computing proposed in the present invention;

[0416] Figure 6 This is a schematic diagram of the architecture of the EC-NLG system in the adaptive interactive language learning system enabled by multimodal affective computing and its supporting method proposed in the present invention;

[0417] Figure 7A schematic diagram of the architecture of the emotion expression speech synthesis system in the adaptive interactive language learning system enabled by multimodal emotion computing and its supporting method proposed in the present invention;

[0418] Figure 8 This is a schematic diagram of the architecture of the virtual image expression and action generation system in the adaptive interactive language learning system enabled by multimodal affective computing and its supporting method proposed in the present invention;

[0419] Figure 9 A schematic diagram of the initialization process during system startup in the adaptive interactive language learning system and its supporting method enabled by multimodal affective computing proposed in the present invention;

[0420] Figure 10 A schematic diagram of the strategy execution and monitoring process in the adaptive interactive language learning system enabled by multimodal affective computing and its supporting method proposed in the present invention;

[0421] Figure 11 A schematic diagram of the dialogue management and content generation process in the adaptive interactive language learning system enabled by multimodal affective computing and its supporting method proposed in the present invention;

[0422] Figure 12 A schematic diagram of the performance control flow of a virtual partner in the adaptive interactive language learning system and its supporting method enabled by multimodal affective computing proposed in the present invention;

[0423] Figure 13 A schematic diagram of the process of continuously collecting user feedback in the adaptive interactive language learning system enabled by multimodal affective computing and its supporting method proposed in the present invention;

[0424] Figure 14 Schematic diagram of the system performance monitoring process in the adaptive interactive language learning system enabled by multimodal emotion computing and its supporting method proposed in the present invention. DETAILED DESCRIPTION

[0425] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0426] Please see the attached Figure 1 -Attached Figure 14 As shown:

[0427] learner initiation and module selection;

[0428] Learners start the language learning system through the system interface. The system provides a variety of language learning modules for learners to choose from, such as basic conversation, business communication, and travel terms, to meet the needs of different learners. Learners choose the appropriate module to study based on their interests, goals, and time arrangements. The system also recommends the most suitable module based on the learner's historical data and preferences to improve learning efficiency and interest.

[0429] The system's interface is simple and intuitive, allowing learners to quickly start their learning process. It provides a clear classification of learning modules, including basic conversation, business communication, and travel terms, to meet learners' different scenarios and needs. Each module includes a series of pre-designed learning activities and practice tasks, such as role-playing, conversation simulations, and pronunciation exercises, to help learners comprehensively improve their speaking skills.

[0430] To help learners choose the right modules, the system makes intelligent recommendations based on their historical data and preferences. It analyzes their learning history, practice performance, and engagement, and combines this with their personal goals and schedule to recommend the most appropriate modules. For example, for learners hoping to improve their business English speaking skills in the short term, the system recommends the Business Communication module, while for learners hoping to use English more confidently while traveling, the system recommends the Travel Terminology module.

[0431] The system will also dynamically adjust the difficulty and content of modules based on learners' learning progress and performance. For example, when a learner demonstrates a high level of mastery in a module, the system will automatically recommend a more difficult module to help the learner further improve their oral English skills. At the same time, the system will also adjust the module content and practice methods in a timely manner based on learners' feedback and needs to maintain learners' interest and motivation.

[0432] Through this personalized module selection and learning path recommendation, the system can better meet the needs of different learners, improve learning efficiency and interest. Learners can choose appropriate modules to study based on their actual situation and learning goals, and gradually improve their oral English skills under the guidance of the system.

[0433] Data collection and privacy protection;

[0434] After the system is started, it prompts learners to start conversation practice and uses microphones, cameras, and wearable devices to collect learners' voice, text conversion through voice recognition technology, facial expressions, and physiological signal data in real time. During the collection process, the system ensures data security and protects personal privacy. The system encrypts the collected data and strictly abides by data protection regulations to ensure that learners' privacy is not violated.

[0435] For speech processing, the system uses advanced end-to-end models such as Conformer or Whisper (OpenAI) for speech-to-text conversion. For spoken language scenarios, the model must be fine-tuned on a spoken language dataset containing a large number of non-fluent speech, pauses, and repetitions. At the same time, acoustic features of the speech are extracted, such as the eGeMAPS (extended Geneva Minimalistic Acoustic Parameter Set) or ComparE (Computational Paralinguistics Challenge) feature set, which covers pitch, intensity, duration, MFCC, formant, jitter, and shimmer. Feature extraction is performed using the OpenSMILE toolkit.

[0436] In terms of text processing, the system cleans the ASR output text (such as removing catchphrases and correcting obvious recognition errors), performs word segmentation, part-of-speech tagging, and syntactic analysis. These processing steps help improve the accuracy and reliability of subsequent sentiment analysis.

[0437] In terms of facial expression processing (optional, user authorization required), the system uses MTCNN or RetinaFace to detect faces and locate key points (such as 68 or more key points). Based on facial key points, OpenFace or other tools are used to extract the intensity and existence of AUs. These AUs are highly correlated with specific emotions (especially micro-expressions). For video streams, the system extracts the dynamic features of AUs changes and head posture changes between consecutive frames, and uses the graph attention network (GAT) to model the topological relationship between facial key points or AUs. The system uses 3D-CNN or CNN+Transformer structures to process the AUs sequence of consecutive frames or the original facial area, capture the dynamic expression, and output the visual embedding vector E. visual ;

[0438] In terms of physiological signal processing (optional, user authorization required), the system obtains heart rate (HR), heart rate variability (HRV), and skin conductivity (GSR / EDA) through compatible smart bracelet / watch APIs (such as Fitbit, Garmin), and uses a temporal convolutional network (TCN) or LSTM to encode the HRV and GSR feature sequences and output the physiological embedding vector E physio , HRV: time domain features (SDNN, RMSSD), frequency domain features (LF, HF, LF / HFratio), GSR: skin conductance level (SCL), skin conductance response (SCR) number, amplitude and rise time, filter and denoise the physiological signal, and align the time stamp with the voice / face data;

[0439] When collecting and processing this data, the system strictly abides by data protection regulations to ensure that learners' privacy is not violated. The system will encrypt the collected data and restrict access rights to the data to prevent data leakage and abuse. At the same time, the system will also conduct regular data security audits to ensure data security and compliance;

[0440] Through these data collection and processing measures, the system can perceive the learner's emotional state in real time and provide personalized emotional support and language practice based on this, effectively alleviating language speaking anxiety and improving learners' speaking confidence, thereby improving language learning outcomes.

[0441] Real-time perception and quantification of multimodal emotions;

[0442] The multimodal emotion real-time perception and quantification module receives the collected data and processes the information in parallel. The speech emotion analysis unit extracts the acoustic features of the speech, such as speaking rate, pitch, and intensity, and uses a deep learning model to classify and quantify emotions. The text emotion analysis unit performs natural language processing on the content after speech-to-text conversion, analyzes the vocabulary and syntactic structure to assist in judging emotions. The facial expression emotion analysis unit analyzes facial micro-expressions and identifies the corresponding emotional state through facial key point detection and expression recognition algorithms. The physiological signal emotion analysis unit receives and analyzes physiological signal data, such as heart rate variability (HRV) and skin conductivity (GSR), as an auxiliary basis for emotion judgment. Finally, the multimodal emotion fusion and decision-making unit integrates the emotion analysis results of each unit and outputs the current emotion state vector. This process utilizes a variety of advanced technologies, such as deep learning, natural language processing, and computer vision, to achieve a comprehensive and accurate judgment of the learner's emotional state.

[0443] For speech processing, the system uses advanced end-to-end models such as Conformer or Whisper (OpenAI) for speech-to-text conversion. For spoken language scenarios, the model must be fine-tuned on a spoken language dataset containing a large number of non-fluent speech, pauses, and repetitions. At the same time, acoustic features of the speech are extracted, such as the eGeMAPS (extended Geneva Minimalistic Acoustic Parameter Set) or ComparE (Computational Paralinguistics Challenge) feature set, which covers pitch, intensity, duration, MFCC, formant, jitter, and shimmer. Feature extraction is performed using the OpenSMILE toolkit.

[0444] In terms of text processing, the system cleans the ASR output text (such as removing catchphrases and correcting obvious recognition errors), performs word segmentation, part-of-speech tagging, and syntactic analysis. These processing steps help improve the accuracy and reliability of subsequent sentiment analysis.

[0445] In terms of facial expression processing (optional, user authorization required), the system uses MTCNN or RetinaFace to detect faces and locate key points (such as 68 or more key points). Based on facial key points, OpenFace or other tools are used to extract the intensity and existence of AUs. These AUs are highly correlated with specific emotions (especially micro-expressions). For video streams, the system extracts the dynamic features of AUs changes and head posture changes between consecutive frames, and uses the graph attention network (GAT) to model the topological relationship between facial key points or AUs. The system uses 3D-CNN or CNN+Transformer structures to process the AUs sequence of consecutive frames or the original facial area, capture the dynamic expression, and output the visual embedding vector E. visual ;

[0446] In terms of physiological signal processing (optional, user authorization required), the system obtains heart rate (HR), heart rate variability (HRV), and skin conductivity (GSR / EDA) through compatible smart bracelet / watch APIs (such as Fitbit, Garmin), and uses a temporal convolutional network (TCN) or LSTM to encode the HRV and GSR feature sequences and output the physiological embedding vector E physio , HRV: time domain features (SDNN, RMSSD), frequency domain features (LF, HF, LF / HFratio), GSR: skin conductance level (SCL), skin conductance response (SCR) number, amplitude and rise time, filter and denoise the physiological signal, and align the time stamp with the voice / face data;

[0447] When collecting and processing this data, the system strictly abides by data protection regulations to ensure that learners' privacy is not violated. The system will encrypt the collected data and restrict access rights to the data to prevent data leakage and abuse. At the same time, the system will also conduct regular data security audits to ensure data security and compliance;

[0448] Through these data collection and processing measures, the system can perceive the learner's emotional state in real time and provide personalized emotional support and language practice based on this, effectively alleviating language speaking anxiety and improving learners' speaking confidence, thereby improving language learning outcomes.

[0449] Emotional Adaptive Strategy Engine;

[0450] The emotion adaptation strategy engine receives the emotion state vector from the Multimodal Hierarchical Attention Emotion Recognition Engine (MHAE-Net) and selects and activates the most appropriate interaction strategy from the strategy library based on preset rules and learner historical data. For example, when a learner shows anxiety, the system will adopt the following strategy:

[0451] Positive reassurance and encouragement: The system generates positive feedback and encouraging messages, such as "I noticed you were a little nervous, that's normal, let's take it slow," or "This section was a bit difficult, but you did a great job!" These messages are designed to help learners relieve anxiety and increase their motivation through empathy and encouragement.

[0452] Task difficulty adjustment: The system dynamically adjusts the complexity of conversation tasks based on the learner's emotional state and oral performance. For example, when a learner expresses anxiety, the system switches to simpler or more familiar topics, reduces the use of long and difficult sentences, and provides clearer prompts or sentence scaffolding to reduce learner stress and help them better adapt to the learning environment.

[0453] Relaxation guidance: The system will insert guiding words or instructions at appropriate times to prompt users to perform brief relaxation exercises, such as "Take a deep breath, relax your shoulders, we are not in a hurry," or play soothing background sounds (subject to user consent) to help learners relieve anxiety and improve learning outcomes;

[0454] Gentle feedback: When correcting pronunciation or grammar errors, the system uses gentle, encouraging, and constructive language, emphasizing learners' progress rather than their mistakes, and avoiding direct negative feedback that can cause frustration. For example, the system might replace "You're wrong" with "Try saying it this way, wouldn't it sound more authentic?" to help learners build confidence and increase their motivation.

[0455] These strategies are designed to help learners relieve anxiety and improve learning efficiency and interest. By continuously learning and optimizing these strategies, the system can better adapt to the needs of different learners and provide a more personalized learning experience.

[0456] conversation management and content generation;

[0457] The core responsibility of the dialogue management unit is to understand the learner's intention and generate appropriate dialogue content and feedback based on the current dialogue context and the selected emotional interaction strategy. This unit uses natural language processing technology to ensure the coherence and naturalness of the dialogue, thereby providing learners with an efficient and targeted language learning experience.

[0458] To achieve this goal, the dialogue management unit first uses natural language understanding (NLU) technology to parse the learner's input. The NLU unit includes intent recognition and slot filling to identify the learner's intent and key information. For example, when a learner asks about the meaning of a word, the NLU unit recognizes this as a "query for word meaning" intent and extracts "word" as the key slot. This step is key to ensuring that the AI ​​can accurately understand the learner's intent.

[0459] After understanding the learner's intention, the dialogue management unit combines the macro-strategy and low-level strategies output by the emotional adaptive strategy engine to generate appropriate dialogue content. For example, if the learner expresses anxiety and the high-level strategy recommends "active comfort and encouragement," the low-level strategy may suggest using simpler vocabulary and sentence patterns. In this case, the dialogue management unit will generate a response containing comforting and encouraging words, while using vocabulary and sentence patterns that are easier for the learner to understand.

[0460] In addition, the dialogue management unit also considers the conversation history and context to ensure the coherence of the conversation. For example, if the learner mentioned a certain topic in the previous round of conversation, the dialogue management unit will take this topic into account when generating a response to avoid situations where the learner's previous content is irrelevant.

[0461] To enhance the naturalness of conversations, the dialogue management unit also leverages natural language generation (NLG) technology. The NLG unit uses pre-trained language models, such as the GPT series, BART, or T5, and combines emotional goals (such as comfort, encouragement, and gentleness) with strategic parameters (such as using simpler vocabulary) to generate natural and fluent responses. For example, when a learner asks about the meaning of a word, the NLG unit generates a response that includes word explanations, examples, and usage, using vocabulary and sentence structures that are easier for learners to understand.

[0462] Through the above technical means, the dialogue management unit can generate accurate, coherent, natural dialogue content that matches the learner's emotional state, thereby providing learners with an efficient and targeted language learning experience;

[0463] interactive feedback from virtual conversation partners;

[0464] The AI ​​virtual companion adjusts its language content, tone, expressions, and movements based on the output of the emotional adaptation strategy engine to better match the learner's emotional state, enhancing the learner's emotional experience and empathy. For example, when a learner is anxious, the virtual companion will display a gentler, more patient expression and gesture to help the learner relax and reduce anxiety levels.

[0465] To achieve this goal, the system uses the following technical means:

[0466] ExpressiveTTS (Text-to-Speech): * Model: Adopts advanced neural TTS models such as VITS, FastSpeech2, Glow-TTS, Tacotron2+WaveGlow / HiFi-GAN, * Emotion Control: Uses the emotional target output by ADIS-RL (such as comforting tone, encouraging intonation) as the conditional input of the TTS model to generate speech with corresponding emotional prosody, which is achieved through global emotion embedding or fine-grained prosody control (such as controlling pitch, speaking rate, and pauses);

[0467] Avatar (3D Avatar - optional): * Expression synchronization: Mapping user expressions recognized by MHAE-Net (for empathic feedback, such as AI expressions showing understanding) and AI emotional expressions specified by ADIS-RL (such as AI smile encouragement) to the facial animation parameters of the avatar (such as BlendShapes), * Action generation: Generate natural head posture and gestures based on the conversation content and emotional state;

[0468] Interface design: Simple and user-friendly, providing clear guidance and feedback, allowing users to customize the appearance and voice of their AI companions, and providing progress tracking and achievement display incentives;

[0469] Through these technologies, the system can provide learners with a highly realistic and natural virtual conversation partner, significantly increasing learners' willingness to practice, reducing anxiety levels, and ultimately improving their oral fluency and confidence.

[0470] System monitoring and performance optimization;

[0471] The system continuously monitors learners' feedback and performance, optimizing emotion recognition models and adaptive interaction strategies. Learners' interaction data, including emotional changes, conversation content, system strategies, and subsequent results, is anonymized and used for model training and strategy adjustments to improve system accuracy and effectiveness. This process utilizes machine learning and data mining technologies to continuously optimize and enhance system performance.

[0472] In terms of data acquisition and continuous learning framework, the system adopts the following strategies:

[0473] Initial data: * Emotional annotation data: Collect or purchase a multimodal spoken emotion database. For FLSA scenarios, you need to record and annotate the data yourself (such as student performance in mock interviews and keynote speeches). Annotators need to receive psychology training. * Conversational data: Collect English teaching conversations and psychological counseling conversations.

[0474] Online data collection and labeling: * Collect anonymized interaction data between users and the system, with user consent and strict compliance with privacy agreements. * For RL, it is necessary to define clear sources of reward signals, including implicit user feedback (such as continued conversation, task completion rate, and reduced anxiety indicators) and explicit user feedback (such as likes and ratings). * For sentiment models, adopt semi-supervised or active learning strategies, manually label some uncertain or valuable unlabeled data, and continuously optimize the model.

[0475] Continuous learning and model updates: * Regularly incrementally train or retrain the MHAE-Net, ADIS-RL, and EC-NLG modules using newly collected and annotated data. * Adopt online learning or federated learning (if multi-client deployment and privacy are considered) to enable the model to adapt to changes in individual users or groups of users. * A / B test different model versions and strategies to iteratively optimize system performance.

[0476] Through these strategies, the system can continuously learn from practical applications, optimize the accuracy of emotion recognition models and the effectiveness of adaptive interaction strategies, and thus provide learners with a more accurate and personalized language learning experience.

[0477] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can replace or change the technical solution and inventive concept of the present invention within the technical scope disclosed by the present invention, and the replacement or change should be covered by the scope of protection of the present invention.

Claims

1. An adaptive interactive language learning system enabled by multimodal affective computing, characterized by: It includes the multimodal emotion real-time perception and quantification module MHAE-Net, the personalized adaptive dialogue and intervention strategy module ADIS-RL, and the virtual dialogue partner interaction interface module EC-NLG; The multimodal emotion real-time perception and quantification module MHAE-Net consists of a speech signal acquisition and processing unit, a speech emotion analysis unit, a text emotion analysis unit, a facial expression emotion analysis unit, a physiological signal emotion analysis unit, and a multimodal emotion fusion and decision-making unit; The personalized adaptive dialogue and intervention strategy module ADIS-RL consists of a dialogue management unit, an emotion adaptation strategy engine, and a pronunciation and fluency assessment unit; The virtual conversation partner interaction interface module EC-NLG consists of an emotion consistency natural language generation system, an emotion expression speech synthesis system, an avatar expression and action generation system, and user interface design and personalization.

2. The adaptive interactive language learning system enabled by multimodal affective computing according to claim 1, characterized in that: In the speech signal acquisition and processing unit, the system collects learners' speech signals in real time and converts speech to text using advanced end-to-end models, such as Conformer or Whisper OpenAI. For spoken language scenarios, the model needs to be fine-tuned on a spoken language dataset containing a large amount of non-fluency, pauses, and repetitions. Its objective function can be expressed as: Among them, x i Represents the characteristics of the speech signal, y i Represents the corresponding text label, θ represents the model parameter, and N represents the number of samples; At the same time, the system extracts the acoustic features of the speech, including the following categories: Time domain features, energy mean, zero crossing rate (ZCR), and voice segment ratio; Frequency domain features, MFCC, formant frequency and bandwidth; Sound quality characteristics, fundamental frequency jitter, amplitude shimmer, harmonic-to-noise ratio (HNR); Prosodic features, pitch contour, speaking rate, pause distribution and intensity variation; These features are extracted using the OpenSMILE toolkit in the following steps: Audio is framed, with a frame length of 25ms and a frame shift of 10ms; Apply a Hamming window to each frame Calculate various acoustic features, such as MFCC: Where S(j) is the output of the Mel filter bank and M is the number of filters; Finally, an 88-dimensional feature vector based on the eGeMAPS standard is formed; The speech emotion analysis unit uses a deep learning model to analyze the extracted acoustic features and designs a hybrid model based on the combination of Transformer encoder and BiLSTM; The unit consists of the following core components: Multi-head self-attention mechanism to capture long-range dependencies in feature sequences; MultiHead(Q,K,V)=Concat(head1,...,head h )W O in BiLSTM layer, further captures the temporal information of features; Multi-task output layer, which performs emotion classification and emotion dimension value regression simultaneously; P(y class |x)=softmax(W class h+b class ) y dimensions =σ(W dim h+b dim ) The model training adopts a multi-task learning framework, and the loss function is: in is the cross entropy loss for classification tasks, is the mean square error loss of the regression task, α is the weight parameter to balance the two tasks, is the L2 regularization term.

3. The adaptive interactive language learning system enabled by multimodal affective computing according to claim 2, characterized in that: The text sentiment analysis unit performs a series of processing on the ASR converted text: Text cleaning, removing catchphrases, correcting ASR errors, and applying regular expression filtering; Token segmentation and standardization, using NLTK or spaCy for token segmentation; Feature extraction, using pre-trained BERT, RoBERTa, or DeBERTa models to extract text representations; The mathematical expression of the model is: h CLS =TransformerEncoder(Embedding(TokenIDs)) CLS y emotion =Classifier(Dropout(h CLS )) In order to improve the accuracy of text sentiment analysis, we integrate sentiment dictionary and syntactic structure analysis to build a sentiment feature enhancer: where f lex (T) is the feature based on the sentiment dictionary, f syn (T) is based on the characteristics of syntactic structure, Represents feature concatenation operation; The facial expression emotion analysis unit, If the user authorizes the use of the camera, the system captures the learner's facial image in real time and processes it according to the following steps: Face detection and alignment: Use the MTCNN or RetinaFace algorithm to detect faces and extract 68 key points: Among them B i are the bounding box coordinates, S i is the detection confidence, L i is the set of key point coordinates; Facial action unit AUs extraction, based on facial key points, uses OpenFace tools to extract AUs intensity and existence: It contains 17 main AUs, each of which has an intensity value (0-5) and existence (0 / 1); Graph Attention Network modeling, using GAT to model the topological relationship between facial key points or AUs: Temporal modeling, using 3D-CNN or Transformer to process continuous frame sequences; For 3D-CNN: F t+1 =Conv3D(F t-k：t ); For Transformer, {F1',...,F T '}=TransformerEncoder({F1,...,F T }); Emotional representation extraction, and finally output visual embedding vector E visual .

4. The adaptive interactive language learning system enabled by multimodal affective computing according to claim 3, characterized in that: In the physiological signal emotion analysis unit, if the user wears a compatible wearable device, the system processes the following physiological signals: Heart rate variability (HRV) analysis: Time domain indicators, SDNN standard deviation, RMSSD root mean square of the difference between adjacent RR intervals; Frequency domain indicators, using fast Fourier transform (FFT) to obtain LF0.04-0.15Hz, HF0.15-0.4Hz power and LF / HF ratio; Skin conductivity GSR / EDA analysis; Skin conductance level SCL, low-frequency baseline changes, reflects the overall level of arousal; Skin conductance response (SCR), peak feature extraction, including number, amplitude and rise time; SCR amplitude =max(GSR peak )-GSR onse t SCR risetime =t peak -t onset Signal processing: Low-pass filtering removes high-frequency noise, Missing value interpolation, based on spline interpolation or forward filling; Timestamp alignment, aligning physiological signals of different sampling rates with frame-level speech / visual features; Temporal coding, using temporal convolutional networks (TCN) or LSTM networks to encode physiological signal sequences: h t =LSTM(x t ,h t-1 ) or Where d is the dilation factor, and the final output is the physiological embedding vector E physio ; The multimodal emotion fusion and decision-making unit fuses the emotion analysis results of each modality and adopts the following strategy: Cross-modal Attention Fusion: Among them E i represents the embedding vector of modality i, and f is the similarity measurement function; Gated multimodal fusion: g i =σ(W g [h1,h2,...,h M ]+b g ) where g i is the gating weight of modality i, ⊙ denotes element-wise multiplication; Tensor fusion, capturing high-order interactions through low-rank multimodal decomposition: Where ×1 represents the modulo-1 product of the tensor, represents the outer product operation; Through the above fusion strategy, the system outputs the standardized emotional state vector E emotion ,Include: Emotion classification results and confidence Emotional dimension values: anxiety a∈[0,1], confidence c∈[0,1]; The selection of fusion strategy is based on a dynamic weighting algorithm, which is adaptively adjusted according to the reliability and consistency of each modality; where λ i is the reliability measure of mode i, and T is the temperature parameter.

5. The adaptive interactive language learning system enabled by multimodal affective computing according to claim 4, characterized in that: The dialogue management unit uses advanced natural language processing technology to analyze the learner's intentions and generate coherent and natural dialogue responses. Its architecture includes: Intent recognition and slot filling use the joint BERT model to simultaneously complete intent recognition and slot filling tasks: P(I,S|X)=P(I|X)·P(S|X) P(I|X)=softmax(W I ·h [CLS] +b I ) P(s j |x j )=softmax(W S ·h j +b S ) Among them, X represents the input user speech, I represents the intent category, S represents the slot sequence, and h [CLS] is the [CLS] tag representation of BERT, h j is the representation of the jth word; Recognized intent types include: Information query Information_Request; Clarification_Request; Opinion_Expression; Emotion_Expression; Help_Request; Feedback_Response; Dialogue state tracking maintains a structured dialogue state DST, which contains the following information: Current Topic T curr , Active slot set S active 、Dialogue History H t-k：t , user emotional state E user and the system's previous response R t-1 ; State update function: DST t =f update (DST t-1 ,I t ,S t ,E user,t ) Among them, f update It is a hybrid update function based on rules and learning, implemented as follows: Topic tracking, updating the current topic based on the latest intent and slot; Slot update, updating or clearing slots based on new information; S active,t =(S active,t-1 \S clear )∪S new History compression, using recursive attention networks to compress conversation history; H compressed =RecurrentAttention(H t-k：t ) The dialogue strategy decision is based on the output of the emotion adaptation strategy engine to determine the dialogue strategy: am t π(DST t ,E user,t ,g ADIS ) where g ADIS is the policy instruction of the emotion adaptive policy engine, and π is the policy function, which is implemented based on the Deep Q Network (DQN): The policy decision optimization goal is to maximize the weighted sum of user satisfaction and sentiment improvement: The knowledge enhancement mechanism enhances the conversational capabilities through the knowledge graph: KG=(E,R,T) Where E is the entity set, R is the relationship set, is a set of triples; For knowledge queries in the field of language learning, the graph attention network GAT is used to extract relevant knowledge: z e NGAT(e,N(e)Θ) where e is the query entity, N(e) is its neighbor entities, and Θ is the model parameter.

6. The adaptive interactive language learning system enabled by multimodal affective computing according to claim 5, characterized in that: The emotion adaptation strategy engine is the core innovation unit of the system. It is based on hierarchical reinforcement learning (HRL). The system architecture includes: Hierarchical reinforcement learning framework, high-level controller Meta-Controller: State space, S H ={E user ,U profile ,T context ,G learning }; E user Emotional state vector, U profile User portrait, T context Theme context and G learning Completion of learning objectives; Action space A H ={g1,g2,...,g K }, including the following macro strategies: g1 represents Soothe & Encourage, g2 represents Difficulty Adjustment, g3 represents Relaxation Guidance, g4 represents Focus Redirection, g5 represents Achievement Reinforcement, g6 represents Gentle Correction, and state transition represents S H,t+1 =f H (S H,t ,g t ,A L,t ); Reward function: R H (S H,t ,g t ,S H,t+1 )=w1ΔE anxiety +w2ΔE confidence +w3P engagement +w4F user Where ΔE anxiety is the decrease in anxiety, ΔE confidence is the confidence improvement value, P engagement is the participation measure, F user It is explicit feedback from users; Low-level controller: State space S L ={D context ,E micro ,T complexity ,g H }、D context Current conversation micro-context, E micro Fine-grained emotional fluctuations, T complexity Current task complexity, g H High-level policy instructions and action space are based on g H Determine, if g H =G2 difficulty adjustment: a1 means switching to a simpler topic, a2 means providing keyword prompts, a3 means providing sentence templates, and a4 means simplifying the question; Reward function, combining internal and external rewards: R L (S L,t ,a t ,S L,t+1 )=R internal +γI goal ·R H where R internal It is the immediate feedback of a specific action. goal is the goal completion indicator, γ is the discount factor learning algorithm The high-level controller uses the PPO algorithm: in is the advantage estimate; The low-level controller uses the SAC (Soft Actor-Critic) algorithm: in is the policy entropy, α is the temperature parameter; Strategy library implementation, anxiety relief strategy sub-library: Active empathy and encouragement, identifying anxiety triggers: E anxiety >θ anxiety Generate parameterized templates for encouraging words: T encourage (p1,p2)={t1(p1,p2),t2(p1,p2),...,t n (p1,p2)} Among them, p1 is the anxiety level parameter, and p2 is the personalized style parameter; The specific implementation is through the adaptive discourse selection function: Difficulty is dynamically adjusted, and task complexity evaluation function is: C task =f complexity (L sentence ,R grammar ,V vocabulary ) Among them L sentence is the sentence length, R grammar is the grammatical complexity, V vocabulary is the vocabulary difficulty; Dynamic adjustment algorithm: if E_anxiety>θ_high: C_target=C_current*(1-α_high) elseifE_anxiety>θ_medium: C_target=C_current*(1-α_medium else: C_target=min(C_current*(1+β),C_max) Then select the task closest to the target complexity from the task library: Guided relaxation intervention: Trigger condition: E anxiety >θ relax ∧tt last_relax >τ interval ; A progressive strategy for relaxation coaching: Level 1, simple breathing prompts ("Take a deep breath...") Level 2, guided breathing exercises (3 seconds inhale - 4 seconds hold - 5 seconds exhale) Level 3, brief mindfulness guidance (focusing on the present moment) Decision tree algorithm for selecting a strategy: Non-judgmental error correction feedback, error correction sensitivity adjustment function: S correction =S base ·(1-β·E anxiety ) Among them S base is the basic error correction sensitivity, β is the adjustment coefficient; Expression adjustment, with expression selected along a continuous spectrum from mild to direct; Style correction =lerp(Style gentle ,Style direct ,1-E anxiety ) Implemented as a parameterized error correction template: M correction (error,severity,anxiety)=template select (error_type,severity,anxiety) Confidence Building Strategy Sublibrary: Progressive challenge, implemented as an adaptive difficulty curve: C next =C current +ΔC(E confidence ,P success ) Where ΔC is the difficulty increment function: ΔC=α·E confidence +β·P success -g·E anxiety P success is the recent success rate, α, β, γ are weight coefficients; The challenge sequence is designed as a Markov decision process (MDP), and the optimal path is calculated through value iteration; the achievement reinforcement and progress detection algorithm: When Progress>θ progress Praise is triggered when the progress is made, and the intensity of the praise is proportional to the degree of progress: Intensity praise =min(1,α·Progress) A virtual reward system that accumulates points based on progress; Points t =Points t-1 +Base points ·(1+β·Progress) Achievement unlocking conditions, when points or specific abilities reach a threshold; 7. The adaptive interactive language learning system enabled by multimodal affective computing according to claim 6, characterized in that: The pronunciation and fluency assessment unit objectively evaluates learners' oral performance and provides a basis for dialogue strategies; Pronunciation assessment module, phoneme-level scoring, using acoustic models to calculate phoneme deviations: Where d is the distance metric based on MFCC, P user and P native is the feature vector of the user and standard pronunciation; Word-level scoring, combining phoneme scoring and prosodic features: Sentence-level scoring, taking into account multiple factors such as linking and intonation: Score sentence =f combine (Score words ,Rhythm,Intonation sentence ) Fluency assessment module, pause analysis, calculation of unnatural pause rate: Speech speed analysis, calculation of effective speech speed, effective words per minute: Repetition and self-correction, detection of repeated words and phrase correction: Overall Fluency Score: Where norm() is a normalization function that maps the speaking rate to the interval [0,1]; Feedback control dynamically adjusts the presentation of evaluation results based on the instructions of the emotional adaptive strategy engine: Feedback=f adjust (Scores,E anxiety ,Strategy ADIS ) The specific implementation is to adjust the score threshold to weaken or strengthen the feedback: if Strategy=="Gentle_Correction": Threshold_display=Threshold_base+α*E_anxiety Intensity_negative=Intensity_base*(1-β*E_anxiety) Intensity_positive=Intensity_base*(1+γ*E_confidence) The evaluation results are filtered through the following functions to ensure that they meet the sentiment strategy requirements: Feedback final =Filter(Raw feedback ,Strategy ADIS ) The filter function is implemented as a neural network classifier to perform emotional style conversion on the original feedback.

8. The adaptive interactive language learning system enabled by multimodal affective computing according to claim 7, characterized in that: The emotionally consistent natural language generation system and the virtual conversation partner interaction interface module provide a friendly and patient AI virtual partner as an interaction interface. Its core is the emotionally consistent natural language generation (EC-NLG) system. The architecture of the EC-NLG system is as follows: The encoder-decoder architecture uses a Transformer-based encoder-decoder architecture with enhanced emotion control: H enc =Encoder(X dialogue ,S emotion ) Among them, X dialogue is the history of dialogue, S emotion are the sentiment strategy parameters, and Y is the generated response; The encoder uses a multi-layer Transformer structure: h l =TransformerLayer(h l-1 ) h0=WordEmbedding(X)+PositionEmbedding(X)+EmotionEmbedding(S emotion ) The decoder also uses a multi-layer Transformer, but adds an emotion control mechanism: s l =TransformerDecoderLayer(s l-1 ,h L ) P(y t |y <t ,H enc ,S emotion )=softmax(W·s L +b) Emotional control mechanisms; To achieve controllable generation of emotions, the following technical solutions are adopted: Prefix-Tuning; Add a learnable prefix vector before the self-attention layers of the encoder and decoder: in is the sentiment-specific prefix parameter, |P| is the prefix length;. These prefix parameters are generated by the sentiment mapping network: Conditional training, adding emotion labels to the training data, the training objectives are: Training data construction method, extracted from language learning dialogue data (X i ,Y i ) and mark the emotional label S emotion,i ; Emotion intensity control, adjust the intensity of emotional expression by inserting special tags: X augmented =Insert(X,"[EMOTION="+e+",INTENSITY="+i+"]") Where e is the emotion type and i is the intensity value 1-5; Decoding stage control, when decoding, the weighted context integration (WACI) method is used: where α t is the dynamically calculated weight, h emotion is the sentiment representation vector; Weight calculation function: a t =σ(w T [h t ;h emotion ]+b) Response Diversity Enhancement,To increase the diversity of generated responses, we use the following,techniques: Nucleus Sampling (Top-p) does not use greedy decoding, but instead samples from a subset of the vocabulary whose cumulative probability reaches a threshold p: Diversity promotes loss, adding the maximum mutual information (MMI) objective during training: Response template library, maintains an emotion-specific response template library, containing templates for different emotion categories and intensities: where t i is the template text, e i is the emotion category, s i It’s the emotional intensity; According to the emotion control requirements, the system retrieves the appropriate template from the template library and dynamically fills it: R=Fill(t selected ,X context ) The template selection function uses cosine similarity matching: Sentiment consistency check: The generated candidate responses are evaluated by the sentiment consistency scorer to ensure that they are consistent with the target sentiment: Score emotion (R)=Classifier emotion (R) If the consistency score falls below a threshold, the system regenerates or adjusts the response: while Score_emotion(R)<θ_consistency: R=RegenerateResponse(X,S_emotion,previous_R).

9. The adaptive interactive language learning system enabled by multimodal affective computing according to claim 8, characterized in that: The emotional expression speech synthesis system uses advanced text-to-speech (TTS) technology to achieve emotion-controllable speech synthesis. The architecture includes: The core TTS architecture adopts a two-stage TTS architecture, including an acoustic model and a vocoder: FastSpeech 2 acoustic model, based on non-autoregressive Transformer, realizes the mapping from text to mel spectrogram: D=PhonemeExtractor(text) H D =PhonemeEncoder(D) H L =H D +VarianceAdaptor(H D ,p,e,d) M=MelDecoder(H L ) Where p is pitch information, e is energy information, d is duration information, and M is the Mel spectrogram. The training objectives of the FastSpeech 2 acoustic model are: HiFi-GAN vocoder, converts mel-spectrograms into waveforms, using a generative adversarial network architecture: The training objective combines multiple loss functions: in It's about fighting losses. is the feature matching loss, is the Mel-spectrogram reconstruction loss; emotion control mechanism, to achieve emotion-controllable speech synthesis, the system adopts the following technologies: global emotion embedding, adding emotion embedding to the acoustic model: H D '=H D +EmotionEmbedding(e emotion ) Sentiment embedding is obtained through a lookup table: e emotion =LookupTable(emotion_id) Or generated by continuous parameters: e emotion =f emotion (valence,arousal,dominance) f emotion (v,a,d)=W e ·[v,a,d]+b e Fine-grained prosody control, controlling the acoustic parameters of speech to express emotions, pitch adjustment: p'=p·(1+α pitch ·f pitch (emotion)) Speech speed adjustment: d'=d·(1+α duration ·f duration (emotion)) Energy Adjustment: e'=e·(1+α energy ·f energy (emotion)) Reference encoder, extracting style features from emotional speech reference: e ref =ReferenceEncoder(x ref ) H D ″=H D +StyleAdapter(e ref ) Prosody model,To enhance the naturalness of emotional speech, we establish a prosody prediction model: p,e,d=VariancePredictor(H D ,emotion) The rhythm prediction model consists of three sub-networks, each of which has the following architecture: Conv1D(kernel_size=3,filters=256)->ReLU->LayerNorm-> Conv1D(kernel_size=3,filters=256)->ReLU->LayerNorm-> Conv1D(kernel_size=3,filters=1) The relationship between prosodic features and emotions is realized through conditional input: h in =[h phoneme ;h emotion ] The statistical characteristics of each prosodic parameter are also adjusted according to the emotion type: μ p (e),σ p (e),μ d (e),σ d (e),μ e (e),σ e (e); The virtual image expression and action generation system provides expression and action generation functions for scenarios where a 3D virtual human image is used. The system architecture includes: Facial expression generation, BlendShape parameter control, based on the standard facial BlendShape system, control the expression of the virtual image: BS={bs1,bs2,...,bs K } Each bs i ∈[0,1] controls an expression parameter; Emotion to BlendShape mapping function: BS=f emotion2bs (emotion,intensity) Implemented as a neural network: BS=σ(W bs ·[e emotion ;intensity]+b bs ) Dynamic expression control to generate coherent expression changes: BS t =α·BS target +(1-α)·BS t-1 α=min(1,Δt / τ) Where τ is the expression transition time constant, which depends on the rapidity of the expression change; Emotional empathy expression: Generate empathic responses based on the user's recognized expressions: BS empathy =f empathy (BS user ,emotion strategy ) When users express anxiety, the AI ​​displays an expression of understanding and concern; Body action generation, based on a rule-based action library, with a series of preset actions related to emotions and dialogue functions: where a i is the action sequence, e i is the associated emotion, f i It is a dialogue function; Action selection function: Based on sequence-to-sequence action generation, train sequence-to-sequence models to generate action sequences from text and emotions: A=Seq2SeqModel(Text,Emotion) The specific implementation is a Transformer model, and the output is a sequence of action parameters; Action fusion and transition, smooth fusion of multiple actions: where s i is the action selection score; Audio and video synchronization, ensuring synchronization of voice and animation: P viseme,t =f viseme (phoneme t-δ ) Among them, P viseme is the lip shape parameter, δ is the time offset, taking into account the processing delay; Lip sync uses a phoneme-based mapping table: The user interface adopts a simple and friendly design and provides the following functions: Interface layout optimization, using attention heat map analysis to optimize the layout of interface elements: Importance scoring function of interface elements: f importance (element) = w1·frequency(element) + w2·criticality(element) Personalized customization: Users customize multiple aspects of their AI companions: Visual image, choose a different avatar or adjust the existing one; Sound characteristics, select different sounds or adjust timbre parameters; Interaction style, set the preferred interaction method and rhythm; Personalization parameters are stored in the User model: U preference ={voice,avatar,style,feedback_level,...} Progress tracking and incentive system; Visualization of learning progress: Design an achievement system, including: fixed achievements based on milestones; Achievements based on skill improvement; Achievement based on consistency; The system uses an achievement unlocking algorithm based on a directed acyclic graph (DAG) to achieve more flexible achievement dependency processing: mermaidgraph TD A[achievement network DAG]-->B[topological sorting]B-->C[user status evaluation]C-->D[condition scoring]D-->E[threshold determination]E-->F[cascade unlocking]F-->G[achievement update]; Progress data is regularly synchronized to the backend and used for model optimization: U progress,t =f update (U progress,t-1 ,Performance t ) Adaptive interface elements dynamically adjust according to the learner's emotional state: Anxiety state, simplify the interface and reduce distracting elements; Confidence state, showing more challenge options and advanced content; Interface element visibility control function: Visibility(element)=f visibility (element,E user ) Overall interface complexity adjustment function: Complexity UI =C base ·(1-α·E anxiety )。 10. Adaptive interactive language learning supporting methods enabled by multimodal affective computing, The adaptive interactive language learning system enabled by multimodal affective computing according to claim 9 is characterized in that: include: System initialization, loading models, connecting hardware, and preparing the environment; Module coordination, startup and monitoring of each core functional module; Scheduled task management, setting and executing periodic tasks, such as model updates; System health monitoring, ensuring the normal operation of each module through heartbeat checks; Event processing, responding to events triggered internally and externally by the system; State synchronization ensures that data and state between modules remain consistent.

Images