Multi-modal self-adaptive oral English training method, system and device and medium

Abstract

The invention relates to a multi-mode self-adaptive oral English training method, system and device and a medium. The method comprises the following steps: acquiring multi-modal data; extracting features of spoken language ability corresponding to the multi-modal data to obtain a feature vector group; performing standardization processing on the feature vector group to obtain an index matrix; inputting the index matrix into a pre-trained attention mechanism model to obtain a spoken language ability portrait corresponding to the user; taking the spoken language ability portrait as an index, and matching corresponding teaching resources in a preset training resource library to obtain modular training resources; the modular training resources are courses corresponding to capability defects of the user; and generating a training scheme of the corresponding user based on the modular training resources. By adopting the method, multi-modal data can be introduced to carry out quantitative evaluation on the spoken language ability of the user, and the reasons of errors in the spoken language dialogue process of the student can be effectively judged.

Description

Technical Field

[0001] This invention belongs to the field of computer-aided instruction, and in particular relates to a multimodal adaptive English oral training method, system, device and medium. Background Technology

[0002] With the advanced development of speech analysis technology, intelligent English speaking training systems are becoming increasingly popular. This technology uses acoustic feature decoding to quantitatively assess surface-level indicators such as pronunciation accuracy and fluency. Its core advantage lies in providing immediate and objective speech feedback. Compared to traditional subjective teacher evaluation or tape-based shadowing training, the new technology can achieve millisecond-level error localization and comparison with standard pronunciation waveforms, significantly improving training efficiency. Traditional speech analysis techniques mainly rely on teachers observing speech spectrograms to identify formant shifts and providing adjustment suggestions based on professional experience; or comparing learners' pronunciation with a standard phoneme library and calculating similarity using the DTW algorithm. However, existing technologies only use single-modal speech data to quantitatively assess learners' speaking abilities, leading to an inability to determine the causes of errors during spoken dialogues and neglecting the influence of the environment on learners. Summary of the Invention

[0003] Therefore, it is necessary to provide a multimodal adaptive English oral training method, system, device, and medium that can quantitatively assess users' oral proficiency through multimodal data, determine the reasons for learners' errors, and consider the impact of the environment on learners, in order to address the above-mentioned technical problems.

[0004] Firstly, this application provides a multimodal adaptive English speaking training method, including:

[0005] Acquire multimodal data; and extract features corresponding to spoken language ability from the multimodal data to obtain feature vector groups; the multimodal data consists of eye-tracking data, speech data, and motion data perceived by various sensors when the user performs a spoken dialogue task;

[0006] The feature vector group is standardized to obtain the index matrix;

[0007] By inputting the indicator matrix into a pre-trained attention mechanism model, a profile of the user's spoken language ability is obtained.

[0008] Using the oral proficiency profile as an index, matching the corresponding teaching resources in the preset training resource library yields modular training resources; the modular training resources are courses that address the corresponding user's proficiency deficiencies.

[0009] Based on modular training resources, training plans are generated for corresponding users.

[0010] Furthermore, features corresponding to spoken language ability in the multimodal data are extracted to obtain a feature vector set, including:

[0011] Based on eye-tracking data, visual attention features are extracted to obtain visual feature vectors;

[0012] Based on speech data, auditory comprehension features are extracted to obtain auditory feature vectors; and spoken speech features are extracted to obtain speech feature vectors.

[0013] Based on motion data, gesture interaction features are extracted to obtain gesture feature vectors;

[0014] Visual feature vectors, auditory feature vectors, speech feature vectors, and gesture feature vectors are concatenated to obtain a feature vector group.

[0015] Furthermore, based on eye-tracking data, visual attention features are extracted to obtain a visual feature vector, including:

[0016] Data points with fixation durations less than a preset threshold are removed from the eye-tracking data to obtain an effective fixation point sequence;

[0017] The valid annotation point sequence is mapped to the AOI region to obtain the AOI mapping table;

[0018] Calculate thermodynamic characteristic scalars based on the AOI mapping table;

[0019] Based on the AOI mapping table, the attention entropy value is calculated using the following formula:

[0020] H = -∑[P(i)×log2(P(i))]

[0021] Where H is the attention entropy value and P(i) is the AOI fixation probability;

[0022] Based on preset rules, consecutive annotation points in the effective gaze point sequence are merged to obtain the effective attention duration; and the effective attention duration, attention entropy value and thermal feature scalar are concatenated to obtain the visual feature vector.

[0023] Furthermore, based on the AOI mapping table, thermodynamic characteristic scalars are calculated, including:

[0024] Based on the AOI mapping table, the proportion of dominant AOIs is calculated using the following formula:

[0025]

[0026] Among them, DAR is the dominant AOI percentage, T maxAOI T represents the cumulative fixation time of the single AOI region that receives the most fixations. allAOI This represents the total cumulative fixation time across all AOI regions.

[0027] Based on the AOI mapping table, the long-term fixation percentage is calculated using the following formula:

[0028]

[0029] Where LFP is the percentage of long-term fixation, N long N represents the number of long-term fixations. total ∑dur is the total number of all valid fixations. long The cumulative duration of all long fixations, ∑dur all The total cumulative duration for all fixations;

[0030] The proportion of dominant AOI and the proportion of long-term gaze are combined into a thermal characteristic scalar.

[0031] Furthermore, based on the speech data, auditory understanding features are extracted, including:

[0032] Speech activity detection is performed on the speech data to obtain a time-stamped sequence;

[0033] The response delay is calculated based on the time-stamped sequence to obtain the response delay value;

[0034] MFCC features are extracted from non-user speech data to obtain MFCC feature vectors, and statistical features are extracted based on the MFCC feature vectors to obtain statistical feature vectors; the MFCC feature matrix is ​​used to characterize the hearing impairments of users in dialogue tasks.

[0035] Based on a pre-trained noise classification model, the noise resistance index is calculated for speech data using the following formula:

[0036] RI=e -0.1×max(SNR-15,0)

[0037] Wherein, RI is the noise immunity index and SNR is the signal-to-noise ratio;

[0038] The response latency value, statistical feature vector, and noise resistance index are concatenated to obtain auditory understanding features. Further, based on motion data, gesture interaction features are extracted to obtain a gesture feature vector, including:

[0039] Based on a preset speed threshold rule, the motion data is segmented into a sequence of valid gesture segments; the sequence of valid gesture segments is a time series, used to represent the start time and end time of the motion.

[0040] Based on the effective gesture segment sequence, the maximum range of motion for each segment is calculated to obtain the motion amplitude vector;

[0041] Based on the effective gesture segment sequence, the peak point of motion intensity is located to obtain the peak time series;

[0042] The synchronization index of peak time series and keyword time points is calculated using the following formula:

[0043]

[0044] Among them, Sync k Let Δt be the synchronization index of the k-th keyword and action. k λ represents the time delay between the k-th keyword and the action, λ is the weighting coefficient, and the keyword time point is the time point in the speech data extracted from the speech feature vector where the keyword appears.

[0045] The gesture feature vector is obtained by concatenating the motion amplitude vector and the synchronization index.

[0046] Furthermore, the feature vector group is standardized to obtain the index matrix, including:

[0047] The features in the feature vector group are classified to obtain the feature types; and the features in the feature vector group are normalized based on the feature types to obtain normalized features.

[0048] Based on cultural background labels, the normalized feature vector is corrected to obtain cultural calibration features; cultural background labels are identifiers used to represent the user's cultural background.

[0049] Based on style features and spoken dialogue tasks, weights are assigned to cultural calibration features to obtain weighted features; style features are preset spoken communication style types corresponding to the users.

[0050] Outlier smoothing is performed on the weighted features to obtain the index matrix.

[0051] Secondly, this application also provides a multimodal adaptive English speaking training system, including:

[0052] The feature module is used to acquire multimodal data and extract features corresponding to spoken language ability from the multimodal data to obtain feature vector groups; the multimodal data consists of eye-tracking data, speech data, and motion data perceived by various sensors when the user performs a spoken dialogue task.

[0053] The standardization module is used to standardize the feature vector set to obtain the index matrix;

[0054] The capability module is used to input the indicator matrix into the pre-trained attention mechanism model to obtain the corresponding user's oral ability profile.

[0055] The resource module is used to match the corresponding teaching resources in the preset training resource library with the oral proficiency profile as the index to obtain modular training resources; the modular training resources are courses corresponding to the user's ability deficiencies.

[0056] The training module is used to generate training plans for corresponding users based on modular training resources.

[0057] Thirdly, this application also provides a computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any step of the method provided in the first aspect of this application.

[0058] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any step of the method provided in the first aspect of this application.

[0059] The aforementioned multimodal adaptive English oral training method, system, device, and medium acquire multimodal data; extract features corresponding to oral abilities from the multimodal data to obtain feature vector sets; the multimodal data consists of eye-tracking data, speech data, and motion data perceived by various sensors when a user performs an oral dialogue task; the feature vector sets are standardized to obtain an index matrix; the index matrix is ​​input into a pre-trained attention mechanism model to obtain an oral ability profile of the corresponding user; using the oral ability profile as an index, corresponding teaching resources in a pre-set training resource library are matched to obtain modular training resources; the modular training resources are courses addressing the corresponding user's ability deficiencies; and a training plan for the corresponding user is generated based on the modular training resources. This technical means enables multimodal data to quantitatively assess learners' oral abilities, effectively determine the causes of learners' errors, consider the impact of the environment on learners, and thus provide learners with more accurate and suitable training plans. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 A flowchart illustrating a multimodal adaptive English oral training method provided in an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of the structure of a multimodal adaptive English oral training system provided in an embodiment of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] In one embodiment, such as Figure 1 As shown, a multimodal adaptive English speaking training method is provided. This embodiment illustrates the method's application to a terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0065] Step 101: Acquire multimodal data; and extract the features of spoken language ability corresponding to the multimodal data to obtain a feature vector group; the multimodal data consists of eye-tracking data, speech data, and motion data perceived by the user through various sensors when performing spoken dialogue tasks.

[0066] Specifically, multimodal data refers to the eye movement trajectory data, audio data of the user and the other party, gesture movement data, and body movement data collected by sensors during a user's oral dialogue task. The feature vector set is a multidimensional vector set representing the user's oral ability, composed of visual sub-feature vectors, auditory sub-feature vectors, speech sub-feature vectors, and gesture sub-feature vectors. The terminal can collect raw data from the user's dialogue in real time through wearable eye trackers, microphone arrays, and motion sensors. Eye tracker data is used to calculate effective fixation duration and attention entropy as visual attention features. Speech data is used to separate user and non-user speech, and then calculates features such as fluency and pronunciation accuracy as spoken speech features. Non-user speech is used to extract features such as response latency and noise resistance as auditory comprehension features. Motion data is used to calculate movement amplitude and language-motor synchronization as gesture interaction features. These sub-features are combined into a structured feature vector set.

[0067] Step 102: Standardize the feature vector group to obtain the index matrix.

[0068] The indicator matrix is ​​a standardized feature data matrix, where each row represents a feature and each column represents a sampling. The terminal can group the feature vectors into visual, auditory, and action categories according to their physical meaning, normalize each category of features to eliminate differences in units, adjust feature weights based on user cultural background tags, assign higher weights to relevant features according to preset spoken language styles, and use outlier smoothing algorithms to handle outliers, resulting in an indicator matrix that can be directly input into the model.

[0069] Step 103: Input the indicator matrix into the pre-trained attention mechanism model to obtain the oral proficiency profile of the corresponding user.

[0070] For example, the attention mechanism model is a pre-trained neural network model, such as a Transformer architecture attention model, which focuses on key features through weight allocation. The spoken language proficiency profile is a quantitative analysis result of a user's abilities, in the form of a multi-dimensional vector; in this embodiment, it is a three-dimensional vector, which may include cognitive, expressive, and interactive dimensions. The terminal inputs the indicator matrix into the pre-trained Transformer architecture attention model, which automatically allocates weights and outputs a three-dimensional vector with cognitive, expressive, and interactive dimensions. The cognitive dimension includes listening comprehension and visual attention stability; the expressive dimension includes pronunciation accuracy and fluency; and the interactive dimension includes the synchronization of action and language, thus obtaining the spoken language proficiency profile.

[0071] Step 104: Using the oral proficiency profile as an index, match the corresponding teaching resources in the preset training resource library to obtain modular training resources; the modular training resources are courses corresponding to the user's ability deficiencies.

[0072] Optionally, the training resource library is a pre-defined course library that can be indexed by competency deficiency tags. Modular training resources are micro-course units dynamically combined according to user deficiencies, including teaching content, practice tools, and assessment rules. The dimensional values ​​of the competency profile are mapped to competency deficiency tags in the resource library. Corresponding courses are retrieved from the resource library based on the competency deficiency tags, prioritizing resources that match the user's cultural background. Multiple micro-course units are packaged into a customized course package to obtain modular training resources.

[0073] Step 105: Based on modular training resources, generate training plans for the corresponding users.

[0074] Specifically, the training plan is an adjustable schedule that includes resource execution order, intensity, and feedback mechanisms. Daily training tasks are allocated based on user idle time, the difficulty is adjusted according to the severity of defects in the user profile, a feedback loop is preset, and subsequent plans are automatically optimized based on new data, resulting in a continuously updated training plan. This forms a closed loop of evaluation, training, and re-evaluation, solving the problem of rigid plans in traditional methods.

[0075] This embodiment provides a multimodal adaptive English oral training method that acquires multimodal data and extracts features corresponding to oral abilities from the multimodal data to obtain feature vector sets. The multimodal data consists of eye-tracking data, speech data, and motion data perceived by various sensors during oral dialogue tasks. The feature vector sets are standardized to obtain an index matrix. The index matrix is ​​input into a pre-trained attention mechanism model to obtain a profile of the corresponding user's oral ability. Using the oral ability profile as an index, corresponding teaching resources in a pre-set training resource library are matched to obtain modular training resources. The modular training resources are courses addressing the corresponding user's ability deficiencies. Based on the modular training resources, a training plan for the corresponding user is generated. Through these technical means, multimodal data can be used to quantitatively assess learners' oral abilities, effectively determine the causes of learners' errors, consider the impact of the environment on learners, and thus provide more accurate and suitable training plans for learners.

[0076] In one embodiment, features corresponding to spoken language ability in multimodal data are extracted to obtain a feature vector set, including:

[0077] Step 201: Based on eye-tracking data, extract visual attention features to obtain visual feature vectors.

[0078] The eye-tracking data consists of eye movement trajectories sampled at a rate of 60 Hz (Hertz) or higher, including gaze coordinates and duration. The visual feature vector is a numerical vector quantifying visual attention. The terminal removes noise points with gaze durations less than a threshold, retains valid gaze sequences, maps gaze points to preset areas of interest (AOIs), generates an AOI mapping table, calculates the proportion of dominant AOIs and long-duration gazes, merges consecutive gazes, accumulates stable attention time, and concatenates the resulting data to obtain the visual feature vector.

[0079] Step 202: Based on the speech data, extract auditory comprehension features to obtain auditory feature vectors; and extract spoken speech features to obtain speech feature vectors.

[0080] Indicatively, the auditory feature vector is a numerical vector representing auditory comprehension ability. The speech feature vector is a numerical vector representing spoken expression ability. The terminal segments user and non-user speech segments, generates time-stamped sequences, defines the difference between the start time of the user's speech and the end time of the other party's speech as the response delay value, extracts MFCC (Mel-Frequency Cepstral Coefficients) features from the non-user speech to represent phoneme quality, calculates MFCC statistical features as a statistical feature vector, and then calculates the noise resistance index. The response delay value, statistical feature vector, and noise resistance index are concatenated to obtain the auditory feature vector; the fundamental frequency and formants are extracted to quantify pronunciation accuracy, and the proportion of silent segments and speech rate are calculated to quantify fluency, thus generating the speech feature vector.

[0081] Step 203: Based on the action data, extract gesture interaction features to obtain gesture feature vectors.

[0082] Specifically, the gesture feature vector is a numerical vector that characterizes the quality of gesture interaction. The terminal performs action segmentation and motion feature extraction on the gesture action data to obtain the motion amplitude vector and peak time series. It calculates the synchronization index between the peak time series and the keyword time points, and concatenates it with the motion amplitude vector to obtain the gesture feature vector.

[0083] Step 204: Concatenate the visual feature vector, auditory feature vector, speech feature vector, and gesture feature vector to obtain a feature vector group.

[0084] The feature vector group is the result of concatenating multimodal features. The terminal concatenates the visual feature vector, auditory feature vector, speech feature vector, and gesture feature vector in modal order, and adds a timestamp and task ID (IdentityDocument, a unique code) to obtain the feature vector group.

[0085] This embodiment incorporates motion and eye-tracking data into the assessment of learners' oral communication skills through cross-modal association, thereby better quantifying learners' oral communication abilities and providing the possibility of targeted training based on learners' weaknesses, thus improving the accuracy of oral communication training.

[0086] In one embodiment, visual attention features are extracted based on eye-tracking data to obtain a visual feature vector, including:

[0087] Step 301: Remove data points from the eye-tracking data whose fixation duration is less than a preset threshold to obtain an effective fixation point sequence.

[0088] Specifically, the preset threshold is the shortest fixation duration used to distinguish between conscious fixation and unconscious saccades. The effective fixation point sequence is a time-ordered set of stable eye rests with fixation durations exceeding the threshold. The terminal filters the raw eye movement data by fixation point duration, removing points with fixation durations less than the threshold, and retaining the coordinates, duration, and timestamps of the remaining points to form an ordered sequence. For example, noise generated by blinks or random eye saccades may be captured, forming instantaneous points, i.e., points with fixation durations less than the threshold.

[0089] Step 302: Map the valid annotation point sequence to the AOI region to obtain the AOI mapping table.

[0090] Optionally, the AOI region is a preset area of ​​interest. The AOI mapping table is a two-dimensional table, with rows representing AOI regions and columns including the number of gaze points, cumulative gaze duration, and gaze probability. The terminal matches the coordinates of each valid gaze point to the nearest AOI region, counts the number of gaze points in each AOI, sums up the duration of all gaze points in that region, calculates the gaze probability, and obtains the AOI mapping table.

[0091] Step 303: Calculate the thermodynamic characteristic scalar based on the AOI mapping table.

[0092] Schematic, the thermal characteristic scalar is a combination of two scalar values: DAR (Dominant AOI Ratio) and LFP (Long Fixation Proportion). DAR represents the percentage of time spent in the most focused area, while LFP is an indicator of the intensity of prolonged fixation. The thermal characteristic scalar is obtained by calculating DAR based on the duration of the most focused single AOI and the total duration of all AOIs, and by calculating LFP based on the number of long-term fixations and the total number of fixations.

[0093] Step 304: Based on the AOI mapping table, calculate the attention entropy value using the following formula:

[0094] H = -∑[P(i)×log2(P(i))]

[0095] Where H is the attention entropy value and P(i) is the AOI fixation probability.

[0096] To illustrate, the attention entropy value is an indicator of attentional disorganization based on information entropy; the higher the value, the more scattered the attention. The terminal obtains the gaze probability of each region from the AOI mapping table and calculates the attention entropy value using a formula.

[0097] Step 305: Based on preset rules, merge consecutive annotation points in the effective gaze point sequence to obtain the effective attention duration; and concatenate the effective attention duration, attention entropy value, and thermal feature scalar to obtain the visual feature vector.

[0098] Effective attention duration is the net attention duration after merging consecutive fixation points. The terminal merges consecutive points based on preset rules, sums all merged durations as the effective attention duration, and combines the three indicators into a vector to obtain a visual feature vector. For example, the preset rule could be to consider points with a spacing of less than 50 ms (milliseconds) and within the same AOI region as consecutive points.

[0099] This embodiment simulates the human brain's attention integration mechanism by merging continuous points, avoiding mechanical segmentation. The combination of attention entropy value and LFP distinguishes between active and passive attention distraction, thereby improving the detection rate of trainees' defects and thus enhancing the pertinence of oral training.

[0100] In one embodiment, the calculation of thermal characteristic scalars based on the AOI mapping table includes:

[0101] Step 401: Based on the AOI mapping table, calculate the proportion of dominant AOIs using the following formula:

[0102]

[0103] Among them, DAR is the dominant AOI percentage, T maxAOI T represents the cumulative fixation time of the single AOI region that receives the most fixations. allAOI This represents the total cumulative fixation time across all AOI regions.

[0104] Specifically, the dominant AOI percentage is the percentage of time spent in the most-watched area. The terminal iterates through the AOI mapping table to extract the cumulative gaze duration for each AOI area, finds the AOI area corresponding to the maximum value, sums up the duration of all areas, and calculates the dominant AOI percentage using a formula.

[0105] Step 402: Based on the AOI mapping table, calculate the long-term fixation percentage using the following formula:

[0106]

[0107] Where LFP is the percentage of long-term fixation, N long N represents the number of long-term fixations. total ∑dur is the total number of all valid fixations. long The cumulative duration of all long fixations, ∑dur all This represents the total cumulative duration for all fixations.

[0108] Optionally, the percentage of long-term fixation is an intensity indicator used to characterize long-term fixation. It is about paying attention to the frequency of events over a long period of time, which can avoid the problem of fragmentation being masked by a single excessively long gaze. This represents the proportion of long-term attention intensity, used to quantify attention duration. The terminal iterates through the effective fixation point sequence, filters out points with a fixation duration greater than 500ms, and substitutes them into the formula to obtain the proportion of long-term fixation.

[0109] Step 403: The proportion of dominant AOI and the proportion of long-term gaze are spliced ​​together to form a thermal characteristic scalar.

[0110] Indicatively, the proportion of dominant AOIs and the proportion of long-term gaze are sequentially concatenated into a two-dimensional vector to obtain a thermal feature scalar.

[0111] This embodiment improves the accuracy of student defect classification by calculating the proportion of dominant AOIs and the proportion of long-term fixation, thereby improving the accuracy of oral training program generation.

[0112] In one embodiment, auditory understanding features are extracted based on speech data, including:

[0113] Step 501: Perform speech activity detection on the speech data to obtain a time-stamped sequence.

[0114] The time-stamped sequence is a structure array that may include start time, end time, and speaker label. For example, the terminal divides the speech data into 20ms frames with a 50% overlap rate, calculates the short-time energy and zero-crossing rate of each frame, and if the short-time energy and zero-crossing rate meet preset conditions, the frame is defined as being within the speech segment. At the same time, the user and non-user speech boundaries are marked to obtain the time-stamped sequence.

[0115] Step 502: Calculate the response delay based on the time stamp sequence to obtain the response delay value.

[0116] Specifically, the response latency value is the difference between the start time of the user's voice and the end time of the preceding non-user voice. The terminal traverses the time stamp sequence, identifies the process from the end of an adjacent non-user segment to the start of a user segment, locates the dialogue turn, and calculates the difference between the end of an adjacent non-user segment and the start of a user segment in each dialogue as the response latency value. For example, if the response latency is greater than 2000ms, it is considered an invalid interaction.

[0117] Step 503: MFCC feature extraction is performed on the non-user speech data in the speech data to obtain MFCC feature vectors, and statistical features are extracted based on the MFCC feature vectors to obtain statistical feature vectors; the MFCC feature matrix is ​​used to characterize the hearing impairments of users in the dialogue task.

[0118] To illustrate, the statistical feature vector is a statistical measure extracted from MFCC (Multi-Functional Characteristic Carrier), which may include dimensions such as mean, variance, skewness, and kurtosis. The terminal only processes the speech segments of the conversation parties, that is, the speech of non-students, excluding interference from the user's own pronunciation. It extracts speech features through MFCC and calculates the mean, variance, skewness, and kurtosis for each dimension of the MFCC features to obtain the statistical feature vector.

[0119] Step 504: Based on the pre-trained noise classification model, calculate the noise resistance index for the speech data using the following formula:

[0120] RI=e -0.1×max(SNR-15,0)

[0121] Where RI is the noise immunity index and SNR is the signal-to-noise ratio.

[0122] Specifically, SNR (Signal-to-Noise Ratio) is output by a pre-trained noise classification model. RI (Robustness Index) characterizes the learner's adaptability to environmental noise. The terminal inputs speech data into the noise classification model to obtain the noise type and SNR value, which are then used to calculate the noise resistance index. An SNR less than 15 dB (decibel) is considered ideal noise resistance; for every 1 dB increase in SNR, the noise resistance index decreases by approximately 9.5%.

[0123] Step 505: The response delay value, statistical feature vector and noise resistance index are concatenated to obtain auditory understanding features.

[0124] Specifically, the terminal aligns the dimensions of the response latency value, statistical feature vector, and noise resistance index, arranges them in order, and splices them together to obtain auditory understanding features.

[0125] This embodiment introduces MFCC statistical features to locate specific phoneme confusions, improving the accuracy of diagnosing students' hearing defects and thus enhancing the reliability of quantifying students' abilities. This provides more multi-dimensional input for subsequent oral training program decisions and improves the accuracy of oral training.

[0126] In one embodiment, the feature is that, based on motion data, gesture interaction features are extracted to obtain a gesture feature vector, including:

[0127] Step 601: Based on a preset speed threshold rule, the motion data is segmented into a sequence of valid gesture segments; the sequence of valid gesture segments is a time series, used to characterize the start time and end time of the motion.

[0128] The velocity threshold rule is a preset angular velocity threshold value used to distinguish between valid gestures and invalid micro-movements. Actions below this threshold are considered non-gestures. Motion data is raw sensor data acquired from the inertial measurement unit, including three-dimensional angular velocity and three-dimensional acceleration. The valid gesture segment sequence is a series of time segments, each representing the start and end times of a valid gesture. The data structure is a sequence of tuples containing start and end times. For example, the terminal calculates the angular velocity modulus for the three-dimensional angular velocity components at each time point. When the angular velocity modulus first exceeds the threshold, it is marked as the gesture start time. When the angular velocity modulus drops below the threshold and remains below for more than 100 milliseconds, it is marked as the gesture end time. The start and end time sequences of all valid segments are output, and segments with a duration of less than 200 milliseconds are discarded to obtain the valid gesture segment sequence.

[0129] Step 602: Based on the effective gesture segment sequence, calculate the maximum range of motion for each segment to obtain the motion amplitude vector.

[0130] Specifically, the range of motion is the extreme difference of hand displacement within a gesture segment, represented using Euclidean distance. The motion amplitude vector is a vector composed of the motion ranges of all valid gesture segments in sequence. For each valid segment, the terminal fuses acceleration and angular velocity data, calculates the hand trajectory in three-dimensional space using a Kalman filter algorithm, calculates the displacement range within the time interval of each segment, and arranges the amplitude values ​​of each segment in chronological order to form a one-dimensional vector, which serves as the motion amplitude vector, where each element represents the motion amplitude of the corresponding gesture segment.

[0131] Step 603: Based on the effective gesture segment sequence, locate the peak point of motion intensity to obtain the peak time sequence.

[0132] Indicatively, motion intensity is the modulus of acceleration, a scalar used to characterize the instantaneous force of a gesture. The peak time series is the sequence of time points in each valid gesture segment where the motion intensity reaches its maximum value. The terminal calculates the modulus of the three-dimensional acceleration components at each time point, finds the global maximum value of the acceleration modulus within the time window of each valid segment, records the timestamp corresponding to this peak, and arranges all peak time points in chronological order to obtain the peak time series. This provides key time anchors for subsequent synchronization analysis, reflecting the timing of the gesture's force application.

[0133] Step 604: Calculate the synchronization index of the peak time series and keyword time points using the following formula:

[0134]

[0135] Among them, Sync k Let Δt be the synchronization index of the k-th keyword and action. kLet λ be the time delay between the k-th keyword and the action, λ be the weighting coefficient, and the keyword time point be the time point in the speech data extracted from the speech feature vector where the keyword appears.

[0136] Specifically, the keyword timing is the moment of occurrence of emphasized or interrogative words detected in the speech data, provided by the speech feature vector. The time delay is the absolute difference between the peak time of the k-th gesture and the time of the most recent keyword occurrence. The synchronization index is an index quantifying the synchronization between gestures and speech keywords, with a value range of 0 to 1, calculated using an exponential decay formula. The weighting coefficient is a parameter controlling the synchronization tolerance. For each gesture's peak time, the terminal searches for the most recent speech keyword time within a 1-second window before and after it, calculates the time difference, substitutes it into the formula to calculate the synchronization index, and generates a sequence composed of the synchronization indices of all gestures.

[0137] Step 605: Concatenate the motion amplitude vector and the synchronization index to obtain the gesture feature vector.

[0138] For example, the terminal calculates the mean and standard deviation of the motion amplitude vector, calculates the mean of the synchronization index sequence, and combines the three statistics into a three-dimensional vector to obtain the gesture feature vector.

[0139] This embodiment uses a synchronization index to correlate gestures and speech, diagnoses deep interaction problems between speech and actions, identifies user expression styles, detects gesture stability, and evaluates the coordination between language and actions to drive the training program to adapt to the user's expression style, thereby improving the relevance of the oral training program.

[0140] In one embodiment, the feature vector group is standardized to obtain an index matrix, including:

[0141] Step 701: Classify the features in the feature vector group to obtain feature types; and normalize the features in the feature vector group based on the feature types to obtain normalized features.

[0142] Specifically, feature types are categories categorized by physical meaning, and can include continuous features, probabilistic features, vector features, etc. Normalized features are the set of feature values ​​after eliminating dimensional differences. The terminal iterates through each dimension of the feature vector group, assigns a type label to the feature based on its attributes, performs Z-score normalization on continuous features, Logit transformation on probabilistic features, and min-max normalization on vector features. The normalized features are then concatenated to obtain the normalized features.

[0143] Step 702: Based on the cultural background label, the normalized feature vector is corrected to obtain the cultural calibration feature; the cultural background label is an identifier used to represent the user's cultural background.

[0144] Among them, cultural background tags are preset classification identifiers used to store user registration information. Cultural calibration features are the set of feature values ​​after eliminating cultural bias. The terminal loads a predefined calibration rule library, which includes a cultural correction parameter table, iterates through the normalized feature vectors, applies culturally specific adjustment features, and obtains the cultural calibration features.

[0145] Step 703: Based on style features and spoken dialogue task, assign weights to cultural calibration features to obtain weighted features; style features are preset spoken communication style types corresponding to the user.

[0146] In illustration, style features represent user-preset communication style types. The spoken dialogue task is the training scenario classification for the current spoken dialogue. Weighted features are the feature set after strengthening task-related features. The terminal loads basic weights based on style features, adds task-specific weights, and finally processes the cultural calibration features according to the weights to obtain the weighted features.

[0147] Step 704: Perform outlier smoothing on the weighted features to obtain the index matrix.

[0148] For example, outliers are feature values ​​that deviate from the normal range by more than 3σ. The index matrix is ​​a two-dimensional matrix of the final standardized output. The terminal calculates dynamic thresholds based on user historical data, replaces isolated outliers with linear interpolation between consecutive frames, fills continuous outlier segments with the historical mean of features, and organizes weighted features according to time series to obtain the index matrix.

[0149] This embodiment eliminates cultural bias by using cultural perception calibration and task-adaptive weighting, enabling differentiated assessment of the same learner across different tasks. This allows for differentiated assessment of learner abilities and provides more targeted oral training solutions.

[0150] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0151] Based on the same inventive concept, this application also provides a multimodal adaptive English speaking training system for implementing the multimodal adaptive English speaking training method described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more embodiments of the multimodal adaptive English speaking training system provided below can be found in the limitations of the multimodal adaptive English speaking training method described above, and will not be repeated here.

[0152] In one exemplary embodiment, such as Figure 2 As shown, a multimodal adaptive English speaking training system 800 is provided, including:

[0153] Feature module 801 is used to acquire multimodal data and extract features corresponding to spoken language ability from the multimodal data to obtain feature vector groups; the multimodal data consists of eye-tracking data, speech data, and motion data perceived by various sensors when the user performs a spoken dialogue task.

[0154] Standardization module 802 is used to standardize the feature vector group to obtain the index matrix;

[0155] Capability module 803 is used to input the indicator matrix into the pre-trained attention mechanism model to obtain the oral proficiency profile of the corresponding user.

[0156] Resource module 804 is used to match the corresponding teaching resources in the preset training resource library with the oral proficiency profile as the index to obtain modular training resources; the modular training resources are courses corresponding to the user's ability deficiencies.

[0157] Training module 805 is used to generate training plans for corresponding users based on modular training resources.

[0158] Furthermore, feature module 801 is also used for:

[0159] Based on eye-tracking data, visual attention features are extracted to obtain visual feature vectors;

[0160] Based on speech data, auditory comprehension features are extracted to obtain auditory feature vectors; and spoken speech features are extracted to obtain speech feature vectors.

[0161] Based on motion data, gesture interaction features are extracted to obtain gesture feature vectors;

[0162] Visual feature vectors, auditory feature vectors, speech feature vectors, and gesture feature vectors are concatenated to obtain a feature vector group.

[0163] Furthermore, feature module 801 is also used for:

[0164] Data points with fixation durations less than a preset threshold are removed from the eye-tracking data to obtain an effective fixation point sequence;

[0165] The valid annotation point sequence is mapped to the AOI region to obtain the AOI mapping table;

[0166] Calculate thermodynamic characteristic scalars based on the AOI mapping table;

[0167] Based on the AOI mapping table, the attention entropy value is calculated using the following formula:

[0168] H = -∑[P(i)×log2(P(i))]

[0169] Where H is the attention entropy value and P(i) is the AOI fixation probability;

[0170] Based on preset rules, consecutive annotation points in the effective gaze point sequence are merged to obtain the effective attention duration; and the effective attention duration, attention entropy value and thermal feature scalar are concatenated to obtain the visual feature vector.

[0171] Furthermore, feature module 801 is also used for:

[0172] Based on the AOI mapping table, the proportion of dominant AOIs is calculated using the following formula:

[0173]

[0174] Among them, DAR is the dominant AOI percentage, T maxAOI T represents the cumulative fixation time of the single AOI region that receives the most fixations. allAOI This represents the total cumulative fixation time across all AOI regions.

[0175] Based on the AOI mapping table, the long-term fixation percentage is calculated using the following formula:

[0176]

[0177] Where LFP is the percentage of long-term fixation, N long N represents the number of long-term fixations. total ∑dur is the total number of all valid fixations. long The cumulative duration of all long fixations, ∑dur all The total cumulative duration for all fixations;

[0178] The proportion of dominant AOI and the proportion of long-term gaze are combined into a thermal characteristic scalar.

[0179] Furthermore, feature module 801 is also used for:

[0180] Speech activity detection is performed on the speech data to obtain a time-stamped sequence;

[0181] The response delay is calculated based on the time-stamped sequence to obtain the response delay value;

[0182] MFCC features are extracted from non-user speech data to obtain MFCC feature vectors, and statistical features are extracted based on the MFCC feature vectors to obtain statistical feature vectors; the MFCC feature matrix is ​​used to characterize the hearing impairments of users in dialogue tasks.

[0183] Based on a pre-trained noise classification model, the noise resistance index is calculated for speech data using the following formula:

[0184] RI=e -0.1×max(SNR-15,0)

[0185] Wherein, RI is the noise immunity index and SNR is the signal-to-noise ratio;

[0186] The auditory comprehension features are obtained by concatenating the response delay value, statistical feature vector, and noise resistance index.

[0187] Furthermore, feature module 801 is also used for:

[0188] Based on a preset speed threshold rule, the motion data is segmented into a sequence of valid gesture segments; the sequence of valid gesture segments is a time series, used to represent the start time and end time of the motion.

[0189] Based on the effective gesture segment sequence, the maximum range of motion for each segment is calculated to obtain the motion amplitude vector;

[0190] Based on the effective gesture segment sequence, the peak point of motion intensity is located to obtain the peak time series;

[0191] The synchronization index of peak time series and keyword time points is calculated using the following formula:

[0192]

[0193] Among them, Sync k Let Δt be the synchronization index of the k-th keyword and action. k λ represents the time delay between the k-th keyword and the action, λ is the weighting coefficient, and the keyword time point is the time point in the speech data extracted from the speech feature vector where the keyword appears.

[0194] The gesture feature vector is obtained by concatenating the motion amplitude vector and the synchronization index.

[0195] Furthermore, the standardized module 802 is also used for:

[0196] The features in the feature vector group are classified to obtain the feature types; and the features in the feature vector group are normalized based on the feature types to obtain normalized features.

[0197] Based on cultural background labels, the normalized feature vector is corrected to obtain cultural calibration features; cultural background labels are identifiers used to represent the user's cultural background.

[0198] Based on style features and spoken dialogue tasks, weights are assigned to cultural calibration features to obtain weighted features; style features are preset spoken communication style types corresponding to the users.

[0199] Outlier smoothing is performed on the weighted features to obtain the index matrix.

[0200] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the multimodal adaptive English speaking training method as described above.

[0201] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0202] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0203] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A multimodal adaptive English oral training method, characterized in that, The method includes: Acquire multimodal data; and extract features corresponding to spoken language ability from the multimodal data to obtain a feature vector group; the multimodal data consists of eye-tracking data, speech data, and motion data perceived by various sensors when the user performs a spoken dialogue task; The feature vector group is standardized to obtain the index matrix; The index matrix is ​​input into a pre-trained attention mechanism model to obtain a profile of the user's spoken language ability. Using the spoken language proficiency profile as an index, corresponding teaching resources in a preset training resource library are matched to obtain modular training resources; the modular training resources are courses corresponding to the user's ability deficiencies. Based on the modular training resources, a training plan for the corresponding user is generated.

2. The method according to claim 1, characterized in that, The step of extracting features corresponding to spoken language ability from the multimodal data to obtain a feature vector set includes: Based on the eye-tracking data, visual attention features are extracted to obtain a visual feature vector; Based on the speech data, auditory comprehension features are extracted to obtain auditory feature vectors; and spoken speech features are extracted to obtain speech feature vectors. Based on the action data, gesture interaction features are extracted to obtain a gesture feature vector; The visual feature vector, the auditory feature vector, the speech feature vector, and the gesture feature vector are concatenated to obtain the feature vector group.

3. The method according to claim 2, characterized in that, The step of extracting visual attention features based on the eye-tracking data to obtain a visual feature vector includes: Data points in the eye movement data whose fixation duration is less than a preset threshold are removed to obtain an effective fixation point sequence; The valid annotation point sequence is mapped to the AOI region to obtain the AOI mapping table; Calculate the thermodynamic characteristic scalar based on the AOI mapping table; Based on the AOI mapping table, the attention entropy value is calculated using the following formula: H = -∑[P(i)×log2(P(i))] Where H is the attention entropy value and P(i) is the AOI fixation probability; Based on preset rules, consecutive annotation points in the effective gaze point sequence are merged to obtain the effective attention duration; and the effective attention duration, the attention entropy value, and the thermal feature scalar are concatenated to obtain the visual feature vector.

4. The method according to claim 3, characterized in that, The calculation of thermal characteristic scalars based on the AOI mapping table includes: Based on the AOI mapping table, the proportion of dominant AOIs is calculated using the following formula: Among them, DAR is the dominant AOI percentage, T maxAOI T represents the cumulative fixation time of the single AOI region that receives the most fixation. allAOI This represents the total cumulative fixation time across all AOI regions. Based on the AOI mapping table, the long-term fixation percentage is calculated using the following formula: Where LFP is the percentage of long-term fixation, N long N represents the number of long-term fixations. total ∑dur is the total number of all valid fixations. long The cumulative duration of all long fixations, ∑dur all The total cumulative duration for all fixations; The dominant AOI ratio and the long-term gaze ratio are concatenated to form the thermal feature scalar.

5. The method according to claim 2, characterized in that, The extraction of auditory comprehension features based on the speech data includes: Speech activity detection is performed on the speech data to obtain a time-stamped sequence; The response delay is calculated based on the time-stamped sequence to obtain the response delay value; MFCC features are extracted from the speech data of non-users in the speech data to obtain MFCC feature vectors, and statistical features are extracted based on the MFCC feature vectors to obtain statistical feature vectors; the MFCC feature matrix is ​​used to characterize the hearing impairments of the user in the dialogue task. Based on the pre-trained noise classification model, the noise resistance index is calculated for the speech data using the following formula: RI = e -0.1×max(SNR-15,0) Wherein, RI is the noise immunity index and SNR is the signal-to-noise ratio; The auditory understanding features are obtained by concatenating the response delay value, the statistical feature vector, and the noise resistance index.

6. The method according to claim 2, characterized in that, The step of extracting gesture interaction features based on the action data to obtain a gesture feature vector includes: Based on a preset speed threshold rule, the motion data is segmented into a sequence of valid gesture segments; the sequence of valid gesture segments is a time series, used to characterize the start time and end time of the motion. Based on the effective gesture segment sequence, the maximum range of motion for each segment is calculated to obtain the motion amplitude vector; Based on the effective gesture segment sequence, the peak point of motion intensity is located to obtain the peak time sequence; The synchronization index of the peak time series and keyword time points is calculated using the following formula: Among them, Sync k Let Δt be the synchronization index of the k-th keyword and action. k The time delay between the k-th keyword and the action is λ, where λ is the weighting coefficient, and the keyword time point is the time point in the speech data extracted from the speech feature vector where the keyword appears. The gesture feature vector is obtained by concatenating the motion amplitude vector and the synchronization index.

7. The method according to claim 1, characterized in that, The standardization process of the feature vector group to obtain the index matrix includes: The features in the feature vector group are classified to obtain feature types; and the features in the feature vector group are normalized based on the feature types to obtain normalized features. Based on the cultural background label, the normalized feature vector is modified to obtain the cultural calibration feature; the cultural background label is an identifier used to characterize the user's cultural background. Based on style features and the spoken dialogue task, weights are assigned to the cultural calibration features to obtain weighted features; the style features are preset spoken communication style types corresponding to the user. The weighted features are subjected to outlier smoothing to obtain the index matrix.

8. A multimodal adaptive English oral training system, characterized in that, The system includes: The feature module is used to acquire multimodal data and extract features corresponding to spoken language ability from the multimodal data to obtain a feature vector group; the multimodal data is eye-tracking data, speech data and motion data perceived by various sensors when the user performs a spoken dialogue task; The standardization module is used to standardize the feature vector group to obtain the index matrix; The capability module is used to input the indicator matrix into a pre-trained attention mechanism model to obtain a profile of the user's oral communication ability. The resource module is used to match the corresponding teaching resources in the preset training resource library with the oral proficiency profile as an index to obtain modular training resources; the modular training resources are courses corresponding to the user's ability deficiencies. The training module is used to generate a training plan for the corresponding user based on the modular training resources.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images