An immersive language interactive teaching system driven by multi-modal input

Abstract

The application relates to the technical field of intelligent education, in particular to a multi-modal input driven immersive language interaction teaching system, which comprises an acoustic monitoring module for analyzing user audio and generating an acoustic blocking timestamp; a motion tracking module for tracking a user's hand and line of sight and calculating a speed vector and an acceleration vector of the hand, and marking the first frame with a sudden change as a motion distortion inflection point; a logical truncation module for triggering a truncation instruction; an intention reconstruction module for generating a gesture ray and a line of sight ray in response to the truncation instruction, establishing an intention anchor point based on the gesture ray and the line of sight ray, and binding a corresponding virtual object as a target interaction intention; and a teaching feedback module for driving a virtual character to output guiding feedback for the virtual object based on the target interaction intention and vocabulary search blocking. The application truncates the motion distortion caused by vocabulary blocking through cross-modal time sequence constraint, and reconstructs a real interaction intention based on spatial geometric intersection.

Description

Technical Field

[0001] This application relates to the technical field of intelligent education, and in particular to a multimodal input-driven immersive language interactive teaching system. Background Technology

[0002] With the evolution of artificial intelligence and immersive spatial interaction technologies, intelligent education is gradually shifting from a single audiovisual and textual presentation to a multi-channel human-computer collaborative training model. In immersive language teaching, especially in advanced oral interaction scenarios, to comprehensively train learners' on-the-spot responsiveness and language organization skills, systems typically create highly realistic communicative contexts, such as simulating an all-English exhibition hall tour or cross-cultural business negotiations. In this environment, learners' interactive behavior exhibits significant cognitive-motor unity characteristics, that is, while performing physical gestures in a three-dimensional virtual space, they simultaneously output the corresponding target language speech. Multimodal input-driven interactive systems capture these physical and acoustic features in real time, and through cross-modal semantic understanding and alignment, drive virtual interactive objects to provide natural responses that conform to specific contexts.

[0003] To address the demand for multi-channel integrated human-computer interaction, the industry has conducted numerous technological explorations. For example, invention patent CN111665941B discloses a multimodal semantic fusion human-computer interaction system and method for virtual experiments. This technical framework relies on gestures and voice to carry out collaborative operations. Specifically, it establishes a relative time window based on voice triggering and performs relevance constraints and intent determination on continuous gesture frames falling within this time interval. Internally, it also introduces intent reasoning logic based on finite state automata, which can effectively identify active and passive objects by combining specific scene knowledge, and even actively switch perspectives to reduce interaction difficulty when faced with physical obstacles such as virtual objects occluding each other. This full data input mode based on timeline alignment and state machine reasoning effectively ensures user interaction efficiency and immersive experience in scenarios with fixed operation steps and strong logical norms, such as virtual chemistry experiments.

[0004] When the aforementioned multimodal fusion logic based on full spatiotemporal mapping is directly applied to immersive oral language teaching scenarios with high cognitive load, it often faces the dilemma of underlying technology failure. In real-world cross-language interaction, non-native language learners are highly susceptible to sudden vocabulary retrieval blockage when attempting to recall an unfamiliar word midway through a spatial instruction. In this microscopic instant, the overload of the brain's language center rapidly deprives cognitive resources originally intended for continuous motor control, causing the normally smooth extension of the instruction gesture to undergo a trajectory distortion that violates the laws of communicative physics. For example, the arm abruptly generates an inward reverse velocity and unconsciously retracts into anxiety-induced compensatory actions such as touching the chin or rubbing hands. Existing multimodal systems overly rely on WYSIWYG data hard mapping logic and continuous temporal tolerance windows. Their core flaw lies in the lack of ability to distinguish between normal interactive actions and physical trajectory distortions induced by cognitive impairment. In such specific situations, the underlying visual tracking device indiscriminately transmits a large amount of invalid and anxiety-inducing limb noise generated after the action inflection point as genuine interactive commands to the system for forced semantic alignment. This processing mode directly leads to the system completely losing sight of the target that the learner was actually trying to point to before the speech block occurred. This not only causes the virtual character to output absurd feedback that is completely out of the current interactive context, but also breaks the logic of the entire dialogue practice, making it impossible to provide smooth communicative support when the learner encounters a bottleneck of cognitive blockage. Summary of the Invention

[0005] In order to truncate action distortions caused by lexical blocking through cross-modal temporal constraints and reconstruct authentic interactive intentions based on spatial geometric intersection, this application provides a multimodal input-driven immersive language interaction teaching system.

[0006] This application provides a multimodal input-driven immersive language interactive teaching system, which adopts the following technical solution: A multimodal input-driven immersive language interactive teaching system, comprising: The acoustic monitoring module is used to analyze user audio. When the duration of silence or non-lexical vowels exceeds a preset duration threshold, it determines that a word retrieval block has occurred and generates an acoustic blockage timestamp. The motion tracking module is used to track the user's hand and gaze. It stores the collected trajectory data of the hand and gaze data in a cache queue of preset duration, and calculates the velocity vector of the hand based on a preset time step. When the displacement distance of the hand between adjacent time steps is greater than a preset noise tolerance, and the direction of the velocity vector of the hand undergoes a sudden change greater than a preset angle threshold, and the velocity vector of the hand after the change points to the user, the first frame in which the change occurs is marked as the motion distortion inflection point. The logic truncation module is used to trigger a truncation command and block the hand trajectory data after the motion distortion inflection point in the cache queue when the motion distortion inflection point falls within a preset time window based on the acoustic blocking timestamp. The intent reconstruction module is used to respond to the truncation command, extract the hand pointing direction and gaze direction of the frame before the motion distortion inflection point, generate gesture ray and gaze ray respectively, establish intent anchor point based on the gesture ray and gaze ray, and bind the corresponding virtual item as the target interaction intent according to the spatial association relationship between the intent anchor point and the virtual item in the system. The teaching feedback module is used to drive the virtual character in the system to output guiding feedback for the virtual item based on the target interaction intent and the vocabulary retrieval block.

[0007] Optionally, the acoustic monitoring module parses the user audio, including: The duration is resolved into the absolute duration of the silence and the energy duration of the non-lexical vowel, respectively. The preset duration thresholds are defined as the physiological threshold for ventilation and the tolerance for normal speech rate, respectively. When the absolute duration exceeds the physiological threshold for ventilation, or the energy duration exceeds the normal speech rate tolerance, the word retrieval block is determined to have occurred, and a time-sensitive acoustic blockage timestamp is generated.

[0008] Optionally, the motion tracking module is configured as follows: Extract the user's head spatial coordinates, shoulder spatial coordinates, and lower chest spatial coordinates to construct a three-dimensional physical bounding box. Use the head spatial coordinates as the top reference benchmark and the three-dimensional physical bounding box as the boundary benchmark for determining that the velocity vector points to the user. Extract the displacement distance of the hand between adjacent time steps. When the displacement distance is greater than the preset noise tolerance, calculate the spatial angle formed by the velocity vectors between adjacent time steps. When the spatial angle is greater than the preset angle threshold, it is determined that the direction of the velocity vector has undergone the abrupt change; Extract the spatial coordinates of the hand when the mutation occurs, and construct a reference direction vector pointing from the spatial coordinates of the hand to the center region of the three-dimensional physical bounding box; Calculate the angle between the velocity vector of the hand after the mutation and the reference direction vector. When the angle is less than a preset judgment angle, determine that the velocity vector of the hand after the mutation points to the user.

[0009] Optionally, the logic truncation module is configured as follows: Based on the acoustic blocking timestamp and the duration, a time interval with time tolerance before and after is constructed as the preset time window; Extract the occurrence time of the motion distortion inflection point and cross-compare the occurrence time with the preset time window; If the occurrence time does not fall within the preset time window, the mutation is determined to be unrelated to the word retrieval blockade, and the truncation instruction is skipped.

[0010] Optionally, the intent reconstruction module is configured as follows: Determine whether the gesture ray and the line of sight ray form skew lines, intersect, or are parallel; If the skew lines are formed, the coordinates of the two feet of the common perpendicular segment of the gesture ray and the line of sight ray are solved; for any foot of the perpendicular, if it falls on the reverse extension of the corresponding ray, the starting point of the corresponding ray is extracted as the coordinates of the endpoint on that side, otherwise the coordinates of the foot of the perpendicular are extracted as the coordinates of the endpoint on that side; and the midpoint of the line segment formed by the two endpoint coordinates is calculated as the intention anchor point. If the gesture ray intersects the gaze ray, then for any ray, determine whether the intersection point falls on its reverse extension line. If so, extract the starting point of the ray as the coordinate of the endpoint on that side; otherwise, extract the intersection point as the coordinate of the endpoint on that side. Finally, calculate the midpoint of the line segment formed by the coordinates of the two endpoints as the intention anchor point, or directly extract the intersection point as the intention anchor point when neither intersection point falls on the reverse extension line. If the gesture ray is parallel to the gaze ray, the collision intersection point generated by the gaze ray within the system is extracted as the intention anchor point; if no collision intersection point is generated, a point at a preset depth distance is intercepted on the gaze ray as the intention anchor point.

[0011] Optionally, the intent reconstruction module is configured as follows: Acquire candidate virtual items within a preset distance range centered on the intent anchor point; When the number of candidate virtual items is equal to one, the candidate virtual item is bound to the target interaction intent; when the number of candidate virtual items is zero, the current running state is maintained. When the number of candidate virtual items is greater than one, extract the user's historical interaction records in the system, and assign target weights to each candidate virtual item based on the historical interaction records; When there is only one candidate virtual item with the highest target weight, bind that candidate virtual item as the target interaction intent. When there are multiple candidate virtual items with the highest target weight, the virtual characters in the system are driven to simultaneously highlight each candidate virtual item with the highest target weight and output an inquiry command. The target interaction intent is determined based on the input of the inquiry command.

[0012] Optionally, the intent reconstruction module is further configured to: Extract the input in response to the inquiry command, restore the acquisition flow of the hand trajectory data, and determine the input as feedback voice or supplementary gesture trajectory; If the input is the feedback voice, then the directional semantics of the feedback voice are parsed, and the directional semantics are compared with the spatial coordinates of the candidate virtual item to determine the target interaction intent; If the input is the supplementary gesture trajectory, then a supplementary gesture ray is generated based on the supplementary gesture trajectory, and the candidate virtual item that intersects with the supplementary gesture ray is established as the target interaction intent.

[0013] Optionally, the teaching feedback module is configured as follows: The hand trajectory data blocked by the truncation command is isolated, and the word retrieval blocking and the target interaction intent are established as collaborative triggering conditions; In response to the collaborative triggering condition, the virtual character is driven to direct its gaze toward the virtual item; Using a pre-defined target teaching language, vocabulary guidance for the virtual item is output as guidance feedback.

[0014] Optionally, a cross-modal alignment module is also included, the cross-modal alignment module being configured as follows: Obtain the target interaction intent and the vocabulary retrieval block; The hand trajectory data blocked by the truncation instruction is isolated, and the target interaction intent and the vocabulary retrieval block are established as the basic input data for performing cross-modal feature alignment; The basic input data is subjected to cross-modal feature alignment to generate the vocabulary guidance, and the vocabulary guidance is transmitted to the teaching feedback module.

[0015] Optionally, the cross-modal alignment module is configured as follows: Analyze the target interaction intent and extract the attribute features of the virtual item; The vocabulary retrieval block is mapped to a status signal that triggers active guidance from the system. In response to the status signal, a preset scaffolding guide template is retrieved; The attribute features are embedded into the scaffolding guidance template to generate the vocabulary guidance.

[0016] In summary, this application includes the following beneficial technical effects: 1. This application uses a cross-modal temporal constraint mechanism between the acoustic monitoring module and the motion tracking module to accurately identify motion distortion caused by word retrieval blockage. When the inflection point of motion distortion falls into the acoustic blockage time window, a truncation command is triggered to block invalid hand trajectory data. This effectively blocks the interference of limb noise caused by cognitive overload on the determination of interaction intent from the source. It completely solves the core technical problem that existing systems cannot distinguish between normal interaction actions and trajectory distortion induced by cognitive impairment, and that invalid limb noise is included in the semantic alignment process indiscriminately, resulting in the loss of the real interaction target.

[0017] 2. This application uses an intent reconstruction module to extract the hand pointing direction and gaze direction of the frame before the motion distortion inflection point after the truncation command is triggered. The intent anchor point is established based on the spatial geometric intersection of the gesture ray and the gaze ray. Combined with the spatial relationship of virtual objects, the target interaction intent is accurately bound. This achieves the restoration of the user's real interaction direction using only uncontaminated pure data without relying on generative models to predict incomplete trajectories. This significantly reduces the system's computing power requirements and achieves microsecond-level response speed, ensuring the low latency requirements of immersive interaction.

[0018] 3. This application establishes vocabulary retrieval blockage and target interaction intent as collaborative triggering conditions through the teaching feedback module, driving the virtual character to output appropriate vocabulary guidance for the target virtual item. At the same time, combined with the cross-modal alignment module, the vocabulary retrieval blockage is mapped into an active guidance status signal and embedded into the scaffolding guidance template, turning learners' speech errors into precise teaching guidance opportunities. This effectively avoids the problem of virtual character feedback being out of context, and achieves a dual improvement in interaction fluency and language teaching adaptability. Attached Figure Description

[0019] Figure 1 This is a structural block diagram of the interactive teaching system according to an embodiment of this application; Figure 2 This is a timing control diagram for acoustic feature analysis and vocabulary retrieval blocking event determination in an embodiment of this application; Figure 3 This is a spatial geometric diagram illustrating the determination of the three-dimensional physical bounding box and motion distortion vector in an embodiment of this application. Detailed Implementation

[0020] The following combination Figures 1-3 This application will be described in further detail.

[0021] This application discloses a multimodal input-driven immersive language interaction teaching system, applicable to head-mounted VR display devices. The device integrates a dual-channel built-in microphone array, an inside-out spatial positioning camera, and an infrared eye-tracking module. The device's spatial positioning accuracy is better than 2mm, the audio sampling rate is 48kHz, the eye-tracking sampling frequency is no less than 90Hz, and the hand skeleton tracking sampling frequency is no less than 60Hz. The real physical space and virtual three-dimensional space coordinate systems are calibrated using a 1:1 mapping, with the coordinate unit being meters (m). In this embodiment, the key points of the user's head are taken as the three-dimensional spatial coordinates of the center of the eyebrows, the key points of the shoulders are taken as the three-dimensional spatial coordinates of the left and right acromions, and the key points of the lower end of the chest cavity are taken as the three-dimensional spatial coordinates of the user's xiphoid process. The recognition of all human body key points is achieved through the device's built-in human skeleton tracking algorithm.

[0022] Existing multimodal interaction systems designed for immersive scenarios generally employ a fusion logic based on full spatiotemporal mapping. A typical solution, such as the invention patent with announcement number CN111665941B, relies on gestures and voice to construct a collaborative operation system. By establishing a relative time window based on voice triggering, it performs relevance constraints and intent determination on continuous gesture frames falling within this time interval. This solution can ensure interaction efficiency in virtual experimental scenarios with fixed operation steps, but when directly applied to immersive language teaching scenarios accompanied by high cognitive load, it has inherent defects that cannot be overcome. When non-native language learners perform limb gestures in a three-dimensional virtual space, they need to simultaneously output non-native language target language speech. They are prone to vocabulary retrieval interruptions during the execution of the action due to the need to retrieve unfamiliar words. At this time, the cognitive overload of the brain's language center will deprive the cognitive resources required for motor nerve control, causing the originally smooth gestures to undergo trajectory distortions that violate the laws of communicative physics, resulting in meaningless anxiety-compensating actions such as arm retraction, touching the chin, and rubbing hands. Existing systems cannot distinguish between normal interactive actions and trajectory distortions induced by cognitive impairments. They indiscriminately include invalid bodily noise after the distortion inflection point into the multimodal semantic alignment process, ultimately causing the system to lose the user's true interaction goals. The virtual character outputs feedback completely out of context, completely disrupting the logical loop of dialogue practice and failing to provide effective teaching support when learners encounter cognitive bottlenecks. This embodiment addresses the above deficiencies by providing a complete and implementable technical solution. It truncates action distortions caused by lexical blockages through cross-modal temporal constraints, reconstructs the true interaction intent based on spatial geometric intersection, and ultimately achieves accurate interaction and guided feedback adapted to language teaching scenarios.

[0023] like Figure 1As shown, the overall system architecture includes core modules such as acoustic monitoring, motion tracking, logic truncation, intent reconstruction, cross-modal feature alignment, and instructional guidance feedback. These modules work in tandem, visually demonstrating the complete closed-loop data flow from dual-channel data (audio and motion) acquisition, cross-validation and truncation of abnormal trajectories, to the final generation of scaffolding instructional feedback. This establishes the underlying logical framework for the accurate fusion and processing of multimodal information in this solution.

[0024] S1 Acoustic Monitoring and Vocabulary Retrieval Blocking Event Anchoring The system uses the device's built-in microphone array to collect the user's spoken audio stream data in real time and performs preprocessing operations on the collected audio stream. Preprocessing operations include pre-emphasis, framing, and windowing. The frame length for preprocessing is set to 20ms, and the frame shift is set to 10ms. These parameters match the short-term stationarity characteristics of human speech signals, ensuring the temporal accuracy of audio timing analysis and avoiding the omission of short-term speech feature changes.

[0025] The system then performs frame-by-frame energy and spectral feature analysis on the preprocessed audio stream. Silence frames are determined by the signal energy of a single audio frame; if the signal energy of a single frame is lower than a preset silence energy threshold, the system marks that frame as a silence frame. The silence energy threshold is set to 1.2 times the average energy of the ambient noise collected by the device within 1 second of no valid voice input in the current operating environment. This value can adapt to the differences in ambient noise in different usage scenarios, effectively distinguishing between ambient noise and valid voice signals.

[0026] The system accumulates the duration of consecutive silence frames to obtain the absolute duration of the silence segment. Non-lexical vowel frames are identified by analyzing the spectral characteristics of individual audio frames. If a single audio frame exhibits a stable fundamental frequency peak within the 80Hz-450Hz range and lacks spectral spikes corresponding to high-frequency consonants, the system marks that frame as a non-lexical vowel frame. This fundamental frequency range matches the typical distribution interval of human speech fundamental frequencies, enabling accurate identification of meaningless filler sounds.

[0027] The system accumulates the duration of consecutive non-lexical vowel frames to obtain the energy duration of non-lexical vowels. Non-lexical vowels specifically refer to drawn-out or filler sounds in spoken expression that lack clear semantic meaning and do not carry effective lexical semantic information. The system presets two types of duration judgment thresholds: a physiological breathing threshold and a normal speech rate tolerance. The physiological breathing threshold is set at 800ms. This value is determined based on the characteristics of breathing behavior in normal human spoken expression. In normal spoken communication, the pause duration required for physiological breathing is usually 200ms to 500ms. A silent pause exceeding 800ms is not considered a normal physiological breathing behavior and can correspond to an unnatural interruption in spoken expression. The normal speech rate tolerance is set at 600ms. This value is determined based on the characteristics of normal spoken output by second language learners. In the spoken expression of non-native language learners, the duration of meaningless filler sounds usually does not exceed 300ms. Continuous filler sounds exceeding 600ms can correspond to retrieval pauses in spoken expression.

[0028] The system then executes the logic for determining word retrieval blocking. When the absolute duration of a silent segment exceeds the physiological threshold for breathing, or the energy duration of a non-lexical vowel exceeds the tolerance of normal speech rate, the system determines that word retrieval blocking has occurred. This logic does not rely on complex semantic recognition engines and large-vocabulary speech models; it can identify cognitive blocking events solely through underlying acoustic features. This effectively avoids the impact of semantic recognition delays and errors on the judgment results, adapting to the needs of users with different accents and language proficiency levels in immersive language teaching scenarios.

[0029] Combination Figure 2 As can be seen, the system analyzes the absolute duration of silence and the duration of non-lexical vowel energy in parallel along the time axis, clearly presenting the judgment logic from the extraction of underlying raw audio features to the triggering of the "acoustic blocking timestamp". This time sequence diagram intuitively proves that this system does not need to rely on a high-latency large model semantic engine, but can quickly and accurately capture the learner's cognitive blocking moment by simply using physiological acoustic boundaries (the physiological threshold of breathing and the tolerance of normal speech rate), which greatly reduces the computational overhead and ensures the real-time response capability of immersive interaction.

[0030] Subsequently, an acoustic blocking timestamp is generated on the global timeline. This timestamp marks the start time of the vocabulary retrieval blocking event, serving as the unique anchor point for that event on the global timeline and possessing temporal accuracy synchronized with the audio stream. Furthermore, to prevent subsequent long-duration speech delays from breaking the time window calculation, the timestamp's lifespan is dynamically bound to the duration of the vocabulary retrieval blocking event. The system continuously maintains the timestamp's validity until the acoustic features determine the end of the blocking event. At the end of the event, the timestamp is automatically invalidated after a preset buffer period (e.g., 500ms), ensuring the integrity of the timeline record and preventing temporal confusion with subsequent blocking events.

[0031] S2 Kinematic Feature Tracking and Motion Distortion Inflection Point Calibration The system synchronizes the acquisition timeline of spatial motion data with the global timeline established in S1. First, all acquired spatial motion data (including hand trajectory data and line-of-sight data) are uniformly mapped to the global timeline benchmark established in S1 and pushed into a data buffer queue of preset duration in real time to cope with the objective time delay in acoustic feature determination and ensure that the timeline marking of acoustic events and motion events has a unified judgment and traceability standard.

[0032] The system uses the device's spatial positioning camera to track the user's hands, head, and shoulders in real time, while simultaneously collecting the user's gaze direction data through an eye-tracking module. To clearly identify the tracking subject, the system uses the user's index fingertip as the core key point for hand tracking, outputting its coordinate data in a 3D virtual space coordinate system in real time. The 3D virtual space coordinate system is mapped to the real physical space at a 1:1 ratio, with the coordinate unit being meters (m).

[0033] Considering the high sampling frequency (no less than 60Hz) of hand skeleton tracking and the extremely short time interval between adjacent physical frames, the system introduces a preset time step (e.g., 100ms, spanning approximately 6 sampling frames) to calculate kinematic parameters in order to accurately capture the macroscopic movement trend of the limbs and filter out minute hovering jitters. The system calculates the displacement based on the index fingertip coordinate data before and after this preset time step and derives the corresponding velocity vector. The system sets a preset noise tolerance of 5mm. It first extracts the hand displacement distance between adjacent time steps; only when this displacement distance is greater than the preset noise tolerance is the action within that time period considered to have effective physical directionality. This allows for subsequent direction change determination, effectively avoiding computational redundancy caused by a large number of invalid still frames.

[0034] When defining the decision boundary, the system extracts the spatial coordinates of the user's head, shoulders, and lower chest to construct a three-dimensional physical bounding box. This bounding box uses a cuboid structure, with the user's head spatial coordinates as the top reference, the line connecting the left and right acromions as the width direction (set to 1.2 times the acromion spacing), a front-to-back depth of 30cm, and extends downwards to the horizontal plane of the lower chest as the bottom boundary. This three-dimensional physical bounding box is used as the boundary reference for determining whether the velocity vector points towards the user. Next, provided the hand displacement distance is greater than the preset noise tolerance, the system calculates the spatial angle formed by the hand velocity vectors between adjacent time steps. The system's preset angle threshold is set to 120°. Due to physical inertia, this spatial angle is usually small for normal spatial pointing actions; only when cognitive blockage-induced anxiety-compensating actions occur (such as a sudden retraction of the hand to touch the chin) will a velocity direction change greater than 120° occur.

[0035] When the spatial angle formed by the hand velocity vectors between adjacent time steps is greater than a preset angle threshold, the system extracts the index fingertip coordinates at that moment and constructs a reference direction vector pointing from these coordinates to the center region of the 3D physical bounding box. Subsequently, the system calculates the angle between the abruptly changed hand velocity vector and this reference direction vector. If this angle is less than a preset judgment angle (e.g., an acute angle of 90°), it indicates that the abruptly changed hand movement trend is geometrically converging towards the user's torso. The system then determines that the abruptly changed hand velocity vector is pointing towards the user and explicitly marks the first frame where this abrupt change occurs as the motion distortion inflection point.

[0036] refer to Figure 3 The system uses the head, shoulders, and lower chest of the human body as reference coordinates to rigorously construct a three-dimensional physical bounding box and quantifies the spatial angle between the velocity vector and the reference direction vector when the hand experiences a sudden change in speed. This geometric determination model, which combines key points of the human skeleton, the trajectory of movement, and the angle between spatial vectors, intuitively and ingeniously transforms abstract "anxiety-induced compensatory actions (such as the subconscious retraction of the arm)" into objective and precisely calculable physical geometric boundaries. This fundamentally overcomes the technical bottleneck of traditional multimodal systems, which easily mistake such unconscious bodily noise for interactive intent.

[0037] This judgment logic based on the vector angle not only avoids the absolute misjudgment problem caused by the fingertip being inside the bounding box in the traditional ray intersection algorithm, but also more accurately quantifies the intention of the "retraction" action. Finally, the system will synchronously mark the occurrence time of this inflection point on the global timeline.

[0038] S3 cross-modal temporal constraints and invalid trajectory data truncation The system extracts the effective acoustic blocking timestamp generated by S1 (i.e., the start time of the blocking event), the duration of the blocking event, and the occurrence time corresponding to the motion distortion inflection point marked by S2 from the synchronized global timeline. Then, based on the acoustic blocking timestamp and the event duration, the system constructs a time interval with pre- and post-event tolerances as a preset time window. Specifically, to fully cover the entire neural delay conduction cycle from overload of the brain's language center to the occurrence of physical limb distortion, the system uses the acoustic blocking timestamp extended forward by 300ms as the start time of the window, and extends the end time of the blocking event (i.e., the start time plus the duration determined by acoustic measurement) forward by 300ms as the end time of the window.

[0039] Because S1 dynamically links the lifecycle of timestamps, this dynamic window construction mechanism not only makes up for the problem that fixed short windows cannot accommodate long-term pauses, but also ensures the smooth and continuous calculation of window endpoints before the long-term pause ends completely, ensuring that compensatory actions occurring during this period can be included.

[0040] The system then cross-references the occurrence time of the motion distortion inflection point with this preset time window. Thanks to the underlying data caching queue mechanism, the system can trace historical motion trajectories even when acoustic judgment lags. Once the occurrence time of the motion distortion inflection point falls within the preset time window, the system determines that the sudden change in action has a direct causal relationship with the blocking of word retrieval. At this point, the system immediately triggers a truncation command, completely blocking all hand trajectory data temporarily stored after the distortion inflection point in the data caching queue and suspending the subsequent flow of data in that channel. This prevents meaningless trajectory data caused by cognitive impairment from entering the semantic calculation stage from the source. Conversely, if the occurrence time of the motion distortion inflection point does not fall within the preset time window, the system considers the sudden change to be a normal interaction action adjustment, thus skipping the truncation command and maintaining the normal flow and distribution of the original trajectory data.

[0041] S4 Space Ray Intersection and Target Interaction Intent Reconstruction In response to the truncation command triggered by S3, the system retrieves hand and gaze data from the frame preceding the motion distortion inflection point from the data cache queue. This frame of data perfectly preserves the user's most authentic interaction intention just before the speech interruption occurs.

[0042] Based on this frame of data, the system extracts the orientation of key hand points (such as the vector connecting the center of the palm to the fingertips or the orientation of the gesture API) to accurately reconstruct the absolute physical direction, and generates a gesture ray that extends infinitely into the virtual space, starting from the coordinates of the index fingertip. At the same time, the system generates a gaze ray based on the coordinates of the midpoints of the pupils of both eyes and the real-time gaze direction.

[0043] Subsequently, the system determines the positional relationship between the gesture ray and the gaze ray in the three-dimensional virtual space, and calculates the intention anchor point according to different cases: If two rays form skew lines, the system will solve for the coordinates of the two feet of the common perpendicular segment. To avoid the geometric calculation blind spot caused by backward divergence, if either foot of the perpendicular falls on the backward extension line, the coordinates of the starting point of the corresponding ray are used to replace the foot of the perpendicular; then the midpoint of the line connecting the coordinates of the two endpoints is calculated as the intended anchor point.

[0044] If two rays intersect, the system will also check for their reverse extensions. If the intersection point falls on the reverse extension of a ray, the coordinates of the ray's starting point are extracted as the endpoint of that side. Finally, the midpoint is determined using the coordinates of both endpoints as the anchor point, or, if the intersection point is perfectly normal, the intersection point itself can be used directly as the intended anchor point.

[0045] If the gesture ray is parallel to the gaze ray, the system prioritizes extracting the first intersection point between the gaze ray and an interactive object or static model in the virtual scene as the intent anchor point. To prevent the user from looking into empty space and causing computational deadlock, if no collision intersection point is generated, the system directly extracts a spatial point on the gaze ray at a preset depth from the starting point (e.g., 2.0m, conforming to the conventional teaching interaction viewing distance) as the intent anchor point, ensuring a smooth closed loop of the entire spatial calculation logic.

[0046] After establishing the intent anchor point, the system retrieves candidate virtual items within a preset 15cm radius of the anchor point. If there is only one candidate item, it is directly bound to the target interaction intent; if there are no candidate items, the current state is maintained. If multiple candidate items exist, the system uses the user's historical interaction records to assign target weights (items that have been interacted with have increasing weights). When there is only one item with the highest weight, the binding is completed directly; if there are still multiple items, the system drives the virtual character to highlight them and output a query command.

[0047] Finally, users can provide feedback in response to the system's queries. At this point, the hand tracking channel resumes operation. If the user inputs directional speech (such as "the one on the left"), the system uses semantic parsing and combines it with the first-person perspective coordinate system to complete the item matching; if the user inputs supplementary gestures, the system uses the newly generated gesture ray to find the collision intersection point, thereby ultimately accurately locking onto the target interaction intent.

[0048] S5 Cross-modal Feature Alignment and Teaching Feedback Output The system features a separate cross-modal alignment module, forming a two-tiered processing architecture that is functionally integrated with the teaching feedback module. The cross-modal alignment module acquires the target interaction intent that is finally bound in S4, as well as the timing and status data of the vocabulary retrieval blocking events that are determined in S1.

[0049] The cross-modal alignment module isolates invalid hand trajectory data blocked by truncated instructions in S3, establishing the target interaction intent and vocabulary retrieval blocking event as the basic input data for performing cross-modal feature alignment. The cross-modal alignment module parses the target interaction intent and extracts the pre-stored attribute features of the corresponding virtual items. These attribute features include the item's standard vocabulary in the target teaching language, its semantic category, and related teaching content in the current teaching scenario. All attribute features are pre-stored in the system's teaching resource library and are bound one-to-one with interactive items in the virtual scene.

[0050] The cross-modal alignment module maps vocabulary retrieval blocking events into state signals that trigger proactive teaching guidance from the system. In response to these signals, the system retrieves pre-set scaffolding guidance templates. These templates, based on second language acquisition scaffolding teaching theory, are divided into three levels: vocabulary prompts, sentence structure guidance, and scenario-based questioning. The vocabulary prompt template provides standard vocabulary for the target item, the sentence structure guidance template contains commonly used expression frameworks for the target item, and the scenario-based questioning template contains guiding questions tailored to the current teaching scenario. The template retrieval rules correspond one-to-one with the user's pre-set language learning level and the current teaching scenario type: beginner level corresponds to the vocabulary prompt template, intermediate level to the sentence structure guidance template, and advanced level to the scenario-based questioning template.

[0051] The cross-modal alignment module extracts virtual item attribute features and embeds them into the retrieved scaffolding guidance template to generate vocabulary guidance content adapted to the current interaction scenario and user level. This generated vocabulary guidance content is then transmitted to the teaching feedback module. The teaching feedback module receives the vocabulary guidance content from the cross-modal alignment module, completely isolating invalid hand trajectory data blocked by truncated instructions in S3. It establishes the valid state of the vocabulary retrieval blocking event and the valid binding state of the target interaction intent as collaborative triggering conditions. Subsequent feedback output operations are only executed when both sets of data are simultaneously complete and valid. This approach fundamentally eliminates the interference of invalid and distorted data on teaching feedback, solving the core problem of virtual character feedback easily becoming detached from the current interaction context under the existing technology's full-data input mode.

[0052] The teaching feedback module responds to collaborative trigger conditions, first driving the virtual character within the virtual scene to perform collaborative physical actions. The virtual character adjusts its gaze in real time, smoothly directing it towards the virtual object bound to the target interaction intent, simultaneously adapting its upper body orientation to match the virtual object's spatial location, thus replicating non-verbal collaborative behavior in real interpersonal communication. The teaching feedback module then outputs adapted vocabulary guidance content through the virtual character's voice output channel, using preset target teaching language, providing immersive guidance feedback to address user vocabulary search obstacles.

[0053] This embodiment constructs a complete closed loop through five progressive execution steps, from blocking event identification, noise data truncation, true intent reconstruction, to adaptive teaching feedback. The temporal anchor points established in the acoustic monitoring stage provide a precise temporal benchmark for determining the causal relationship of motion distortion, avoiding misidentification of normal actions. The physical quantification rules established in the motion tracking stage provide an objective basis for judging motion distortion caused by cognitive overload, without relying on complex semantic recognition models. The cross-modal temporal constraint stage achieves source truncation of invalid noise data through cross-validation of data from two independent modalities, fundamentally avoiding interference from junk data on subsequent intent recognition and solving the inherent defects of the full-data input mode in existing technologies. The intent reconstruction stage adopts a purely geometric ray intersection method, which does not rely on generative models to predict incomplete trajectories. It can accurately reconstruct the user's true interaction intent using only uncontaminated valid data, significantly reducing the system's computational requirements while achieving microsecond-level response speeds, meeting the low-latency requirements of immersive interaction. The teaching feedback process uses only purified and valid data as input, transforming users' speech errors into precise teaching guidance triggers. This deeply aligns with the cognitive patterns of second language acquisition and effectively solves the problems of existing systems providing mechanical and context-detached feedback when users stumble, achieving a dual improvement in interactive fluency and teaching adaptability.

[0054] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A multimodal input-driven immersive language interactive teaching system, characterized in that, include: The acoustic monitoring module is used to analyze user audio. When the duration of silence or non-lexical vowels exceeds a preset duration threshold, it determines that a word retrieval block has occurred and generates an acoustic blockage timestamp. The motion tracking module is used to track the user's hand and gaze. It stores the collected trajectory data of the hand and gaze data in a cache queue of preset duration, and calculates the velocity vector of the hand based on a preset time step. When the displacement distance of the hand between adjacent time steps is greater than a preset noise tolerance, and the direction of the velocity vector of the hand undergoes a sudden change greater than a preset angle threshold, and the velocity vector of the hand after the change points to the user, the first frame in which the change occurs is marked as the motion distortion inflection point. The logic truncation module is used to trigger a truncation command and block the hand trajectory data after the motion distortion inflection point in the cache queue when the motion distortion inflection point falls within a preset time window based on the acoustic blocking timestamp. The intent reconstruction module is used to respond to the truncation command, extract the hand pointing direction and gaze direction of the frame before the motion distortion inflection point, generate gesture ray and gaze ray respectively, establish intent anchor point based on the gesture ray and gaze ray, and bind the corresponding virtual item as the target interaction intent according to the spatial association relationship between the intent anchor point and the virtual item in the system. The teaching feedback module is used to drive the virtual character in the system to output guiding feedback for the virtual item based on the target interaction intent and the vocabulary retrieval block.

2. The system according to claim 1, characterized in that, The acoustic monitoring module analyzes the user's audio, including: The duration is resolved into the absolute duration of the silence and the energy duration of the non-lexical vowel, respectively. The preset duration thresholds are defined as the physiological threshold for ventilation and the tolerance for normal speech rate, respectively. When the absolute duration exceeds the physiological threshold for ventilation, or the energy duration exceeds the normal speech rate tolerance, the word retrieval block is determined to have occurred, and a time-sensitive acoustic blockage timestamp is generated.

3. The system according to claim 1, characterized in that, The motion tracking module is configured as follows: Extract the user's head spatial coordinates, shoulder spatial coordinates, and lower chest spatial coordinates to construct a three-dimensional physical bounding box. Use the head spatial coordinates as the top reference benchmark and the three-dimensional physical bounding box as the boundary benchmark for determining that the velocity vector points to the user. Extract the displacement distance of the hand between adjacent time steps. When the displacement distance is greater than the preset noise tolerance, calculate the spatial angle formed by the velocity vectors between adjacent time steps. When the spatial angle is greater than the preset angle threshold, it is determined that the direction of the velocity vector has undergone the abrupt change; Extract the spatial coordinates of the hand when the mutation occurs, and construct a reference direction vector pointing from the spatial coordinates of the hand to the center region of the three-dimensional physical bounding box; Calculate the angle between the velocity vector of the hand after the mutation and the reference direction vector. When the angle is less than a preset judgment angle, determine that the velocity vector of the hand after the mutation points to the user.

4. The system according to claim 1, characterized in that, The logic truncation module is configured as follows: Based on the acoustic blocking timestamp and the duration, a time interval with time tolerance before and after is constructed as the preset time window; Extract the occurrence time of the motion distortion inflection point and cross-compare the occurrence time with the preset time window; If the occurrence time does not fall within the preset time window, the mutation is determined to be unrelated to the word retrieval blockade, and the truncation instruction is skipped.

5. The system according to claim 1, characterized in that, The intent reconstruction module is configured as follows: Determine whether the gesture ray and the line of sight ray form skew lines, intersect, or are parallel; If the skew lines are formed, the coordinates of the two feet of the common perpendicular segment of the gesture ray and the line of sight ray are solved; for any foot of the perpendicular, if it falls on the reverse extension of the corresponding ray, the starting point of the corresponding ray is extracted as the coordinates of the endpoint on that side, otherwise the coordinates of the foot of the perpendicular are extracted as the coordinates of the endpoint on that side; and the midpoint of the line segment formed by the two endpoint coordinates is calculated as the intention anchor point. If the gesture ray intersects the gaze ray, then for any ray, determine whether the intersection point falls on its reverse extension line. If so, extract the starting point of the ray as the coordinate of the endpoint on that side; otherwise, extract the intersection point as the coordinate of the endpoint on that side. Finally, calculate the midpoint of the line segment formed by the coordinates of the two endpoints as the intention anchor point, or directly extract the intersection point as the intention anchor point when neither intersection point falls on the reverse extension line. If the gesture ray is parallel to the gaze ray, then the collision intersection point generated by the gaze ray within the system is extracted as the intention anchor point; If no collision point is generated, a point at a preset depth distance is intercepted on the line of sight as the intended anchor point.

6. The system according to claim 5, characterized in that, The intent reconstruction module is configured as follows: Acquire candidate virtual items within a preset distance range centered on the intent anchor point; When the number of candidate virtual items is equal to one, the candidate virtual item is bound to the target interaction intent; When the number of candidate virtual items is zero, the current operating state is maintained; When the number of candidate virtual items is greater than one, extract the user's historical interaction records in the system, and assign target weights to each candidate virtual item based on the historical interaction records; When there is only one candidate virtual item with the highest target weight, bind that candidate virtual item as the target interaction intent. When there are multiple candidate virtual items with the highest target weight, the virtual characters in the system are driven to simultaneously highlight each candidate virtual item with the highest target weight and output an inquiry command. The target interaction intent is determined based on the input of the inquiry command.

7. The system according to claim 6, characterized in that, The intent reconstruction module is also configured to: Extract the input in response to the inquiry command, restore the acquisition flow of the hand trajectory data, and determine the input as feedback voice or supplementary gesture trajectory; If the input is the feedback voice, then the directional semantics of the feedback voice are parsed, and the directional semantics are compared with the spatial coordinates of the candidate virtual item to determine the target interaction intent; If the input is the supplementary gesture trajectory, then a supplementary gesture ray is generated based on the supplementary gesture trajectory, and the candidate virtual item that intersects with the supplementary gesture ray is established as the target interaction intent.

8. The system according to claim 1, characterized in that, The teaching feedback module is configured as follows: The hand trajectory data blocked by the truncation command is isolated, and the word retrieval blocking and the target interaction intent are established as collaborative triggering conditions; In response to the collaborative triggering condition, the virtual character is driven to direct its gaze toward the virtual item; Using a pre-defined target teaching language, vocabulary guidance for the virtual item is output as guidance feedback.

9. The system according to claim 8, characterized in that, It also includes a cross-modal alignment module, which is configured as follows: Obtain the target interaction intent and the vocabulary retrieval block; The hand trajectory data blocked by the truncation instruction is isolated, and the target interaction intent and the vocabulary retrieval block are established as the basic input data for performing cross-modal feature alignment; The basic input data is subjected to cross-modal feature alignment to generate the vocabulary guidance, and the vocabulary guidance is transmitted to the teaching feedback module.

10. The system according to claim 9, characterized in that, The cross-modal alignment module is configured as follows: Analyze the target interaction intent and extract the attribute features of the virtual item; The vocabulary retrieval block is mapped to a status signal that triggers active guidance from the system. In response to the status signal, a preset scaffolding guide template is retrieved; The attribute features are embedded into the scaffolding guidance template to generate the vocabulary guidance.

Images