Multifunctional english auxiliary learning device

Abstract

The application provides a multifunctional English auxiliary learning device, which comprises a hardware main body and an embedded software system; the hardware main body is integrated with an AI acceleration processor, a multi-modal perception module, a foldable support mechanism, an eye-protecting display and a directional sound emission unit, and the computing power is not less than 12 TOPS; the software system comprises five core modules, i.e., a multi-modal AI interaction engine, an intelligent pronunciation correction system, an AR scene-based learning module, an adaptive learning management system and a safe concentration mode; the application realizes phoneme-level pronunciation correction and visualized guidance of mouth shape through four-mode fusion interaction of voice, vision, gesture and touch; the application solves the problem of disconnection between learning and use by integrating English learning into a real scene through AR technology; and the application eliminates learning distraction through a physical concentration switch. The application breaks through the technical bottleneck of the existing English learning device, realizes major innovation in the dimensions of interaction mode, pronunciation correction, learning immersion and use experience, and significantly improves the English learning efficiency and effect.

Description

Technical Field

[0001] This invention relates to the field of intelligent educational equipment technology, specifically to a multifunctional English learning aid and control method based on multimodal AI interaction, AR scenario-based learning, and phoneme-level pronunciation correction. It is particularly suitable for English learning, oral correction, immersive scenario training, focused learning management, and personalized adaptive learning for teenagers. Background Technology

[0002] With the acceleration of globalization and the development of artificial intelligence technology, the importance of English as an international language is becoming increasingly prominent, leading to rapid growth in the English learning device market. Currently, various English learning aids are available, including reading pens, dictionary pens, learning tablets, and vocabulary memorization machines. While these devices meet learners' needs to some extent, they still exhibit the following significant technical shortcomings and deficiencies in practical applications: 1. The interaction mode is singular and lacks multimodal integration: Most existing English learning devices only support single interaction methods such as touch or voice, and fail to organically integrate multiple sensory channels such as visual recognition, voice interaction, gesture control, and tactile feedback. For example, the intelligent English auxiliary learning device disclosed in CN112991834A only provides display and basic writing functions, lacks real-time voice-visual linkage interaction, and cannot achieve a truly immersive learning experience. 2. Insufficient precision in pronunciation correction and lack of guidance on mouth shape: While current speech recognition technologies have achieved an accuracy rate of over 98%, most can only score at the sentence or word level, failing to achieve precise pronunciation correction at the phoneme level. More importantly, the vast majority of devices lack computer vision-based lip-reading recognition and correction guidance functions, making it difficult for learners to master the correct pronunciation mouth shape and tongue position, resulting in the long-standing problem of "mute English." 3. The learning scenarios are detached from reality and lack immersion: Traditional learning devices employ a passive learning model with pre-prepared content, which separates learning from real-life scenarios. The vocabulary and sentence patterns that learners acquire on the device cannot be effectively transferred to real-life communicative situations, resulting in a "disconnect between learning and application." There are few existing solutions that deeply integrate augmented reality (AR) technology with English learning. 4. The hardware design has many flaws. Display issues: Most devices use ordinary LCD screens, which can easily cause eye strain after prolonged use. Some low-priced products even use black and white screens, which seriously affect the learning experience and eye health. Angle adjustment limitations: The displays of existing devices are usually fixed or have a limited range of adjustment, which cannot adapt to the viewing angle requirements of different usage scenarios such as standing, sitting, and lying down; Insufficient privacy protection: The external speaker design disturbs others during listening practice, and there is a lack of directional sound production technology; Poor battery life: The high-performance processor and large screen result in short battery life, which cannot meet the needs of long-term offline learning.

[0003] 5. Easily distracted and lacking a focused learning mechanism: Most existing smart learning devices are based on the common Android or iOS systems, and come pre-installed or downloadable with a large number of entertainment apps, making learners easily distracted. Surveys show that over 60% of students switch to games or social media apps when using learning devices, severely impacting their learning efficiency.

[0004] 6. Insufficient personalized learning ability: Most devices adopt a "one-size-fits-all" content delivery model, failing to plan truly personalized and adaptive learning paths based on learners' actual levels, weaknesses, and learning habits. Memory science principles such as the Ebbinghaus forgetting curve have not been effectively integrated into learning algorithms. In summary, the core technological bottleneck of existing English learning aids lies in: It failed to break through the limitations of a single interaction mode and achieve deep integration of multimodal perception; Pronunciation correction remains at the level of surface scoring, lacking precise phoneme-level analysis and visual guidance on mouth shape; The learning content is disconnected from real-world scenarios, lacking an immersive learning experience based on AR (Augmented Reality). The hardware design failed to fully consider the ergonomics and specific learning scenarios; There is a lack of effective mechanisms to prevent distraction and to ensure focused learning. The lack of intelligence prevents truly personalized adaptive learning. At the same time, existing devices generally lack hardware-level focused control, offline multimodal learning, and cross-modal data linkage decision-making, making it difficult to meet the needs of efficient, safe, and immersive modern English learning.

[0005] Therefore, there is an urgent need to develop a new type of multifunctional English learning aid that comprehensively solves the aforementioned problems of existing technologies by organically combining multimodal artificial intelligence technology, AR augmented reality technology, and user-friendly hardware design. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a multifunctional English learning aid device. Through the innovative integration of a multimodal AI interaction engine, an intelligent real-time pronunciation correction system, an AR-based scenario-based learning module, and an integrated, user-friendly hardware design, the following technical objectives are achieved: Breaking through the limitations of interaction modes: Constructing a four-modal fusion interaction system of voice, vision, gesture and touch to achieve natural and smooth human-computer interaction; Improve pronunciation correction accuracy: Achieve phoneme-level pronunciation analysis and real-time mouth shape correction guidance based on computer vision; Enhance learning immersion: Integrate English learning into real-life scenarios through AR (Augmented Reality) technology; Optimize the hardware user experience: Design an integrated hardware system that is foldable, multi-angle adjustable, eye-friendly, and features directional sound. Ensure focused learning: Establish a physically isolated focused learning model to eliminate distractions at the source; Achieving Personalized Learning: Constructing Adaptive Learning Paths Based on Big Data and Learning Science Principles The technical solution adopted by the present invention to achieve the technical objective is: a multifunctional English auxiliary learning device, comprising: a hardware main body and an embedded software system, wherein the hardware main body includes a device host 1, a foldable support mechanism 2, a multimodal perception module 3, a human-computer interaction module 4, and a power management module 5; The device host 1 integrates a main controller 11, an AI acceleration processor 12, a memory 13, and a wireless communication module 14. The AI ​​acceleration processor 12 adopts an edge-side independent NPU neural network processor with a computing power of no less than 12 TOPS, supporting offline AI computing. The multimodal perception module 3 includes a high-definition binocular camera 31 and a four-microphone array 32. The human-computer interaction module 4 includes an eye-protection display screen 41 and a directional sound unit 42; The embedded software system includes a multimodal AI interaction engine 100, an intelligent pronunciation correction system 200, and an AR scenario-based learning module 300. The multimodal AI interaction engine 100 supports simultaneous input through four channels: voice, vision, gesture, and touch. The intelligent pronunciation correction system 200 works in conjunction with the multimodal perception module 3 to achieve real-time pronunciation correction at the phoneme level.

[0007] Furthermore: the multimodal AI interaction engine 100 includes a speech recognition submodule 101, a computer vision submodule 102, a multimodal fusion submodule 103, and a natural language processing submodule 104; the speech recognition submodule 101 is based on an end-to-end Transformer model, supports mixed Chinese and English recognition, and has a recognition accuracy of no less than 98%; the computer vision submodule 102 integrates face detection, lip recognition, gesture recognition, and object recognition algorithms; the multimodal fusion submodule 103 uses an attention mechanism to achieve dynamic weight adjustment of multimodal features; and the natural language processing submodule 104 supports intent recognition, sentiment analysis, and contextual understanding.

[0008] Furthermore, the intelligent pronunciation correction system 200 includes an acoustic feature analysis submodule 201, a mouth shape visual correction submodule 202, a dual-modal fusion submodule 203, a pronunciation scoring submodule 204, and a correction suggestion generation submodule 205. The acoustic feature analysis submodule 201 extracts MFCC, FBank, and formant acoustic features to perform phoneme-level pronunciation quality assessment. The mouth shape visual correction submodule 202 captures the learner's mouth shape in real time and compares it with a standard mouth shape, detecting the tongue tip position and lip shape to generate visual correction guidance. The dual-modal fusion submodule 203 integrates acoustic and visual features to perform dual-modal pronunciation assessment. The pronunciation scoring submodule 204 outputs phoneme-level accuracy, fluency, and completeness scores. The correction suggestion generation submodule 205 generates personalized correction suggestions that include acoustic feedback and visual guidance.

[0009] Furthermore, the AR scenario-based learning module 300 includes a real-time scene capture submodule 301, a 3D reconstruction submodule 302, a virtual element overlay submodule 303, a scenario dialogue engine 304, and an object recognition and annotation submodule 305. The real-time scene capture submodule 301 captures the real environment in real time using a camera and a depth sensor. The 3D reconstruction submodule 302 reconstructs the captured scene in 3D space. The virtual element overlay submodule 303 overlays English annotations, virtual characters, and speech bubbles onto the real scene. The scenario dialogue engine 304 generates virtual dialogue characters to engage in immersive English scenario dialogues with learners in the scene. The object recognition and annotation submodule 305 identifies real objects and overlays corresponding English words, phonetic symbols, and pronunciation buttons.

[0010] Furthermore, the embedded software system also includes an adaptive learning management system 400, which comprises a learning data acquisition submodule 401, a personalized recommendation submodule 402, an intelligent review submodule 403, and a learning analysis submodule 404. The learning data acquisition submodule 401 collects learning duration, accuracy rate, error type, and memory curve data. The personalized recommendation submodule 402 recommends learning content based on collaborative filtering and deep learning algorithms. The intelligent review submodule 403 predicts the optimal review time based on the Ebbinghaus forgetting curve and automatically pushes review tasks. The learning analysis submodule 404 generates visual learning reports and progress tracking.

[0011] Furthermore, the embedded software system also includes a secure focus mode 500, which includes an application whitelist mechanism 501, a learning time management submodule 502, a distraction detection submodule 503, and a data encryption submodule 504. The application whitelist mechanism 501 only allows access to learning-related applications in focus mode. The learning time management submodule 502 supports setting learning and rest times. The distraction detection submodule 503 detects the learner's attention state through a camera. The data encryption submodule 504 performs on-device encryption to protect the privacy and security of learning data.

[0012] Furthermore, the multimodal perception module 3 also includes a TOF depth sensor 33 and a vibration feedback motor 34; the TOF depth sensor 33 is used for three-dimensional spatial perception, gesture tracking, and AR scene depth measurement; the vibration feedback motor 34 provides tactile interactive feedback and supports differentiated vibration prompts for different learning scenarios.

[0013] Furthermore, the multimodal fusion submodule 103 adopts a multi-head self-attention mechanism to dynamically adjust the fusion weights of speech features, visual features, gesture features, and tactile features, thereby achieving deep association of cross-modal information; the fusion process supports asynchronous input and synchronous output, with a single interaction response time of no more than 500ms.

[0014] Furthermore: the acoustic feature analysis submodule 201 adopts an end-to-end deep learning model to automatically learn acoustic representations related to pronunciation quality and outputs an acoustic score covering multiple dimensions such as phoneme boundaries, duration, and energy; the scoring algorithm comprehensively considers phoneme accuracy, intonation fluctuation, and prosodic naturalness.

[0015] Furthermore: the lip shape visual correction submodule 202 adopts a deep convolutional neural network model to detect 68 facial key points in the lip shape image, extract parameters such as lip opening degree, tongue tip position, and mandibular opening degree, compare them with the preset standard lip shape parameter library, and generate lip shape correction guidance animation; the standard lip shape parameter library covers 48 international phonetic symbols in English.

[0016] Furthermore, the dual-modal fusion submodule 203 employs a cross-modal attention network to map the representations of acoustic and visual features to a unified semantic space. Through contrastive learning, it achieves effective fusion of dual-modal features and outputs a phoneme-level comprehensive pronunciation evaluation result.

[0017] Furthermore: the 3D reconstruction submodule 302 uses SLAM technology to achieve real-time positioning and map building, supports dynamic object detection and occlusion processing, and achieves centimeter-level reconstruction accuracy; the virtual element overlay adopts the ARCore / ARKit framework to achieve visual consistency of virtual and real fusion.

[0018] Furthermore, the Ebbinghaus forgetting curve parameters of the intelligent review submodule 403 are adaptively adjusted based on individual learning data to form a personalized forgetting curve model; the review tasks are prioritized based on memory strength, forgetting speed, and the relevance of knowledge points.

[0019] Furthermore: the foldable support mechanism 2 includes a multi-level folding bracket 21 and a magnetic angle positioner 22. The multi-level folding bracket 21 provides continuous angle adjustment from 0 to 180 degrees, and the magnetic angle positioner 22 provides positioning feedback at four commonly used angles: 15 degrees, 45 degrees, 90 degrees, and 135 degrees. The human-computer interaction module 4 also includes a focus mode physical switch 44 located on the device body, which physically cuts off access to entertainment applications when toggled.

[0020] Furthermore: the eye-protection display screen 41 adopts an AMOLED display screen with a size of 8.9 inches, a resolution of no less than 2560x1600, supports DC dimming and low blue light certification, and a refresh rate of no less than 90Hz; the directional sound unit 42 adopts ultrasonic directional sound technology to achieve directional audio propagation within a range of ±15°; the power management module 5 is equipped with a lithium battery of no less than 10000mAh, supports PD fast charging and wireless charging, and has a typical battery life of no less than 15 hours.

[0021] Furthermore, the human-computer interaction module 4 also includes a pressure-sensitive stylus 43, which supports no less than 4096 levels of pressure sensitivity for writing and annotation; the pressure-sensitive stylus can be magnetically attached to the device body for storage.

[0022] Furthermore: A method for English pronunciation correction based on multimodal fusion includes the following steps: S1. Synchronous acquisition steps: Acquire the learner's voice signal through a microphone array, and simultaneously acquire the learner's lip-sync video stream through a camera; S2. Acoustic feature extraction step: Preprocess the speech signal to remove noise and silence segments, and extract Mel frequency cepstral coefficient (MFCC) features, filter bank (FBank) features, and first, second, and third formant frequency features. S3. Visual feature extraction step: Perform face detection and mouth region localization on the mouth shape video stream, use a deep convolutional neural network model to extract 68 facial key points, and calculate parameters such as lip opening degree, relative position of tongue tip, and mandibular opening degree. S4. Multimodal feature fusion step: After encoding the acoustic features and visual features respectively, input them into a cross-modal attention network, realize the semantic alignment and fusion of the two-modal features through contrastive learning, and output a phoneme-level comprehensive feature representation; S5. Phoneme-level evaluation steps: Based on the comprehensive feature representation, score the pronunciation quality of each phoneme and output the phoneme accuracy score, fluency score, and integrity score; S6. Error correction suggestion generation step: Based on the scoring results, retrieve the standard mouth shape parameters and acoustic parameters of the target phoneme from the standard pronunciation library, and generate personalized error correction suggestions that include visual animation comparison, voice demonstration and text guidance.

[0023] Furthermore: In step S2, the acoustic feature extraction adopts an end-to-end deep learning model to automatically learn acoustic representations related to pronunciation quality, supporting automatic detection and annotation of mixed Chinese and English pronunciation.

[0024] Furthermore: In step S3, the deep convolutional neural network model adopts a ResNet or MobileNet architecture, the input mouth shape image size is 224x224 pixels, and the output is 68 two-dimensional facial key point coordinates; the tongue tip position parameter is defined as the relative offset of the tongue tip key point relative to the nose tip.

[0025] Furthermore: In step S4, the cross-modal attention network adopts a multi-head self-attention mechanism with no less than 8 attention heads, generates attention weights for acoustic and visual features, and realizes dynamic association of cross-modal information; the fusion process supports asynchronous input and synchronous output, and the single fusion processing time is no more than 100ms.

[0026] Furthermore: In step S5, the phoneme-level assessment covers 48 international phonetic symbols in English, including 12 vowels, 28 consonants, and 8 diphthongs; the scoring results are divided into four levels, A, B, C, and D, from high to low accuracy.

[0027] Furthermore: An augmented reality-based contextualized English learning approach, comprising the following steps: S1. Environmental perception step: Capture RGB images and depth information of the real environment in real time through a camera and a TOF depth sensor; S2, 3D reconstruction step: The RGB image and depth information are processed, and the SLAM algorithm is used for real-time localization and map construction to realize the 3D spatial reconstruction of the real environment; S3. Object recognition steps: Perform semantic segmentation and object recognition on the reconstructed 3D scene, and detect and classify the types and locations of objects in the scene; S4. Virtual annotation step: Overlay virtual English annotations on the identified objects, including word spelling, phonetic pronunciation, part-of-speech description, and example sentence display; S5. Contextual Interaction Steps: Generate virtual dialogue characters and generate English dialogue content that matches the context based on the current scene and the user's learning progress, so as to achieve immersive contextual dialogue with the user. S6. Learning Recording Steps: Automatically record the user's interaction data with virtual annotations and contextual dialogues to the adaptive learning system, triggering an intelligent review mechanism based on the forgetting curve.

[0028] Furthermore: In step S2, the SLAM algorithm uses visual inertial odometry and combines it with IMU sensor data to improve positioning accuracy, achieving centimeter-level reconstruction accuracy; it also supports dynamic object detection and occlusion handling.

[0029] Furthermore: In step S3, the object recognition adopts a deep learning model for target detection, which supports the recognition of no less than 1,000 common object categories; the object detection confidence threshold is set to 0.7.

[0030] Furthermore: In step S4, the style of the virtual annotation is adaptively adjusted according to the object size, distance, and lighting conditions to ensure visual clarity; the annotation content supports clicking to trigger the standard pronunciation playback of the corresponding word.

[0031] Furthermore: In step S5, the virtual dialogue characters are generated using AIGC technology, supporting no fewer than 10 preset character images and 3 dialogue scenarios; the dialogue content is generated based on a large language model, supporting both free dialogue and task-oriented dialogue modes.

[0032] Furthermore: An adaptive English review method based on the forgetting curve includes the following steps: S1. Modeling steps: Build a personalized forgetting curve model for each learner and record the first learning time, number of reviews, historical scores and learning intervals for each knowledge point; S2. Prediction Steps: Based on the Ebbinghaus forgetting curve formula, predict the forgetting time point for each knowledge point, and adjust the parameters in conjunction with the personalized forgetting curve model. S3. Scheduling steps: Based on the forgotten time point, automatically push review tasks to learners at the best review time; prioritize review tasks according to the importance, mastery, and relevance of knowledge points; S4. Update Steps: After the learner completes the review, the parameters of the forgetting curve model are updated based on the review results and time spent; the review frequency for knowledge points with a fast forgetting rate is automatically increased. S5. Optimization Steps: Based on the analysis of the forgetting curve, the assessment of knowledge mastery, and the statistics of learning efficiency, optimize the order of subsequent learning content delivery and the planning of learning paths.

[0033] Furthermore: In step S1, the forgetting curve model parameters include the initial forgetting rate, the review decay coefficient, and the stabilization threshold; after the new knowledge point is learned for the first time, the review is automatically arranged at intervals of 1 minute, 10 minutes, 1 hour, 1 day, 3 days, and 7 days.

[0034] Furthermore: in step S2, the parameters in the Ebbinghaus forgetting curve formula are adaptively adjusted based on individual learning data, and the model parameters are updated in real time using an online learning algorithm; the forgetting prediction supports concurrent scheduling of multiple knowledge points.

[0035] Furthermore: In step S3, the priority calculation formula for the review task is: priority = knowledge point weight multiplied by forgetting risk coefficient multiplied by learning efficiency coefficient; when the priority score is higher than the set threshold, an instant review reminder is triggered.

[0036] Compared with the prior art, the present invention has the following beneficial effects: 1. Multimodal interaction brings a revolutionary learning experience. This invention is the first to create a four-modal fusion interaction system of voice + vision + gesture + touch. Learners can interact with the device through natural language, gestures, writing and other methods, which completely changes the limitations of traditional touch interaction and improves the naturalness of human-computer interaction by more than 80%.

[0037] 2. A Qualitative Leap in Pronunciation Correction Accuracy: Through dual-modal verification using acoustic feature analysis and visual lip-shape recognition, this invention achieves precise pronunciation correction at the phoneme level. Experimental data shows that the system's pronunciation problem detection rate is 65% higher than traditional speech recognition systems. Combined with visual lip-shape correction guidance, learners' pronunciation accuracy improves by an average of 40%.

[0038] 3. AR-based scenario-based learning solves the problem of the "disconnect between learning and application." It deeply integrates English learning with real-life scenarios, allowing learners to engage in immersive situational dialogue training in a real environment, effectively solving the problem of traditional learning being "theoretical." User research shows that scenario-based learning increases vocabulary retention by 55% and significantly enhances confidence in spoken English.

[0039] 4. User-friendly hardware design comprehensively enhances the user experience. 0°-180° stepless angle adjustment adapts to all usage scenarios, and the ergonomic design reduces the burden on the cervical spine; The eye-protecting AMOLED screen, combined with DC dimming, reduces visual fatigue by 60%, effectively protecting the eyesight of teenagers; Targeted sound technology protects students' privacy and does not disturb others; With an ultra-long battery life of 15 hours, it meets all-day learning needs.

[0040] 5. Focus Mode Guarantees Learning Efficiency from the Root: The physically controlled focus mode eliminates entertainment applications at the hardware level, solving the industry pain point of student distraction. Actual usage data shows that learning efficiency is increased by an average of 2.3 times in focus mode, and effective study time is significantly increased.

[0041] 6. Adaptive learning enables truly personalized education. Based on big data and learning science, the adaptive learning system builds a unique learning path for each learner. The intelligent review mechanism improves memory efficiency by 300%, truly realizing personalized education for each individual.

[0042] 7. This invention, through the deep integration of multimodal artificial intelligence, augmented reality, and hardware innovation, comprehensively solves the technical pain points of existing English learning devices. It achieves major technological breakthroughs in six dimensions: interaction mode, pronunciation correction, learning immersion, user experience, focus guarantee, and personalization, and has outstanding substantive features and significant technological progress. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the overall structure of a multifunctional English learning aid device.

[0044] Figure 2 This is a block diagram showing the module connection of the hardware body of the present invention.

[0045] Figure 3 This is an architecture diagram of the embedded software system of the present invention.

[0046] Figure 4 This is a schematic diagram illustrating the multi-angle adjustment of the foldable support mechanism of the present invention; Figure 5 This is a flowchart illustrating the workflow of the intelligent pronunciation correction system of the present invention. Figure 6 This is a schematic diagram illustrating the working principle of the AR scene-based learning module of the present invention; Figure 7 This is a schematic diagram of data fusion for the multimodal AI interaction engine of the present invention.

[0047] In the diagram: 1-Device host, 11-Main controller, 12-AI accelerator processor, 13-Memory, 14-Wireless communication module; 2-Foldable support mechanism, 21-Multi-level folding bracket, 22-Magnetic angle positioner, 23-Anti-slip base; 3-Multimodal perception module, 31-High-definition binocular camera, 32-Four-microphone array, 33-TOF depth sensor, 34-Vibration feedback motor; 4-Human-computer interaction module, 41-Eye-protecting AMOLED display, 42-Directional sound unit, 43-Pressure-sensitive stylus, 44-Focus mode physical switch; 5-Power management module; 100-Multimodal AI interaction engine, 200-Intelligent pronunciation correction system, 300-AR scenario-based learning module, 400-Adaptive learning management system, 500-Safe focus mode. Detailed Implementation

[0048] The present invention will be further described below with reference to the accompanying drawings: Example 1: This invention provides the specific functions and operation process of a multifunctional English learning aid device: Application scenarios for English pronunciation correction: Use case: Middle school students practicing English pronunciation at home. Operating steps: Device Preparation and Start-up: The learner places the device on the desktop, adjusts it to a 45° viewing angle using the multi-level folding stand (21), and toggles the focus mode physical switch (44) to enter focus learning mode. After the device is started, the eye-friendly AMOLED display (41) automatically adjusts to a brightness suitable for indoor lighting.

[0049] Learners can access the intelligent pronunciation correction system (200) by selecting pronunciation training courses via touch or voice command to "open pronunciation training". Based on the learner's historical data, the system recommends training content suitable for the current level through the personalized recommendation submodule (402). In this embodiment, "English International Phonetic Symbols Special Training" is selected.

[0050] Real-time pronunciation acquisition and analysis: Learners follow the system to read the word "think" aloud. A four-microphone array (32) clearly acquires speech signals through beamforming technology, while a high-definition binocular camera (31) simultaneously captures the learner's lip-sync video.

[0051] Bimodal pronunciation assessment Acoustic Analysis: The acoustic feature analysis submodule (201) extracts the MFCC features of the speech and compares them with the standard pronunciation, finding that the learner mispronounces / θ / as / s / ; Visual analysis: The lip-shape visual correction submodule (202) detected that the learner's tongue tip was not extended, which does not conform to the correct lip shape for the / θ / pronunciation; Multimodal fusion verification: The multimodal fusion submodule (103) integrates acoustic and visual evidence to confirm the existence of pronunciation errors.

[0052] The intelligent error correction guidance system immediately generates error correction feedback: The screen displays an animation comparing the learner's mouth shape with the standard mouth shape, with the position of the tongue tip marked with a red arrow; Directional sound unit (42) plays correction instructions: "Please gently place the tip of your tongue between your upper and lower teeth and let the airflow out from between your tongue and teeth"; The pronunciation scoring submodule (203) gives the pronunciation score for this time: accuracy 58 points, fluency 85 points, completeness 90 points, and overall score 72 points.

[0053] Repeated practice and progress tracking: After the learner practiced repeatedly 5 times according to the guidance, the pronunciation score improved to 92 points. The intelligent review submodule (403) automatically recorded the weak phonetic symbol and arranged review after 1 day, 3 days and 7 days.

[0054] After the training is completed, the learning analysis submodule (404) generates a visual report: This training lasted 25 minutes, 32 words were practiced, 3 weak phonetic symbols were found, and the overall improvement rate was 18%.

[0055] Results: Through the pronunciation correction training in this embodiment, learners significantly improved their pronunciation of phonemes such as / θ / , / ð / , and / r / , which are not present in Chinese, within two weeks, and their spoken language accuracy was greatly improved.

[0056] Example 2: This invention provides the specific functions and operation process of a multifunctional English learning aid device: AR-based immersive learning application scenarios: Use case: Learners engage in everyday English learning in a kitchen setting. Operating steps: Entering AR learning mode, the learner stands in the kitchen holding the device and uses the voice command "Open AR Real-Scene Learning" to activate the AR scenario-based learning module (300). The high-definition binocular camera (31) and TOF depth sensor (33) begin to capture the surrounding environment in real time.

[0057] The real-time scene 3D reconstruction and real-time scene capture submodule (301) performs real-time 3D reconstruction of the kitchen and identifies objects such as refrigerators, microwave ovens, kettles, vegetables, and fruits.

[0058] Intelligent Overlay of Virtual Elements: The virtual element overlay submodule (302) overlays English annotations onto real objects. The refrigerator displays "refrigerator" along with its phonetic symbol. The microwave oven is labeled "microwave oven". Each object has a speaker icon next to it; click it to hear the standard pronunciation.

[0059] Immersive scenario dialogue learners click "Start Scenario Dialogue," and the virtual character "Chef Mike" appears on the screen. The scenario dialogue engine (303) initiates a kitchen scenario dialogue: Mike: "What would you like for breakfast?" Learner: "I would like some milk and bread." The system identifies and evaluates in real time: "Excellent! Pronunciation is standard, grammar is correct." Interactive vocabulary learning The learner picks up an apple and points it at the camera; the object recognition and annotation submodule (304) immediately recognizes and displays it. Word: apple Phonetic symbols:

[0060] Example: "An apple a day keeps the doctor away." Expanded vocabulary: apple juice, apple pie The learning data is automatically recorded and added to the vocabulary list. The intelligent review submodule (403) arranges reviews according to the forgetting curve. The system identifies that the learner has a good grasp of "vegetable" vocabulary, but needs to strengthen their knowledge of "kitchen appliance" vocabulary.

[0061] Implementation Results: Through AR-based scenario-based learning, learners naturally acquire 30+ kitchen-related vocabulary words and 5 sets of commonly used dialogues in real-life environments. The learning process is easy and fun, and the memory effect is significantly better than traditional rote memorization.

[0062] Any technical solution that achieves the above-mentioned technical effects by utilizing the technical solutions described in this invention, or by designing similar technical solutions by those skilled in the art under the inspiration of the technical solutions described in this invention, falls within the protection scope of this invention.

Claims

1. A multifunctional English learning aid device, comprising: The hardware main body and embedded software system are characterized in that: the hardware main body includes a device host (1), a foldable support mechanism (2), a multimodal sensing module (3), a human-computer interaction module (4), and a power management module (5); The device host (1) integrates a main controller (11), an AI acceleration processor (12), a memory (13), and a wireless communication module (14). The AI ​​acceleration processor (12) is an edge-side independent NPU neural network processor with a computing power of ≥12 TOPS, which can realize offline AI computing. The foldable support mechanism (2) includes a multi-level folding bracket (21), a magnetic angle positioner (22), and an anti-slip base (23). The multi-level folding bracket (21) is a hidden integrated structure. After being stored, it can completely fit the back of the main unit of the device without protruding the external contour. The support mechanism can achieve continuous angle adjustment from 0° to 180°. The multimodal sensing module (3) includes a high-definition binocular camera (31), a four-microphone array (32), a TOF depth sensor (33), and a vibration feedback motor (34); The human-computer interaction module (4) includes an eye-protection AMOLED display (41), a directional sound unit (42), a pressure-sensitive stylus (43), and a focus mode physical switch (44). The pressure-sensitive stylus (43) supports 4096 levels of pressure sensitivity and is used for writing and annotation. The embedded software system includes a multimodal AI interaction engine (100), an intelligent pronunciation correction system (200), an AR scenario-based learning module (300), an adaptive learning management system (400), and a safe focus mode (500); The power management module (5) is equipped with a 10000mAh lithium battery, supports PD fast charging and wireless charging, and has a typical battery life of no less than 15 hours.

2. The multifunctional English learning aid device according to claim 1, characterized in that: The multimodal AI interaction engine (100) includes a speech recognition submodule (101), a computer vision submodule (102), a multimodal fusion submodule (103), and a natural language processing submodule (104). The speech recognition submodule (101) is based on an end-to-end Transformer model and supports mixed Chinese and English recognition; The computer vision submodule (102) integrates face, lip, gesture and object recognition algorithms; The multimodal fusion submodule (103) uses a cross-attention mechanism to fuse speech, visual, and text features; The natural language processing submodule (104) enables intelligent dialogue, syntax correction and semantic understanding.

3. The multifunctional English learning aid device according to claim 1, characterized in that: The intelligent pronunciation correction system (200) includes an acoustic feature analysis submodule (201), a mouth shape visual correction submodule (202), a pronunciation scoring submodule (203), and a correction suggestion generation submodule (204). The acoustic feature analysis submodule (201) extracts MFCC, FBank, and formant acoustic features to perform phoneme-level pronunciation quality assessment. The lip-shape visual correction submodule (202) captures the learner's lip shape in real time and compares it with the standard lip shape to generate visual correction guidance; The pronunciation scoring submodule (203) provides a comprehensive score of 0 to 100 points based on three dimensions: accuracy, fluency, and completeness. The error correction suggestion generation submodule (204) provides specific adjustment suggestions for tongue position, mouth shape, and breath for pronunciation problems.

4. The multifunctional English learning aid device according to claim 1, characterized in that: The AR scenario-based learning module (300) includes a real-time scene capture submodule (301), a virtual element overlay submodule (302), a scenario dialogue engine (303), and an object recognition and annotation submodule (304). The real-time scene capture submodule (301) captures the real environment through a camera and performs three-dimensional reconstruction; The virtual element overlay submodule (302) overlays English annotations, virtual characters, and speech bubbles in a real scene; The scenario dialogue engine (303) constructs dialogue training for various typical life scenarios; The object recognition and labeling submodule (304) automatically recognizes objects within the field of view and displays their English names and pronunciations.

5. The multifunctional English learning aid device according to claim 1, characterized in that: The adaptive learning management system (400) includes a learning data collection submodule (401), a personalized recommendation submodule (402), an intelligent review submodule (403), and a learning analysis submodule (404). The learning data acquisition submodule (401) collects learning duration, accuracy rate, types of incorrect questions, and memory curve data; The personalized recommendation submodule (402) recommends learning content based on collaborative filtering and deep learning algorithms; The intelligent review submodule (403) automatically arranges a review plan based on the Ebbinghaus forgetting curve; The learning analysis submodule (404) generates a visual learning report and identifies weaknesses.

6. The multifunctional English learning aid device according to claim 1, characterized in that: The secure focus mode (500) includes an application whitelist mechanism (501), a learning time management submodule (502), a distraction detection submodule (503), and a data encryption submodule (504). The application whitelist mechanism (501) only allows access to learning-related applications in focus mode; The learning duration control submodule (502) supports setting learning and rest times; The distraction detection submodule (503) detects the learner's attention state through a camera; The data encryption submodule (504) performs end-side encryption on the learning data to protect privacy and security.

7. The multifunctional English learning aid device according to claim 1, characterized in that: The high-definition binocular camera (31) has a resolution of no less than 12 million pixels and supports 4K@30fps video capture; the four-microphone array (32) adopts a ring layout, supports beamforming and noise reduction algorithms, and achieves accurate voice pickup within a 5-meter range.

8. The multifunctional English learning aid device according to claim 1, characterized in that: The eye-protection AMOLED display (41) is 9 inches in size with a resolution of 2560×1600, supports DC dimming and low blue light certification, and has a refresh rate of 90Hz; the directional sound unit (42) adopts ultrasonic directional sound technology, supports directional volume adaptive adjustment, realizes directional audio propagation within a range of ±15°, can output synchronously with headphones, and supports single-person focused listening.

9. A multifunctional English learning aid device according to claim 1, characterized in that: The magnetic angle locator (22) provides positioning feedback at four common angles: 15°, 45°, 90°, and 135°. The physical switch (44) of the focus mode cuts off hardware-level permissions after being toggled, directly disabling the running permissions of non-learning applications.

Images