Holographic remote interactive teaching system

Abstract

The invention discloses a holographic remote interactive teaching system, and relates to the technical field of teaching systems, and the system comprises an integrated teaching system which comprises a three-dimensional dynamic capturing module, a holographic imaging module, an intelligent interactive processing module and a low-delay communication module. The three-dimensional dynamic capture module is used for collecting three-dimensional holographic data of teachers and students in real time, and the three-dimensional holographic data comprises multi-angle depth images and skeleton joint point data. According to the invention, natural eye contact, body language capture and real-time feedback between teachers and students are realized, a dynamic holographic image with a real space depth is generated, the immersion and interactivity of teaching are effectively improved, the problems of eye contact deficiency and student attention distraction in traditional video teaching are solved, and remote teaching is closer to real classroom experience.

Description

Technical Field

[0001] This invention relates to the field of teaching systems, specifically a holographic remote interactive teaching system. Background Technology

[0002] In recent years, with the rapid development of information technology, distance education has gradually become an important development direction in the education field. Especially under the influence of the global public health event, the demand for distance learning has been growing, and various online teaching platforms such as video conferencing systems and live online classes have been widely used.

[0003] Most existing distance learning systems are still based on two-dimensional video transmission technology, which has problems such as poor interactivity, insufficient immersion, and limited teaching effectiveness. For example, traditional video teaching makes it difficult to achieve natural eye contact, capture and feedback of body language between teachers and students, and cannot create a sense of presence and space in a real classroom, which can easily lead to learning fatigue and distraction for students. In addition, existing systems also have limitations in teaching demonstrations, especially in teaching content that requires three-dimensional display. Two-dimensional images cannot provide enough visual information and operational experience, which affects teaching quality and learning outcomes. Therefore, this invention provides a holographic remote interactive teaching system. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a holographic remote interactive teaching system. This system solves the problems of existing remote teaching systems, which are mostly based on two-dimensional video transmission technology, resulting in poor interactivity, insufficient immersion, and limited teaching effectiveness. For example, traditional video teaching struggles to achieve natural eye contact, capture and respond to body language between teachers and students, and fails to create a realistic classroom atmosphere and spatial feel, leading to student fatigue and distraction. Furthermore, existing systems have limitations in teaching demonstrations, especially for content requiring three-dimensional presentation, where two-dimensional images cannot provide sufficient visual information and operational experience, thus affecting teaching quality and learning outcomes.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a holographic remote interactive teaching system, comprising an overall teaching system, which includes a three-dimensional dynamic capture module, a holographic imaging module, an intelligent interactive processing module, and a low-latency communication module. The three-dimensional dynamic capture module is used to collect three-dimensional holographic data of teachers and students in real time. The three-dimensional holographic data includes multi-angle depth images, skeletal joint data, eye gaze direction, and micro-expression changes. The holographic imaging module uses light field display technology and programmable diffractive optical elements to generate dynamic holographic images with real spatial depth. The intelligent interactive processing module uses artificial intelligence algorithms to fuse the collected three-dimensional data in real time to generate an interactive teaching scene with feedback capabilities. The low-latency communication module uses adaptive bit rate transmission and multiplexing technology to achieve synchronous transmission of holographic data between distributed terminals.

[0006] Preferably, the three-dimensional dynamic capture module includes multiple sets of time-of-flight (ToF) cameras and structured light sensors arranged around the teaching space for synchronously acquiring three-dimensional point cloud data. It also includes an infrared thermal imaging unit for capturing human body heat distribution to assist in emotional state recognition and a gaze tracking subsystem. The gaze tracking subsystem is based on corneal reflection and pupil center tracking technology to calculate the point of eye contact between teachers and students in real time.

[0007] Preferably, the intelligent interaction processing module includes an embedded neural network processor, a teaching context understanding unit, and a real-time rendering engine. The embedded neural network processor is used to run a multimodal behavior recognition model to recognize teaching behaviors such as raising hands, nodding, and writing. The teaching context understanding unit dynamically adjusts the teaching elements and interaction logic in the virtual scene according to the recognition results. The real-time rendering engine supports physical lighting models and material reflection simulation to enhance the realism of holographic images.

[0008] Preferably, the overall teaching system further includes a tactile feedback subsystem, a speech enhancement module, and a spatial audio synthesizer. The tactile feedback subsystem uses an integrated piezoelectric vibration unit and a pneumatic tactile array to simulate the texture, resistance, and collision feedback of objects. The speech enhancement module uses beamforming technology and a sound source separation algorithm to achieve directional sound pickup and noise suppression. The spatial audio synthesizer dynamically adjusts the sound field according to the user's position to create a three-dimensional stereo sound effect.

[0009] Preferably, the holographic imaging module includes a rotatable holographic projection screen, a stereoscopic light field display unit, and a projection calibration system. The rotatable holographic projection screen is used to adaptively adjust the display angle in accordance with eye-tracking results. The stereoscopic light field display unit uses a multilayer liquid crystal phase modulator to generate a holographic image with continuous depth information. The projection calibration system corrects image distortion and color deviation in real time based on camera feedback.

[0010] Preferably, the low-latency communication module supports an edge computing collaborative architecture under 5G network, which offloads some rendering tasks to terminal devices. It also supports an adaptive streaming media transmission protocol and a user data synchronization mechanism. The adaptive streaming media transmission protocol dynamically adjusts the resolution and frame rate of holographic data according to the network status. The multi-user data synchronization mechanism adopts a timestamp and state prediction algorithm to ensure the consistency of interactive scenarios.

[0011] Preferably, the overall teaching system further includes an emotion computing unit, an adaptive teaching strategy engine, and a learning analysis dashboard. The emotion computing unit analyzes students' emotional states based on facial expressions, voice tone, and physiological signals. The adaptive teaching strategy engine dynamically recommends teaching content and interaction methods based on the emotion analysis results. The learning analysis dashboard visually displays students' participation, comprehension, and emotional change trends.

[0012] Preferably, the overall teaching system also includes cross-platform multi-terminal access, distributed classroom architecture, and teaching scene memory function. The cross-platform multi-terminal access includes holographic display cabins, AR glasses, mobile terminals, and desktop devices. The distributed classroom architecture allows students from multiple physical classrooms to enter the same virtual classroom in the form of holographic avatars. The teaching scene memory function can save and reproduce holographic scenes of specific teaching moments, supporting playback and review analysis.

[0013] Beneficial effects This invention provides a holographic remote interactive teaching system. Compared with existing technologies, it has the following advantages: (1) This holographic remote interactive teaching system, through the collaborative work of the three-dimensional dynamic capture module, the holographic imaging module and the intelligent interactive processing module, realizes natural eye contact, body language capture and real-time feedback between teachers and students, generates dynamic holographic images with real spatial depth, effectively enhances the immersion and interactivity of teaching, solves the problems of lack of eye contact and distraction of students in traditional video teaching, and makes remote teaching closer to the real classroom experience.

[0014] (2) This holographic remote interactive teaching system, through the integration of an emotion computing unit, an adaptive teaching strategy engine and a low-latency communication module, can analyze students' emotional state and behavioral data in real time, dynamically adjust teaching content and interaction methods, and realize personalized teaching; at the same time, by using edge computing and adaptive streaming media transmission technology, it ensures efficient synchronization and low-latency transmission of holographic data, improves teaching efficiency and learning effect, and solves the problems of traditional systems being unable to provide real-time feedback on student status and lag caused by network latency. Attached Figure Description

[0015] Figure 1This is a block diagram of the overall teaching system of the present invention; Figure 2 This is a block diagram of the three-dimensional dynamic capture module of the present invention; Figure 3 This is a block diagram of the intelligent interaction processing module of the present invention; Figure 4 This is a block diagram of the rotatable holographic projection screen of the present invention; Figure 5 This is a block diagram of the low-latency communication module of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Please see Figure 1 and Figure 5 This invention provides a technical solution: a holographic remote interactive teaching system, comprising an overall teaching system, characterized in that the overall teaching system includes a three-dimensional dynamic capture module, a holographic imaging module, an intelligent interactive processing module, and a low-latency communication module. The three-dimensional dynamic capture module is used to collect three-dimensional holographic data of teachers and students in real time. The three-dimensional holographic data includes multi-angle depth images, skeletal joint point data, eye gaze direction, and micro-expression changes. The holographic imaging module uses light field display technology and programmable diffractive optical elements to generate dynamic holographic images with real spatial depth. The intelligent interactive processing module uses artificial intelligence algorithms to fuse the collected three-dimensional data in real time to generate an interactive teaching scene with feedback capabilities. The low-latency communication module uses adaptive bit rate transmission and multiplexing technology to realize synchronous transmission of holographic data between distributed terminals.

[0018] In a preferred embodiment, the 3D dynamic capture module includes multiple sets of Time-of-Flight (ToF) cameras and structured light sensors arranged around the teaching space to synchronously acquire 3D point cloud data. It also includes an infrared thermal imaging unit to capture human body heat distribution to assist in emotional state recognition and a gaze tracking subsystem. The gaze tracking subsystem, based on corneal reflection and pupil center tracking technology, calculates the point of eye contact between the teacher and student in real time. Specifically, the ToF cameras and structured light sensors are arranged around the teaching space to synchronously acquire 3D point cloud data, generating high-precision depth images and skeletal joint data. The infrared thermal imaging unit, by capturing human body heat distribution data, assists in recognizing the student's emotional state. The gaze tracking subsystem, utilizing corneal reflection and pupil center tracking technology, calculates the point of eye contact between the teacher and student in real time, achieving natural eye contact. Through multimodal data acquisition, comprehensive 3D dynamic capture of the teacher and student is achieved, including micro-expressions and eye direction, providing an accurate data foundation for subsequent holographic imaging and intelligent interaction. This enhances the immersiveness and interactivity of teaching, solves the problem of missing eye contact in traditional video teaching, and makes it easier for students to concentrate.

[0019] In a preferred embodiment, the intelligent interaction processing module includes an embedded neural network processor, a teaching context understanding unit, and a real-time rendering engine. The embedded neural network processor runs a multimodal behavior recognition model to identify teaching behaviors such as raising a hand, nodding, and writing. The teaching context understanding unit dynamically adjusts teaching elements and interaction logic in the virtual scene based on the recognition results. The real-time rendering engine supports physical lighting models and material reflection simulation to enhance the realism of the holographic images. The embedded neural network processor runs the multimodal behavior recognition model to identify teaching behaviors such as raising a hand, nodding, and writing in real time. The teaching context understanding unit dynamically adjusts teaching elements and interaction logic in the virtual scene based on the recognition results. The real-time rendering engine uses physical lighting models and material reflection simulation technology to render the holographic images with high quality, enhancing visual realism. Through real-time processing of behavioral data using artificial intelligence algorithms, the teaching scene can intelligently respond to teacher-student interactions, improving teaching efficiency and naturalness. The rendering engine enhances the realism of the holographic images, making it easier for students to immerse themselves in the virtual environment and reducing learning fatigue.

[0020] In a preferred embodiment, the overall teaching system further includes a tactile feedback subsystem, a speech enhancement module, and a spatial audio synthesizer. The tactile feedback subsystem uses an integrated piezoelectric vibration unit and a pneumatic tactile array to simulate object texture, resistance, and collision feedback. The speech enhancement module uses beamforming technology and a sound source separation algorithm to achieve directional sound pickup and noise suppression. The spatial audio synthesizer dynamically adjusts the sound field according to the user's position to create a three-dimensional stereo sound effect. The tactile feedback subsystem simulates object texture (such as the resistance of writing with chalk) and collision feedback through vibration and air pressure. The speech enhancement module uses beamforming to directionally pick up the teacher's speech and suppresses environmental noise through sound source separation. The spatial audio synthesizer calculates sound field parameters based on the student's position to generate a three-dimensional stereo sound effect. By providing multi-sensory feedback (tactile and auditory), it enhances immersion and makes remote teaching closer to the real classroom experience. Speech enhancement ensures speech clarity, and spatial audio creates a sense of presence, solving the problems of monotonous audio and insufficient interactivity in traditional systems.

[0021] In a preferred embodiment, the holographic imaging module includes a rotatable holographic projection screen, a stereoscopic light field display unit, and a projection calibration system. The rotatable holographic projection screen is used to adaptively adjust the display angle in accordance with eye-tracking results. The stereoscopic light field display unit uses a multilayer liquid crystal phase modulator to generate a holographic image with continuous depth information. The projection calibration system corrects image distortion and color deviation in real time based on camera feedback. The rotatable holographic projection screen adaptively adjusts the display angle according to eye-tracking results to ensure the optimal viewing angle. The stereoscopic light field display unit generates a holographic image with continuous depth information through phase modulation. The projection calibration system uses camera feedback to correct image distortion and color deviation in real time, ensuring image stability and generating high-quality holographic images with realistic spatial depth, improving visual realism and comfort. The rotatable screen and calibration system solve the problems of fixed projection angle and image distortion in traditional projection, making it suitable for stereoscopic display of complex teaching content.

[0022] In a preferred embodiment, the low-latency communication module supports an edge computing collaborative architecture under a 5G network, offloading some rendering tasks to terminal devices. It also supports an adaptive streaming media transmission protocol and a user data synchronization mechanism. The adaptive streaming media transmission protocol dynamically adjusts the resolution and frame rate of the holographic data based on network conditions. The multi-user data synchronization mechanism uses timestamps and state prediction algorithms to ensure consistency in interactive scenarios. Edge computing offloads some rendering tasks to terminal devices, reducing cloud load. The adaptive streaming media transmission protocol dynamically adjusts the resolution and frame rate of the holographic data based on network conditions. The user data synchronization mechanism, through timestamps and state prediction algorithms, ensures consistency in multi-user interactive scenarios, significantly reducing transmission latency, improving data synchronization and real-time performance, and ensuring smooth and stable holographic interaction. This solves the stuttering and desynchronization problems caused by network latency in traditional remote teaching systems and is suitable for large-scale distributed classrooms.

[0023] In a preferred embodiment, the overall teaching system further includes an emotion computing unit, an adaptive teaching strategy engine, and a learning analysis dashboard. The emotion computing unit analyzes students' emotional states based on facial expressions, voice tone, and physiological signals. The adaptive teaching strategy engine dynamically recommends teaching content and interaction methods based on the emotion analysis results. The learning analysis dashboard visually displays students' participation, comprehension, and emotional trends. The emotion computing unit uses computer vision and speech analysis technologies to identify students' emotions (such as confusion or interest). The adaptive teaching strategy engine adjusts teaching content (such as switching explanation methods) using a rule engine or machine learning algorithm. The learning analysis dashboard presents participation, comprehension, and emotional trends in chart form, enabling personalized teaching, improving student learning outcomes and participation. Emotion analysis helps teachers adjust teaching strategies in a timely manner, and the dashboard provides data support, solving the problem that traditional systems cannot provide real-time feedback on student status.

[0024] In a preferred embodiment, the overall teaching system further includes cross-platform multi-terminal access, a distributed classroom architecture, and a teaching scene memory function. The cross-platform multi-terminal access includes holographic display pods, AR glasses, mobile terminals, and desktop devices. The distributed classroom architecture allows students from multiple physical classrooms to enter the same virtual classroom in holographic avatar form. The teaching scene memory function can save and reproduce holographic scenes of specific teaching moments, supporting playback and review analysis. Students access the system through different devices; the holographic display pods and AR glasses provide an immersive experience, while mobile terminals support convenient access. The distributed classroom architecture integrates multiple physical classrooms, allowing students to enter the same virtual classroom in holographic avatar form. The teaching scene memory function records holographic scene data, supports playback and review analysis, improves system accessibility and flexibility, supports large-scale collaborative learning and remote collaboration, and facilitates review and teaching evaluation, solving the problems of poor device compatibility and unreproducible scenes in traditional systems.

[0025] The aforementioned device models include: a ToF camera using Microsoft Azure Kinect DK with a depth resolution of 640×576 and a maximum frame rate of 30fps; a structured light sensor using Intel RealSense D415, supporting 1080p resolution depth sensing and employing a FLIR Lepton 3.5 thermal imaging module; an embedded neural network processor using NVIDIA Jetson Nano, running a behavior recognition model optimized based on TensorRT; a piezoelectric vibration unit using Tactile Labs Haptic Actuator; a pneumatic haptic array using the UltraHaptics development kit; and a stereoscopic light field display unit using a multilayer liquid crystal phase modulator to generate holographic images with continuous depth information, for example, using Sony SpatialLight Modulator to achieve phase modulation.

[0026] Furthermore, any content not described in detail in this specification is existing technology known to those skilled in the art.

[0027] During operation, the system uses multiple sets of time-of-flight cameras and structured light sensors deployed around the teaching space to collect real-time 3D holographic data of teachers and students, including multi-angle depth images, skeletal joint data, eye gaze direction, and micro-expression changes. It also utilizes an infrared thermal imaging unit to assist the emotion recognition and gaze tracking subsystems in achieving natural eye contact. The collected data is transmitted efficiently and synchronously between distributed terminals via a low-latency communication module through a 5G network and edge computing collaborative architecture, employing an adaptive streaming media transmission protocol and a multi-user data synchronization mechanism. The intelligent interaction processing module runs a multimodal behavior recognition model through an embedded neural network processor to identify teaching behaviors, which are then used by the teaching context understanding unit. The system dynamically adjusts virtual teaching scenarios, and a real-time rendering engine combines physical lighting and material reflection simulation to enhance the realism of images. The holographic imaging module generates dynamic holographic images with continuous depth information through a rotatable holographic projection screen, a stereoscopic light field display unit, and a projection calibration system. Simultaneously, the system integrates a haptic feedback subsystem, a speech enhancement module, a spatial audio synthesizer, an emotion computing unit, an adaptive teaching strategy engine, and a learning analysis dashboard to achieve multi-sensory interaction, personalized teaching, and learning behavior analysis. Through cross-platform multi-terminal access, a distributed classroom architecture, and teaching scenario memory functions, it supports multi-device access, multi-classroom collaboration, and teaching playback review, thereby comprehensively enhancing the immersion, interactivity, and realism of remote teaching.

[0028] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0029] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A holographic remote interactive teaching system, comprising an overall teaching system, characterized in that, The overall teaching system includes a 3D motion capture module, a holographic imaging module, an intelligent interactive processing module, and a low-latency communication module. The 3D motion capture module is used to collect 3D holographic data of teachers and students in real time. The 3D holographic data includes multi-angle depth images, skeletal joint data, eye gaze direction, and micro-expression changes. The holographic imaging module uses light field display technology and programmable diffractive optical elements to generate dynamic holographic images with real spatial depth. The intelligent interactive processing module uses artificial intelligence algorithms to fuse the collected 3D data in real time to generate interactive teaching scenarios with feedback capabilities. The low-latency communication module uses adaptive bit rate transmission and multiplexing technology to realize synchronous transmission of holographic data between distributed terminals.

2. The holographic remote interactive teaching system according to claim 1, characterized in that: The three-dimensional dynamic capture module includes multiple sets of time-of-flight (ToF) cameras and structured light sensors arranged around the teaching space to synchronously collect three-dimensional point cloud data. It also includes an infrared thermal imaging unit to capture human body heat distribution to assist in emotional state recognition and a gaze tracking subsystem. The gaze tracking subsystem is based on corneal reflection and pupil center tracking technology to calculate the point of eye contact between teachers and students in real time.

3. The holographic remote interactive teaching system according to claim 1, characterized in that: The intelligent interactive processing module includes an embedded neural network processor, a teaching context understanding unit, and a real-time rendering engine. The embedded neural network processor is used to run a multimodal behavior recognition model to recognize teaching behaviors such as raising hands, nodding, and writing. The teaching context understanding unit dynamically adjusts the teaching elements and interactive logic in the virtual scene based on the recognition results. The real-time rendering engine supports physical lighting models and material reflection simulation to enhance the realism of holographic images.

4. The holographic remote interactive teaching system according to claim 1, characterized in that: The overall teaching system also includes a tactile feedback subsystem, a speech enhancement module, and a spatial audio synthesizer. The tactile feedback subsystem uses an integrated piezoelectric vibration unit and a pneumatic tactile array to simulate the texture, resistance, and collision feedback of objects. The speech enhancement module uses beamforming technology and a sound source separation algorithm to achieve directional sound pickup and noise suppression. The spatial audio synthesizer dynamically adjusts the sound field according to the user's position to create a three-dimensional stereo sound effect.

5. The holographic remote interactive teaching system according to claim 1, characterized in that: The holographic imaging module includes a rotatable holographic projection screen, a stereoscopic light field display unit, and a projection calibration system. The rotatable holographic projection screen is used to adaptively adjust the display angle in conjunction with eye-tracking results. The stereoscopic light field display unit uses a multilayer liquid crystal phase modulator to generate a holographic image with continuous depth information. The projection calibration system corrects image distortion and color deviation in real time based on camera feedback.

6. The holographic remote interactive teaching system according to claim 1, characterized in that: The low-latency communication module supports an edge computing collaborative architecture under 5G networks, offloading some rendering tasks to terminal devices. It also supports an adaptive streaming media transmission protocol and a user data synchronization mechanism. The adaptive streaming media transmission protocol dynamically adjusts the resolution and frame rate of holographic data according to network conditions. The multi-user data synchronization mechanism uses timestamp and state prediction algorithms to ensure the consistency of interactive scenarios.

7. The holographic remote interactive teaching system according to claim 1, characterized in that: The overall teaching system also includes an emotion computing unit, an adaptive teaching strategy engine, and a learning analysis dashboard. The emotion computing unit analyzes students' emotional states based on facial expressions, voice tone, and physiological signals. The adaptive teaching strategy engine dynamically recommends teaching content and interaction methods based on the emotion analysis results. The learning analysis dashboard visually displays students' participation, comprehension, and emotional change trends.

8. The holographic remote interactive teaching system according to claim 1, characterized in that: The overall teaching system also includes cross-platform multi-terminal access, a distributed classroom architecture, and a teaching scene memory function. The cross-platform multi-terminal access includes holographic display cabins, AR glasses, mobile terminals, and desktop devices. The distributed classroom architecture allows students from multiple physical classrooms to enter the same virtual classroom in the form of holographic avatars. The teaching scene memory function can save and reproduce holographic scenes of specific teaching moments, supporting playback and review analysis.

Images