Smart classroom multi-modal fusion system and method based on AI technology

Abstract

The invention discloses a smart classroom multi-modal fusion system and method based on an AI technology, and relates to the technical field of smart classrooms, the smart classroom multi-modal fusion system comprises an equipment control unit, a real-time monitoring unit and an image acquisition unit, the image acquisition unit acquires three-dimensional image information by deploying various types of image acquisition equipment, a three-dimensional model is synthesized in real time, and the three-dimensional model is displayed on the equipment control unit; holographic images of teachers in local classrooms are captured in real time. The image acquired by the holographic image acquisition system of the teaching classroom is transmitted to a plurality of remote teaching classrooms, and is restored and presented by the holographic image restoration system of the remote teaching classrooms, so that students who listen to a class can not only display a holographic three-dimensional model in a naked eye 3D effect, but also project holographic three-dimensional images of teachers into the remote classrooms, and the teaching effect of the students is improved. The learning enthusiasm and initiative of the students are improved, and the distance between the remote listening students and the teaching teacher can be shortened.

Description

Technical Field

[0001] The present invention relates to the technical field of smart classrooms, and specifically to a smart classroom multimodal fusion system and method based on AI technology. Background Art

[0002] A smart classroom is a typical materialization of a smart learning environment and a high-end form of multimedia and network classrooms. It is a new type of classroom built with the help of Internet of Things technology, cloud computing technology and intelligent technology. This new type of classroom includes tangible physical space and intangible digital space. It uses various intelligent equipment to assist in the presentation of teaching content, facilitate the acquisition of learning resources, promote classroom interaction, and realize situational awareness and environmental management functions. The smart classroom aims to provide a humanized and intelligent interactive space for teaching activities. Through the combination of physical space and digital space, and the combination of local and remote, it improves the relationship between people and the learning environment, realizes the natural interaction between people and the environment in the learning space, and promotes personalized learning, open learning and ubiquitous learning.

[0003] Traditional classrooms are just classrooms where teachers give lectures to students in real time. However, due to the limitation of classroom space, it is impossible to teach multiple people. If colleges and universities have to conduct remote lectures through the Internet and display them on the screen due to special needs, it is impossible to provide the sense of presence and immersion to the remote students in the remote interactive teaching. It is difficult to improve the enthusiasm and initiative of students in learning. Moreover, when encountering complex problems in learning, they cannot intuitively feel the teaching principles, which will reduce the students' observation ability. In traditional classrooms, it is difficult for teachers to obtain students' learning status and learning trajectory, and it is difficult to provide timely help to students with poor grades. Summary of the invention

[0004] The purpose of the present invention is to provide a smart classroom multimodal fusion system and method based on AI technology to solve the problems raised in the above background technology.

[0005] To achieve the above-mentioned purpose, the present invention provides the following technical solutions: a multimodal fusion system for a smart classroom based on AI technology, comprising a device control unit and a real-time monitoring unit;

[0006] An image acquisition unit, which collects stereoscopic image information by deploying various image acquisition devices, synthesizes a stereoscopic model in real time, and captures the holographic image of the teacher teaching in the local classroom in real time. The blackboard and PPT projection screen used by the teacher are transmitted to the remote classroom together with the audio information through the 5G network transmission system;

[0007] An image processing unit, which renders and preprocesses the collected dynamic images, constructs an imaging field of a holographic virtual image, dynamically tracks the trajectory of the object and adjusts the light during the collection process, uses communication technology to achieve real-time interactive transmission of two-way signals between the local classroom and the remote classroom, and constructs a holographic three-dimensional engine to complete resource retrieval operations;

[0008] An image restoration unit, wherein the image restoration unit restores the collected and processed holographic image of the lecturer in a 1:1 three-dimensional virtual manner in the podium area of ​​the remote classroom by means of a holographic film polymer nano-optical material. The remote classroom simultaneously displays the holographic teaching resource model and the three-dimensional image of the lecturer. When using the holographic teaching resource model, the lecturer performs conventional interactive operations;

[0009] An image presentation unit, wherein the image presentation unit divides the classroom into two areas, one side is a holographic image display area, and the other side is a student listening area. The holographic image display area maximizes the prominence of the virtual portrait, and the student listening area sees the teacher's holographic image. The unit also adaptively adjusts the air conditioning, ventilation system and lighting according to preset parameters to provide sufficient external learning conditions.

[0010] Intuitive teaching unit, teachers use virtual environments to simulate various experimental environments, allowing students to observe and verify teaching principles in interactive operations. The models uploaded before class are holographically projected through a holographic 3D engine. All real-time holographic images will be synchronously stored in real time for one month for further study;

[0011] A real-time interactive unit, which realizes real-time perception of the status of teachers and students by deploying cameras in the local classroom and the listening classroom. Teachers use professional classroom interactive software to post questions at any time and track students' learning status through data analysis tools. Students respond through their own devices and ask teachers for help anytime and anywhere.

[0012] Preferably, the device control unit is responsible for centralized control and management of various types of devices in the classroom, completing the opening and closing operations of various components of the system, and the teacher can conveniently and quickly start and close the system with one key and switch between various functions. The classroom is deployed with a holographic image acquisition system and a holographic image restoration system. Each classroom is interconnected and uniformly controlled by the device control unit. The holographic image restoration systems of the teaching classroom and the remote listening classroom also holographically present the holographic three-dimensional model played by the teacher.

[0013] The real-time monitoring unit collects and compresses the video and audio of students in the remote classroom through ultra-high-definition cameras and microphones, and then sends them to the local classroom through the 5G network transmission system for signal decoding and media distribution. The classroom uses surveillance cameras and wireless microphones to transmit the students' lectures to the image acquisition end classroom. Among them, the video of students in the remote classroom is displayed in real time through the monitor in front of the lecturer, which is convenient for the lecturer to grasp the listening situation of students in the remote classroom.

[0014] Preferably, the image acquisition unit includes an image scanning module, an image capture module and a synchronous transmission module. The image scanning module deploys a blackboard, a projector, a projection screen, a light enhancement matrix, a dual monitoring screen and an image acquisition camera in the classroom podium area, and uses a high-sensitivity camera module to collect stereoscopic image information, and performs full-dimensional image scanning on the collected object to synthesize a stereoscopic model in real time. The image capture module captures the dynamic image of the teacher in the teaching classroom in real time and transmits it to the remote classroom. In the local classroom, the holographic image of the teaching teacher is collected and processed, and then compressed and encoded together with the audio information by the signal coding system. The synchronous transmission module sends it to the remote classroom through the 5G network transmission system for signal decoding, media distribution and holographic image restoration. The teacher's blackboard writing and PPT projection screen in the local classroom are also transmitted synchronously, and the scene and lighting system are built in the classroom.

[0015] Preferably, the image processing unit includes a feature processing module, a data compression module and an interactive transmission module. The feature processing module renders and pre-processes the collected dynamic images with holographic features, and makes the images more visually characteristic of 3D stereoscopic display by changing color highlights, sharpening, contrast, shadows and edge enhancement, so as to construct an imaging field of a holographic virtual image. The data compression module dynamically tracks the trajectory of the object and adjusts the light during the collection process to maintain a balanced value of binocular difference. In addition, the image is encoded and compressed to reduce the amount of data transmitted and reduce network delay. The interactive transmission module uses communication technology to realize real-time interactive transmission of two-way signals between the local classroom and the remote classroom, ensuring the real-time and smooth interaction between the local classroom and the remote classroom in terms of three-dimensional images, audio and video signals.

[0016] The feature processing module includes feature processing algorithms, specifically:

[0017]

[0018] In the formula, g(X|μ i ,∑i) represents the i-th component in the feature model, x represents the three-dimensional continuous data feature, w i represents the mixture weight, i=1,···,M,μ i represents the mean vector, ∑i represents the covariance matrix, Xj represents the value of pixel j, λ represents the value under specified conditions, and p(·) represents the weighted sum of feature components;

[0019] Preferably, the image restoration unit includes a virtual restoration module, a common display module and an interactive operation module. The virtual restoration module uses a holographic film polymer nano-optical material to restore the collected and processed holographic image of the lecturer in a 1:1 three-dimensional virtual manner in the podium area of ​​the remote classroom. Remote students can achieve naked-eye three-dimensional visual effects without the aid of any wearable devices. Through ultra-high-definition video capture technology, students can clearly see the facial expressions and body movements of the lecturer. The common display module can retrieve holographic teaching resources from the holographic three-dimensional engine at any time during the lecture, and display them together with the lecturer in the local classroom. At the same time, the remote classroom also displays the holographic teaching resource model and the three-dimensional image of the lecturer. The interactive operation module enables the lecturer to perform conventional interactive operations on the holographic teaching resource model, including displacement, rotation viewing, zooming in and out, and decomposition and synthesis of the model.

[0020] Preferably, the image presentation unit includes an area division module, an area display module, a virtual display module and an adaptive adjustment module. The area division module divides the classroom into two areas, one side is a holographic image display area for presenting a 1:1 holographic stereoscopic image and a holographic courseware model of the lecturer, and the other side is a student listening area. The area display module makes a holographic image display area with a cubic structure into a pure dark environment to reduce the interference of external ambient light and maximize the highlighting of the virtual portrait. Students see the holographic image of the teacher through this spatial structure and hear the stereo sound of the teacher through the embedded speaker system. The virtual display module positions the image of the holographic teacher 1:1 standing in the middle of the podium and displaces it horizontally along the central axis. The local classroom blackboard writing is synchronously displayed in the virtual scene. The adaptive adjustment module uses sensors to sense the temperature, humidity, air quality and illumination in the classroom in real time, and adaptively adjusts the air conditioning, ventilation system and lighting according to preset parameters.

[0021] Preferably, the intuitive teaching unit includes a device selection module, a model demonstration module and a synchronous storage module. The device selection module uses the device as needed. When holographic projection support is not needed, teachers and students only need to keep the device turned off and carry out normal traditional teaching activities. According to the needs of teaching, holographic courseware is uploaded in advance, and the device is turned on with one click for three-dimensional presentation when the 3D model needs to be displayed. In the model demonstration module, the teacher simulates various experimental environments with the help of a virtual environment, allowing students to observe and verify the teaching principles in interactive operations. When explaining the structure of the ladder, the evolution of the universe, presenting the knowledge of the human brain structure, and introducing complex mechanical parts, the model uploaded before class is holographically projected through a holographic three-dimensional engine, so that students can intuitively see the holographic picture of the three-dimensional model. The synchronous storage module stores all real-time presented holographic pictures synchronously in real time for one month, so that some students who do not understand in class can increase their knowledge understanding by replaying.

[0022] Preferably, the real-time interaction unit includes a real-time perception module, an instant feedback module and a tracking and analysis module. The real-time perception module perceives the status of teachers and students in real time through a camera inside the classroom, and intelligently analyzes the attendance rate, head-up rate and interaction rate of students in local and remote classrooms, so as to provide data reference for process evaluation and continuous improvement of teaching quality. The instant feedback module enables teachers to use special classroom interactive software to post questions, discuss topics and conduct quizzes at any time, and students can respond through their own devices and ask teachers for help anytime and anywhere. The interactive software supports content display in various forms such as text, pictures, audio and video. The tracking and analysis module tracks students' learning status through data analysis tools, including learning progress, learning time and homework completion status. Teachers can clearly see the learning trajectory and learning status of each student on the screen to obtain students' needs and make plans for them.

[0023] The multimodal fusion method of smart classroom based on AI technology includes the following steps:

[0024] S1. Collecting image information: By deploying various image acquisition devices to collect stereoscopic image information, a stereoscopic model is synthesized in real time, and the holographic image of the teacher teaching in the local classroom is captured in real time. The blackboard and PPT projection screen used by the teacher and the audio information are transmitted to the remote classroom through the 5G network transmission system;

[0025] S2. Processing image information: Rendering and holographic feature preprocessing of the collected dynamic images, constructing the imaging field of the holographic virtual image, dynamically tracking the object trajectory and adjusting the light during the collection process, using communication technology to achieve real-time interactive transmission of two-way signals between the local classroom and the remote classroom, and building a holographic three-dimensional engine to complete resource retrieval operations;

[0026] S3, virtual restoration of images: the collected and processed holographic image of the lecturer is restored in a 1:1 three-dimensional virtual manner on the podium area of ​​the remote classroom with the help of holographic film polymer nano-optical materials. The remote classroom simultaneously displays the holographic teaching resource model and the three-dimensional image of the lecturer. The lecturer performs conventional interactive operations on the holographic teaching resource model when using it;

[0027] S4. Control equipment operation: responsible for centralized control and management of various types of equipment in the classroom, completing the opening and closing operations of various components of the system. The teacher can conveniently and quickly start and close the system with one key and switch between various functions. The classroom is equipped with a holographic image acquisition system and a holographic image restoration system. Each classroom is interconnected and uniformly controlled so that the holographic image restoration system of the teaching classroom and the remote listening classroom can also holographically present the holographic 3D model played by the teacher at the same time;

[0028] S5. Present holographic images: The classroom is divided into two areas, one side is the holographic image display area, and the other side is the student listening area. The holographic image display area maximizes the prominence of the virtual portrait, and the students in the listening area see the teacher's holographic image. The air conditioning, ventilation system and lighting are adaptively adjusted according to preset parameters to provide sufficient external learning conditions;

[0029] S6. Real-time monitoring of the lecture end: The video and audio of the students in the remote lecture classroom are collected and compressed by ultra-high-definition cameras and microphones, and then sent to the local classroom through the 5G network transmission system for signal decoding and media distribution. The lecture classroom uses surveillance cameras and wireless microphones to transmit the students' lecture images to the image acquisition end classroom. The video images of the students in the remote lecture classroom are displayed in real time through the monitor in front of the lecturer, which is convenient for the lecturer to grasp the lecture situation of the students in the remote classroom;

[0030] S7. Intuitive display of simulated teaching: Teachers use virtual environments to simulate various experimental environments, allowing students to observe and verify teaching principles in interactive operations. The models uploaded before class are projected holographically through a holographic 3D engine. All real-time holographic images will be stored synchronously in real time for one month for further study.

[0031] S8. Real-time interactive tracking of learning: By deploying cameras in the local classroom and the listening classroom, the status of teachers and students can be perceived in real time. Teachers can use professional classroom interactive software to post questions at any time and track students' learning status through data analysis tools. Students can respond through their own devices and ask teachers for help anytime and anywhere.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] 1. The present invention uses the cooperation of an image acquisition unit, an image processing unit, an image restoration unit and a device control unit to transmit images collected by a holographic image acquisition system of a teaching classroom to multiple remote classrooms for listening to lectures, and the holographic image restoration systems of the remote classrooms for listening to lectures restore and present the images. At the same time, the holographic image restoration systems of the teaching classroom and the remote classrooms also holographically present the holographic three-dimensional model played by the teacher. Students who attend lectures can not only display the holographic three-dimensional model with a naked-eye 3D effect, but also project the holographic three-dimensional image of the lecturer into the remote classroom, thereby enhancing the sense of presence and immersion of students who attend lectures remotely in remote interactive teaching, improving students' enthusiasm and initiative in learning, and facilitating the distance between students who attend lectures remotely and lecturers.

[0034] 2. The present invention divides the classroom into two areas, a holographic image display area and a student listening area, through an image presentation unit. Students can see the teacher's holographic image and the writing on the local classroom blackboard through the holographic image display area, and hear the teacher's stereo sound through the embedded speaker system. The air conditioner, ventilation system and lighting are adaptively adjusted according to preset parameters to adjust the temperature, humidity, air quality and illumination in the classroom. The real-time monitoring unit is used to facilitate the teaching teacher to understand the listening situation of students in the remote classroom. The intuitive teaching unit simulates various experimental environments with the help of a virtual environment, allowing students to observe and verify teaching principles in interactive operations. The real-time interactive unit provides data reference for process evaluation and continuous improvement of teaching quality, obtains each student's learning trajectory and learning status, and makes plans for students. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the overall system flow provided by an embodiment of the present invention;

[0036] Figure 2 A block diagram of the internal modules of the image acquisition unit and the image processing unit provided in the embodiment of the present invention;

[0037] Figure 3 A block diagram of the internal modules of the image restoration unit and the image presentation unit provided in an embodiment of the present invention;

[0038] Figure 4 A block diagram of the internal modules of the intuitive teaching unit and the real-time interactive unit provided in the embodiment of the present invention;

[0039] In the figure: 1. Image acquisition unit; 101. Image scanning module; 102. Image capture module; 103. Synchronous transmission module; 2. Image processing unit; 201. Synchronous transmission module; 202. Data compression module; 203. Interactive transmission module; 3. Image restoration unit; 301. Virtual restoration module; 302. Common display module; 303. Interactive operation module; 4. Equipment control unit; 5. Image presentation unit; 501. Area division module; 502. Area display module; 503. Virtual display module; 504. Adaptive adjustment module; 6. Real-time monitoring unit; 7. Intuitive teaching unit; 701. Equipment selection module; 702. Model demonstration module; 703. Synchronous storage module; 8. Real-time interaction unit; 801. Real-time perception module; 802. Instant feedback module; 803. Tracking and analysis module. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1-4 ,The present invention provides a technical solution: a smart classroom multimodal fusion system based on AI technology, including a device control unit 4 and a real-time monitoring unit 6;

[0042] Image acquisition unit 1, which collects stereoscopic image information by deploying various image acquisition devices, synthesizes a stereoscopic model in real time, and captures the holographic image of the teacher teaching in the local classroom in real time. The blackboard and PPT projection screen used by the teacher are transmitted to the remote classroom together with the audio information through the 5G network transmission system;

[0043] Image processing unit 2, which renders and pre-processes the collected dynamic images, constructs the imaging field of the holographic virtual image, dynamically tracks the object trajectory and adjusts the light during the collection process, uses communication technology to realize the real-time interactive transmission of two-way signals between the local classroom and the remote classroom, and constructs a holographic three-dimensional engine to complete the resource retrieval operation;

[0044] Image restoration unit 3, image restoration unit 3 restores the collected and processed holographic image of the lecturer in a 1:1 three-dimensional virtual manner in the podium area of ​​the remote classroom by means of holographic film polymer nano-optical materials, and the remote classroom simultaneously displays the holographic teaching resource model and the three-dimensional image of the lecturer, and the lecturer performs conventional interactive operations on the holographic teaching resource model when using it;

[0045] Image presentation unit 5: The image presentation unit 5 divides the classroom into two areas: one side is the holographic image display area, and the other side is the student listening area. The holographic image display area maximizes the virtual portrait, and the student listening area sees the teacher's holographic image. The air conditioner, ventilation system and lighting are adaptively adjusted according to preset parameters to provide sufficient external learning conditions.

[0046] Intuitive teaching unit 7, teachers use virtual environments to simulate various experimental environments, allowing students to observe and verify teaching principles in interactive operations. The models uploaded before class are projected holographically through a holographic 3D engine. All real-time holographic images will be stored synchronously in real time for one month for further study.

[0047] Real-time interactive unit 8, real-time interactive unit 8 realizes real-time perception of the status of teachers and students by deploying cameras in local classrooms and listening classrooms respectively. Teachers use professional classroom interactive software to post questions at any time and track students' learning status through data analysis tools. Students respond through their own devices and ask teachers for help anytime and anywhere.

[0048] The device control unit 4 is responsible for centralized control and management of various types of equipment in the classroom, and completes the opening and closing operations of various components of the system. The teacher can conveniently and quickly start and close the system with one key and switch between various functions. The classroom is deployed with a holographic image acquisition system and a holographic image restoration system. Each classroom is interconnected and uniformly controlled by the device control unit 4. The holographic image restoration system of the teaching classroom and the remote listening classroom also holographically presents the holographic 3D model played by the teacher;

[0049] The real-time monitoring unit 6 collects and compresses the video and audio of students in the remote classroom through an ultra-high-definition camera and a microphone, and then sends them to the local classroom through a 5G network transmission system for signal decoding and media distribution. The classroom uses a surveillance camera and a wireless microphone to transmit the student lecture screen to the image acquisition end classroom. The video screen of the students in the remote classroom is displayed in real time through the monitor in front of the lecturer, which is convenient for the lecturer to grasp the lecture situation of the students in the remote classroom;

[0050] The image acquisition unit 1 includes an image scanning module 101, an image capturing module 102 and a synchronous transmission module 103. The image scanning module 101 deploys a blackboard, a projector, a projection screen, a light enhancement matrix, a dual monitoring screen and an image acquisition camera in the classroom podium area, uses a high-sensitivity camera module to collect stereoscopic image information, and performs full-dimensional image scanning on the collected object to synthesize a stereoscopic model in real time. The image capturing module 102 captures the dynamic image of the teacher in the teaching classroom in real time and transmits it to the remote classroom. In the local classroom, the holographic image of the teaching teacher is collected and processed, and then compressed and encoded by the signal coding system together with the audio information. The synchronous transmission module 103 sends it to the remote classroom through the 5G network transmission system for signal decoding, media distribution and holographic image restoration. The teacher's blackboard writing and PPT projection screen in the local classroom are also transmitted synchronously, and the scene and lighting system are built in the classroom;

[0051] The image processing unit 2 includes a feature processing module 201, a data compression module 202 and an interactive transmission module 203. The feature processing module 201 renders and pre-processes the collected dynamic images with holographic features. By changing the color highlights, sharpening, contrast, shadows and edge enhancement, the images are given more visual features of 3D stereoscopic display, and the imaging field of the holographic virtual image is constructed. During the collection process, the data compression module 202 dynamically tracks the trajectory of the object and adjusts the light to maintain the balanced value of binocular difference. In addition, the image is encoded and compressed to reduce the amount of data transmitted and the network delay. The interactive transmission module 203 uses communication technology to realize the real-time interactive transmission of two-way signals between the local classroom and the remote classroom, ensuring the real-time and smooth interaction between the local classroom and the remote classroom in terms of three-dimensional images, audio and video signals.

[0052] The feature processing module includes feature processing algorithms, specifically:

[0053]

[0054] In the formula, g(X|μ i ,∑i) represents the i-th component in the feature model, x represents the three-dimensional continuous data feature, w i represents the mixture weight, i=1,···,M,μ i represents the mean vector, ∑i represents the covariance matrix, X j represents the value of pixel j, λ represents the value under specified conditions, and p(·) represents the weighted sum of feature components;

[0055] The data compression module includes compression algorithms, specifically:

[0056]

[0057] Where Q(u,v) represents the quantized coefficient amplitude, S(u,v) represents the quantization step size, which usually takes different values ​​depending on the position of the discrete cosine coding coefficient and the color component, F(u,v) represents the average quantization step size, Round(·) represents the coefficient quantization amplitude, u represents encoding, and v represents quantization.

[0058] The image restoration unit 3 includes a virtual restoration module 301, a common display module 302 and an interactive operation module 303. The virtual restoration module 301 uses a holographic film polymer nano-optical material to restore the collected and processed holographic image of the lecturer in a 1:1 three-dimensional virtual manner in the podium area of ​​the remote classroom. Remote students can achieve naked-eye three-dimensional visual effects without the help of any wearable devices. Through ultra-high-definition video capture technology, students can clearly see the facial expressions and body movements of the lecturer. The common display module 302 retrieves holographic teaching resources from the holographic three-dimensional engine at any time during the lecture, and displays them together with the lecturer in the local classroom. At the same time, the remote classroom also displays the holographic teaching resource model and the three-dimensional image of the lecturer. The interactive operation module 303 enables the lecturer to perform conventional interactive operations on the holographic teaching resource model, including displacement, rotation viewing, zooming in and out, and decomposition and synthesis of the model.

[0059] The image presentation unit 5 includes a region division module 501, a region display module 502, a virtual display module 503 and an adaptive adjustment module 504. The region division module 501 divides the classroom into two areas. One side is a holographic image display area for presenting a 1:1 holographic stereoscopic image and a holographic courseware model of the lecturer, and the other side is a student listening area. The region display module 502 makes a holographic image display area of ​​a cubic structure into a pure dark environment to reduce the interference of external ambient light and maximize the highlighting of the virtual portrait. Students see the holographic image of the teacher through this space structure and hear the stereo sound of the teacher through the embedded speaker system. The virtual display module 503 positions the image of the holographic teacher 1:1 standing in the middle of the podium and shifts it horizontally along the central axis. The local classroom blackboard writing is synchronously displayed in the virtual scene. The adaptive adjustment module 504 uses sensors to sense the temperature, humidity, air quality and light intensity in the classroom in real time, and adaptively adjusts the air conditioning, ventilation system and lighting according to preset parameters.

[0060] Intuitive teaching unit 7 includes device selection module 701, model demonstration module 702 and synchronous storage module 703. Device selection module 701 uses devices as needed. When holographic projection support is not needed, teachers and students only need to keep the devices turned off and carry out normal traditional teaching activities. According to the needs of teaching, holographic courseware is uploaded in advance. When the 3D model needs to be displayed, the device can be turned on with one click for three-dimensional presentation. In model demonstration module 702, teachers simulate various experimental environments with the help of virtual environments, allowing students to observe and verify teaching principles in interactive operations. When explaining the structure of the ladder, the evolution of the universe, the knowledge of the human brain structure, and the introduction of complex mechanical parts, the model uploaded before the class is holographically projected through a holographic three-dimensional engine, so that students can intuitively see the holographic picture of the three-dimensional model. Synchronous storage module 703 stores all real-time holographic pictures synchronously for one month, so that some students who do not understand in class can increase their knowledge understanding through replay.

[0061] The real-time interaction unit 8 includes a real-time perception module 801, an instant feedback module 802 and a tracking and analysis module 803. The real-time perception module 801 perceives the status of teachers and students in real time through a camera inside the classroom, and intelligently analyzes the attendance rate, head-up rate and interaction rate of students in local and remote classrooms, and provides data reference for process evaluation and continuous improvement of teaching quality. The instant feedback module 802 enables teachers to use special classroom interactive software to post questions, discuss topics and conduct quizzes at any time. Students respond through their own devices and ask teachers for help anytime and anywhere. The interactive software supports content display in multiple forms such as text, pictures, audio and video. The tracking and analysis module 803 tracks students' learning status through data analysis tools, including learning progress, learning time and homework completion status. Teachers can clearly see the learning trajectory and learning status of each student on the screen to obtain students' needs and make plans for them.

[0062] The multimodal fusion method of smart classroom based on AI technology includes the following steps:

[0063] S1. Collecting image information: By deploying various image acquisition devices to collect stereoscopic image information, a stereoscopic model is synthesized in real time, and the holographic image of the teacher teaching in the local classroom is captured in real time. The blackboard and PPT projection screen used by the teacher and the audio information are transmitted to the remote classroom through the 5G network transmission system;

[0064] S2. Processing image information: Rendering and holographic feature preprocessing of the collected dynamic images, constructing the imaging field of the holographic virtual image, dynamically tracking the object trajectory and adjusting the light during the collection process, using communication technology to achieve real-time interactive transmission of two-way signals between the local classroom and the remote classroom, and building a holographic three-dimensional engine to complete resource retrieval operations;

[0065] S3, virtual restoration of images: the collected and processed holographic image of the lecturer is restored in a 1:1 three-dimensional virtual manner on the podium area of ​​the remote classroom with the help of holographic film polymer nano-optical materials. The remote classroom simultaneously displays the holographic teaching resource model and the three-dimensional image of the lecturer. The lecturer performs conventional interactive operations on the holographic teaching resource model when using it;

[0066] S4. Control equipment operation: responsible for centralized control and management of various types of equipment in the classroom, completing the opening and closing operations of various components of the system. The teacher can conveniently and quickly start and close the system with one key and switch between various functions. The classroom is equipped with a holographic image acquisition system and a holographic image restoration system. Each classroom is interconnected and uniformly controlled so that the holographic image restoration system of the teaching classroom and the remote listening classroom can also holographically present the holographic 3D model played by the teacher at the same time;

[0067] S5. Present holographic images: The classroom is divided into two areas, one side is the holographic image display area, and the other side is the student listening area. The holographic image display area maximizes the prominence of the virtual portrait, and the students in the listening area see the teacher's holographic image. The air conditioning, ventilation system and lighting are adaptively adjusted according to preset parameters to provide sufficient external learning conditions;

[0068] S6. Real-time monitoring of the lecture end: The video and audio of the students in the remote lecture classroom are collected and compressed by ultra-high-definition cameras and microphones, and then sent to the local classroom through the 5G network transmission system for signal decoding and media distribution. The lecture classroom uses surveillance cameras and wireless microphones to transmit the students' lecture images to the image acquisition end classroom. The video images of the students in the remote lecture classroom are displayed in real time through the monitor in front of the lecturer, which is convenient for the lecturer to grasp the lecture situation of the students in the remote classroom;

[0069] S7. Intuitive display of simulated teaching: Teachers use virtual environments to simulate various experimental environments, allowing students to observe and verify teaching principles in interactive operations. The models uploaded before class are projected holographically through a holographic 3D engine. All real-time holographic images will be stored synchronously in real time for one month for further study.

[0070] S8. Real-time interactive tracking of learning: By deploying cameras in the local classroom and the listening classroom, the status of teachers and students can be perceived in real time. Teachers can use professional classroom interactive software to post questions at any time and track students' learning status through data analysis tools. Students can respond through their own devices and ask teachers for help anytime and anywhere.

[0071] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A smart classroom multimodal fusion system based on AI technology, comprising a device control unit (4) and a real-time monitoring unit (6), characterized in that: An image acquisition unit (1), wherein the image acquisition unit (1) acquires stereoscopic image information by deploying various image acquisition devices, synthesizes a stereoscopic model in real time, and captures a holographic image of a teacher teaching in a local classroom in real time. The blackboard writing and PPT projection screen used by the teacher are transmitted to a remote classroom together with audio information via a 5G network transmission system; An image processing unit (2) is used to render and pre-process the collected dynamic images, construct an imaging field of a holographic virtual image, dynamically track the trajectory of the object and adjust the light during the collection process, use communication technology to achieve real-time interactive transmission of two-way signals between the local classroom and the remote classroom, and simultaneously construct a holographic three-dimensional engine to complete resource retrieval operations; An image restoration unit (3), wherein the image restoration unit (3) restores the collected and processed holographic image of the lecturer in a 1:1 three-dimensional virtual manner in a podium area of ​​a remote classroom by means of a holographic film polymer nano-optical material, and the remote classroom simultaneously displays the holographic teaching resource model and the three-dimensional image of the lecturer, and the lecturer performs conventional interactive operations on the holographic teaching resource model when using it; An image presentation unit (5), wherein the image presentation unit (5) divides the classroom into two areas, one side being a holographic image display area and the other side being a student listening area, wherein the holographic image display area maximizes the prominence of the virtual human image, and the student listening area sees the holographic image of the teacher, and adaptively adjusts the air conditioning, ventilation system and lighting according to preset parameters to provide sufficient external learning conditions; Intuitive teaching unit (7), in which the teacher simulates various experimental environments with the help of a virtual environment, allowing students to observe and verify the teaching principles in interactive operations, and the model uploaded before class is holographically projected through a holographic three-dimensional engine, and all real-time holographic images are synchronously stored in real time for one month for further study; A real-time interactive unit (8), wherein the real-time interactive unit (8) senses the status of teachers and students in real time by deploying cameras in the local classroom and the classroom where the lecture is held. The teacher uses professional classroom interactive software to post questions at any time and tracks the students' learning status through data analysis tools. The students respond through their own devices and ask the teacher for help anytime and anywhere.

2. The AI-based smart classroom multimodal fusion system according to claim 1 is characterized by: The device control unit (4) is responsible for centralized control and management of various types of equipment in the classroom, and completes the opening and closing operations of various components of the system. The teacher can conveniently and quickly start and close the system with one key, and switch between various functions. The classroom is equipped with a holographic image acquisition system and a holographic image restoration system. Each classroom is interconnected and uniformly controlled by the device control unit (4). The holographic image restoration systems of the teaching classroom and the remote listening classroom also holographically present the holographic three-dimensional model played by the teacher. The real-time monitoring unit (6) collects and compresses the video and audio of students in the remote classroom through an ultra-high-definition camera and a microphone, and then sends them to the local classroom through a 5G network transmission system for signal decoding and media distribution. The classroom uses a monitoring camera and a wireless microphone to transmit the student listening video to the image acquisition end classroom. The video of the students in the remote classroom is displayed in real time through a monitor in front of the lecturer, so that the lecturer can understand the listening situation of the students in the remote classroom.

3. The AI-based smart classroom multimodal fusion system according to claim 1 is characterized by: The image acquisition unit (1) comprises an image scanning module (101), an image capturing module (102) and a synchronous transmission module (103). The image scanning module (101) deploys a blackboard, a projector, a projection screen, a light enhancement matrix, a dual monitoring screen and an image acquisition camera in the classroom podium area, uses a high-sensitivity camera module to collect stereoscopic image information, and performs full-dimensional image scanning on the collected object to synthesize a stereoscopic model in real time. The image capturing module (102) captures the dynamic image of the teacher in the teaching classroom in real time and transmits it to the remote classroom. In the local classroom, the holographic image of the teaching teacher is collected and processed, and then compressed and encoded by the signal encoding system together with the audio information. The synchronous transmission module (103) is sent to the remote classroom through the 5G network transmission system for signal decoding, media distribution and holographic image restoration. The teacher's blackboard writing and PPT projection screen in the local classroom are also transmitted synchronously, and a scene and lighting system are built in the classroom.

4. The AI-based smart classroom multimodal fusion system according to claim 1 is characterized by: The image processing unit (2) comprises a feature processing module (201), a data compression module (202) and an interactive transmission module (203). The feature processing module (201) renders and pre-processes the collected dynamic images for holographic features, and makes the images more visually characteristic of 3D stereoscopic display by changing color highlights, sharpening, contrast, shadows and edge enhancement, thereby constructing an imaging field of a holographic virtual image. The data compression module (202) dynamically tracks the trajectory of the object and adjusts the light during the collection process to maintain a balanced value of binocular difference. In addition, the image is encoded and compressed to reduce the amount of data transmitted and reduce network delay. The interactive transmission module (203) uses communication technology to realize real-time interactive transmission of two-way signals between the local classroom and the remote classroom, thereby ensuring the real-time and smooth interaction between the local classroom and the remote classroom in terms of three-dimensional images, audio and video signals.

5. The AI-based smart classroom multimodal fusion system according to claim 1 is characterized by: The image restoration unit (3) comprises a virtual restoration module (301), a common display module (302) and an interactive operation module (303). The virtual restoration module (301) uses a holographic film polymer nano-optical material to perform a 1:1 three-dimensional virtual restoration of the holographic image of the lecturer after acquisition and processing in the podium area of ​​the remote classroom. Remote students can achieve a naked-eye three-dimensional visual effect without the help of any wearable equipment. Through ultra-high-definition video capture technology, the students can clearly see the facial expressions and body movements of the lecturer. The common display module (302) retrieves holographic teaching resources from the holographic three-dimensional engine at any time during the lecture, and displays them together with the lecturer in the local classroom. At the same time, the remote classroom also displays the holographic teaching resource model and the three-dimensional image of the lecturer. The interactive operation module (303) enables the lecturer to perform conventional interactive operations on the holographic teaching resource model, including displacement, rotation viewing, zooming in and out, and decomposition and synthesis of the model.

6. The AI-based smart classroom multimodal fusion system according to claim 1 is characterized by: The image presentation unit (5) comprises an area division module (501), an area display module (502), a virtual display module (503) and an adaptive adjustment module (504). The area division module (501) divides the classroom into two areas, one side is a holographic image display area for presenting a 1:1 holographic stereoscopic image and a holographic courseware model of the lecturer, and the other side is a student listening area. The area display module (502) creates a pure dark environment in a holographic image display area with a cubic structure to reduce the interference of external ambient light and maximize the highlighting of the virtual portrait. Students see the holographic image of the teacher through this spatial structure and hear the stereo sound of the teacher through an embedded speaker system. The virtual display module (503) positions the image of the holographic teacher 1:1 standing in the middle of the podium and shifts it horizontally along the central axis. The local classroom blackboard writing is synchronously displayed in the virtual scene. The adaptive adjustment module (504) senses the temperature, humidity, air quality and light intensity in the classroom in real time through sensors, and adaptively adjusts the air conditioner, ventilation system and lighting according to preset parameters.

7. The AI-based smart classroom multimodal fusion system according to claim 1 is characterized by: The intuitive teaching unit (7) includes a device selection module (701), a model demonstration module (702) and a synchronous storage module (703). The device selection module (701) uses the device as needed. When the holographic projection support is not needed, teachers and students only need to keep the device turned off and carry out normal traditional teaching activities. According to the teaching needs, the holographic courseware is uploaded in advance. When the 3D model needs to be displayed, the device is turned on with one click for three-dimensional presentation. In the model demonstration module (702), the teacher simulates various experimental environments with the help of a virtual environment, allowing students to observe and verify the teaching principles in interactive operations. When explaining the structure of the ladder, the evolution of the universe, the knowledge of the human brain structure, and the introduction of complex mechanical parts, the model uploaded before the class is holographically projected through a holographic three-dimensional engine, so that students can intuitively see the holographic picture of the three-dimensional model. The synchronous storage module (703) stores all the real-time presented holographic pictures in real time and synchronously for one month, so that some students who do not understand in class can increase their knowledge understanding by replaying.

8. The AI-based smart classroom multimodal fusion system according to claim 1 is characterized by: The real-time interaction unit (8) comprises a real-time perception module (801), an instant feedback module (802) and a tracking and analysis module (803). The real-time perception module (801) perceives the status of teachers and students in real time through a camera inside the classroom, and intelligently analyzes the attendance rate, head-up rate and interaction rate of students in the local classroom and the remote classroom, so as to provide data reference for process evaluation and continuous improvement of teaching quality. The instant feedback module (802) enables teachers to use special classroom interactive software to post questions, discuss topics and conduct quizzes at any time, and students can respond through their own devices and ask teachers for help anytime and anywhere. The interactive software supports content display in multiple forms such as text, pictures, audio and video. The tracking and analysis module (803) tracks students' learning status, including learning progress, learning time and homework completion status, through data analysis tools. Teachers can clearly see the learning trajectory and learning status of each student on the screen to understand students' needs and make plans for them.

9. The multimodal fusion method of smart classroom based on AI technology is characterized by: The AI-based smart classroom multimodal fusion method has the AI-based smart classroom multimodal fusion system described in any one of 1-8, and includes the following steps: S1. Collecting image information: By deploying various image acquisition devices to collect stereoscopic image information, a stereoscopic model is synthesized in real time, and the holographic image of the teacher teaching in the local classroom is captured in real time. The blackboard and PPT projection screen used by the teacher and the audio information are transmitted to the remote classroom through the 5G network transmission system; S2. Processing image information: Rendering and holographic feature preprocessing of the collected dynamic images, constructing the imaging field of the holographic virtual image, dynamically tracking the object trajectory and adjusting the light during the collection process, using communication technology to achieve real-time interactive transmission of two-way signals between the local classroom and the remote classroom, and building a holographic three-dimensional engine to complete resource retrieval operations; S3, virtual restoration of images: the collected and processed holographic image of the lecturer is restored in a 1:1 three-dimensional virtual manner on the podium area of ​​the remote classroom with the help of holographic film polymer nano-optical materials. The remote classroom simultaneously displays the holographic teaching resource model and the three-dimensional image of the lecturer. The lecturer performs conventional interactive operations on the holographic teaching resource model when using it; S4. Control equipment operation: responsible for centralized control and management of various types of equipment in the classroom, completing the opening and closing operations of various components of the system. The teacher can conveniently and quickly start and close the system with one key and switch between various functions. The classroom is equipped with a holographic image acquisition system and a holographic image restoration system. Each classroom is interconnected and uniformly controlled so that the holographic image restoration system of the teaching classroom and the remote listening classroom can also holographically present the holographic 3D model played by the teacher at the same time; S5. Present holographic images: The classroom is divided into two areas, one side is the holographic image display area, and the other side is the student listening area. The holographic image display area maximizes the prominence of the virtual portrait, and the students in the listening area see the teacher's holographic image. The air conditioning, ventilation system and lighting are adaptively adjusted according to preset parameters to provide sufficient external learning conditions; S6. Real-time monitoring of the lecture end: The video and audio of the students in the remote lecture classroom are collected and compressed by ultra-high-definition cameras and microphones, and then sent to the local classroom through the 5G network transmission system for signal decoding and media distribution. The lecture classroom uses surveillance cameras and wireless microphones to transmit the students' lecture images to the image acquisition end classroom. The video images of the students in the remote lecture classroom are displayed in real time through the monitor in front of the lecturer, which is convenient for the lecturer to grasp the lecture situation of the students in the remote classroom; S7. Intuitive display of simulated teaching: Teachers use virtual environments to simulate various experimental environments, allowing students to observe and verify teaching principles in interactive operations. The models uploaded before class are projected holographically through a holographic 3D engine. All real-time holographic images will be stored synchronously in real time for one month for further study. S8. Real-time interactive tracking of learning: By deploying cameras in the local classroom and the listening classroom, the status of teachers and students can be perceived in real time. Teachers can use professional classroom interactive software to post questions at any time and track students' learning status through data analysis tools. Students can respond through their own devices and ask teachers for help anytime and anywhere.

Images