Intelligent teacher system and method for fusion teaching of virtual human and real teacher

Abstract

The invention belongs to the technical field of smart education and virtual reality, and discloses a smart teacher system and method for fusion teaching of virtual people and real teachers, and the system comprises an MR access synchronous classroom module which is used for accessing a video stream of a first visual angle in MR equipment into a synchronous classroom for live broadcast; the virtual human interaction module is used for interacting with a virtual human through voice, and the virtual human interaction module comprises a voice control SDK (Software Development Kit) and client voice interaction logic; according to the invention, the live broadcast time delay is relatively small, and students can conveniently observe the live broadcast demonstration of teachers through the smart blackboard screen; the function that the user interacts with the virtual human through voice can be realized; according to the technical scheme, when the user moves or the environment changes, the position of the virtual content can be automatically adjusted, synchronism of the virtual content and the real world is ensured, smooth and efficient remote interaction experience can be provided for the user, and seamless connection and intelligent assistance of teaching activities are achieved.

Description

Technical Field

[0001] The present invention relates to the fields of smart education and virtual reality technology, and more specifically, to a smart teacher system and method for teaching with virtual humans and real teachers. Background Art

[0002] With the rapid development of information technology, the field of education is undergoing unprecedented changes. Traditional teaching methods can no longer meet the current diversified and personalized teaching needs. With the continuous innovation of virtual reality (VR) and augmented reality (AR) technologies, the field of smart education is gradually integrating virtual and real teaching scenes, forming a collaborative teaching model between virtual teachers and real teachers. As an important tool to enhance interactive experience and improve learning effects, virtual human technology has been widely used in distance learning, online education and virtual classrooms.

[0003] However, when existing virtual humans and real teachers are integrated into teaching, there are often problems such as large live broadcast delays, insufficient interactivity, and poor synchronization, which leads to a poor teaching experience for students; and it is also inconvenient to correct spatial coordinates in real time.

[0004] In view of this, the present invention proposes an intelligent teacher system and method for integrated teaching of virtual humans and real teachers to solve the above problems. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions: a smart teacher system and method for integrated teaching of virtual people and real teachers, 1. A smart teacher system for integrated teaching of virtual people and real teachers, characterized in that it includes:

[0006] MR access synchronous classroom module, which is used to connect the first-person video stream in the MR device to the synchronous classroom for live broadcast;

[0007] A virtual human interaction module, which is used to interact with a virtual human through voice, and the virtual human interaction module includes a voice control SDK and client voice interaction logic;

[0008] A spatial coordinate real-time correction module, which is used to dynamically adjust the spatial coordinates;

[0009] Pixel Streaming Service module, which is used to establish the Pixel Streaming service on the Ubuntu system;

[0010] The smart teacher module is used to integrate pre-class preparation, classroom auxiliary teaching and after-class tutoring in teaching activities.

[0011] Furthermore, the method of connecting the first-perspective video stream in the MR device to the synchronous classroom for live broadcast includes:

[0012] The instructor conducts a live demonstration through the MR device, which starts the video stream and projects it onto the laptop. The laptop starts the synchronous classroom client and synchronizes the local video stream to the server. Then, the synchronous classroom client pulls the live video stream to observe the live broadcast, and the students watch the instructor's live demonstration through the synchronous classroom client.

[0013] Furthermore, the voice control SDK includes three working states: STATE_IDLE, STATE_READY and STATE_WORKING.

[0014] Furthermore, the client voice interaction logic includes:

[0015] The client needs to integrate the wake-up SDK and voice control SDK;

[0016] The client establishes a websocket connection with the speech analysis service and performs login authentication.

[0017] Furthermore, the method of interacting with the virtual person through voice includes:

[0018] When the user says the wake-up word, the wake-up SDK throws a wake-up event, the client generates a wake-up id, and then the client sends the wake-up event to the speech analysis service. Then the client calls the pixel push service to play the virtual person's answering words;

[0019] After the hololens finishes playing the wake-up response, it sends a voice stream to the voice analysis service. When sending the voice stream, it needs to carry the voice stream ID until the backend responds to the command to stop the voice stream.

[0020] After the speech analysis service receives the voice stream from the HoloLens, it eventually responds to the HoloLens with a set of instructions.

[0021] Furthermore, the method of dynamically adjusting the spatial coordinates includes:

[0022] By placing a holographic object in HoloLens and fixing the position and rotation of the holographic object by adding a spatial anchor point;

[0023] Use HoloLens' built-in sensors and cameras to capture and analyze environmental information, and create a spatial map by scanning the surrounding environment and identifying feature points;

[0024] Add a spatial anchor component to the root GameObject of the holographic object, and attach a spatial anchor component with a relative position offset to its child GameObject;

[0025] Use HoloLens to capture and identify QR codes in the scene, and use the camera and image processing system to detect and decode the information in the QR code in real time. After identifying the QR code, the location information can be obtained.

[0026] Subsequently, the position and orientation of the virtual content in the mixed reality environment are adjusted according to the position information of the QR code.

[0027] Furthermore, the method of establishing the pixel streaming service on the Ubuntu system includes:

[0028] Build a server environment that supports pixel streaming and deploy applications and operations that need to be streamed on the server;

[0029] Then, after deploying the corresponding version of Unreal Engine on the Ubuntu system, it is transmitted to the terminal device in the form of a video stream through the pixel streaming plug-in for display and operation.

[0030] Furthermore, the pre-class preparation includes a learning situation analysis unit, a teaching activity classification & intelligent test paper composition & auxiliary review unit, a knowledge base and intelligent question and answer unit, a resource library and resource retrieval unit, and a word cloud analysis unit;

[0031] The classroom auxiliary teaching includes a virtual-reality integrated scene generation unit, an MR & tablet simulation training unit, a large-screen queuing and calling unit, an MR mixed reality training unit, and a large-screen step statistics unit;

[0032] The after-class tutoring includes a class content summary unit, a personalized resource recommendation unit, a class quality analysis unit, an electronic teaching plan unit and a statistical analysis unit.

[0033] The smart teacher method for integrated teaching of virtual humans and real teachers includes the following steps:

[0034] S1, connect the first-person video stream in the MR device to the synchronous classroom for live broadcast;

[0035] S2. Interacting with a virtual human through voice, wherein the virtual human interaction module includes a voice control SDK and client voice interaction logic;

[0036] S3, dynamically adjust the spatial coordinates;

[0037] S4. Establish pixel streaming service on Ubuntu system;

[0038] S5. Integrate teaching activities including pre-class preparation, classroom auxiliary teaching and after-class tutoring.

[0039] Technical effects and advantages of the intelligent teacher system and method for integrated teaching of virtual humans and real teachers of the present invention:

[0040] The present invention can reduce the live broadcast delay, making it convenient for students to observe the instructor's live demonstration through the smart blackboard screen; it can control the audio and video equipment and interact with the back-end voice analysis service, thereby realizing the function of users interacting with virtual people through voice; it can automatically adjust the position of virtual content when the user moves or the environment changes to ensure its synchronization with the real world, can transmit the virtual human application screen on the server to the remote client device in real time, and provide users with a smooth and efficient remote interaction experience, can achieve seamless connection and intelligent assistance of teaching activities, and ensure continuous optimization and improvement of teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a structural schematic diagram of the intelligent teacher system for virtual human and real teacher integrated teaching of the present invention;

[0042] Figure 2 A flow chart of the intelligent teacher method for integrated teaching of virtual humans and real teachers according to the present invention;

[0043] Figure 3 It is a schematic diagram of the bridging solution in the MR access synchronous classroom module of the present invention;

[0044] Figure 4 A physical topology diagram of the bridging solution in the MR access synchronous classroom module of the present invention;

[0045] Figure 5 This is a schematic diagram of the state conversion relationship of the voice control SDK of the present invention;

[0046] Figure 6 This is a schematic diagram of the client voice interaction logic of the present invention;

[0047] Figure 7 A schematic diagram of the voice interaction logic of the present invention;

[0048] Figure 8 Schematic diagram of the smart teacher module of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] Example 1

[0051] See also Figure 1 , Figures 3 to 8As shown, the smart teacher system for virtual human and real teacher integrated teaching described in this embodiment includes:

[0052] MR access synchronous classroom module, which is used to connect the first-person video stream in the MR device to the synchronous classroom for live broadcast. Among them, MR can only provide video streams in webrtc format, and the synchronous classroom can only receive rtsp video streams;

[0053] A virtual human interaction module, which is used to interact with a virtual human through voice. The virtual human interaction module includes a voice control SDK and client voice interaction logic;

[0054] A spatial coordinate real-time correction module, which is used to dynamically adjust the spatial coordinates;

[0055] Pixel Streaming Service module, which is used to establish the Pixel Streaming service on the Ubuntu system;

[0056] The smart teacher (teacher) module is used to integrate pre-class preparation, classroom auxiliary teaching and after-class tutoring in teaching activities.

[0057] Furthermore, if Figure 3 As shown, the method of connecting the first-person video stream in the MR device to the synchronous classroom for live broadcast includes:

[0058] The instructor conducts a live demonstration through the MR device. The MR device starts the video stream and projects it to the laptop. The laptop starts the synchronous classroom client and synchronizes the local video stream to the server. Then, the synchronous classroom client pulls the live video stream to observe the live broadcast. The students watch the instructor's live demonstration through the synchronous classroom client.

[0059] Specifically, this live broadcast solution has a relatively small delay and no obvious freezes in the picture, but the process logic is relatively complex and the link is relatively long, so this solution is selected;

[0060] Among them, the screen projection solution is:

[0061] Confirm that the device has a built-in unlimited screen projection function, which requires Internet access to download (or install third-party software);

[0062] Screen projection depends on the network card, and the device network card must support the Miracast protocol;

[0063] The physical topology of live broadcast is:

[0064] The physical topology of the entire live broadcast is as follows: Figure 4 As shown, the field is connected to the teaching building network through optical fiber, and the field equipment is connected to the local area network;

[0065] The equipment in the preparation area includes an 86-inch all-in-one computer, a podium host and an audio host, and the equipment outside includes MR equipment, a live broadcast notebook, and Bluetooth headphones;

[0066] The instructor broadcasts live through the MR device outside the experiment field, and the MR device synchronizes the video stream to the laptop. The AI ​​classroom client on the laptop will take the video stream on the local computer and upload it to the server. The AI ​​classroom client on the smart blackboard in the preparation area outside the experiment field will pull the live stream of the server, and the students will observe the instructor's live demonstration through the smart blackboard screen.

[0067] Furthermore, the voice control SDK includes three working states: STATE_IDLE (service not started), STATE_READY (awakening state) and STATE_WORKING (working state), which are described in the following table. Specifically, the SDK is in different states at different stages during operation, and different states can handle different operations:

[0068] Status Name illustrate STATE_IDLE If the service is not started, you can only perform the start (start service) operation. STATE_READY In the waiting state, you can wake up the service by voice (wake-up word) or by sending a CMD_WAKEUP message directly to the service. After calling AIUIAgent.createAgent to create an object, the service is in the ready state. STATE_WORKING In working state, you can input voice and text to interact with the AIUI background.

[0069] The client can control the running state of the SDK through SDKMessage. When SDKMessage is just created, the SDK is in the service-inactive state. After sending the CMD_START message, the SDK will switch to the wake-up state. When the SDK receives the wake-up command, the SDK will switch to the working state. At this time, it can interact with the backend service through voice or text. The specific conversion relationship is as follows: Figure 5 As shown, the states of the state transition diagram are as follows:

[0070] Operation Name illustrate start Default state after startup or send CMD_START message to SDK. stop Send a CMD_STOP message to the SDK. wakeup Say a custom wake-up word (the default is "ding dong ding dong"), or send a CMD_WAKEUP message to the SDK. reset_wakeup Send a CMD_RESET_WAKEUP message to the SDK. sleep Sleep, when no valid interaction (semantics) occurs for a period of time. re_wakeup In the STATE_WORKING state, say the wake-up word again or send a CMD_WAKEUP message to the SDK.

[0071] Specifically, the status query can be sent to the SDK by constructing a CMD_GET_STATE query message. The SDK will return the current status through the EVENT_STATE event. The arg1 parameter of the EVENT_STATE event indicates that the status value has the following values:

[0072] 1=>STATE_IDLE (idle state);

[0073] 2=>STATE_READY (ready state, waiting to be woken up);

[0074] 3=>STATE_WORKING (working state, awakened).

[0075] Open and close the SDK by using CMD_START and CMD_STOP to control the start and stop of the SDK;

[0076] SDK does not have any operation in the stopped state, and the power consumption is also the lowest at this time. It cannot be awakened in the stopped state and needs to enter the ready state through CMD_START to wake up;

[0077] CMD_RESET is used to reset the service in case of a fatal error that cannot be recovered or to re-read the configuration file.

[0078] SDK wake-up and sleep are as follows:

[0079] When the SDK is in sleep mode, you can wake it up by saying a custom wake-up word (such as "Hello, Qihang") or sending a CMD_WAKEUP message to the SDK.

[0080] After entering the working state, you can interact through voice or text, but if there is no effective interaction for a period of time (configurable in the configuration file interact_timeout), it will enter the ready state. You can also enter the dormant state by manually sending CMD_RESET_WAKEUP;

[0081] Both of the above sleep modes will throw out the EVENT_SLEEP event to indicate that the SDK has entered the sleep state, and the arg1 field indicates the mode of entering the sleep state.

[0082] 0=>TYPE_AUTO (automatic sleep, i.e. interaction timeout),

[0083] 1=>TYPE_COMPEL (external forced sleep, i.e. sending CMD_RESET_WAKEUP).

[0084] The wake-up result is as follows. After the SDK enters the wake-up state, the corresponding wake-up event is thrown through the EVENT_WAKEUP type message.

[0085] Furthermore, the client voice interaction logic includes:

[0086] The client needs to integrate the wake-up SDK and voice control SDK;

[0087] The client establishes a websocket connection with the voice analysis service and performs login authentication. The authentication information includes device ID, login name, login password, user information, etc.

[0088] Specifically, the client carries the image of a virtual person, can control the audio and video equipment and interact with the back-end voice analysis service, so as to realize the function of users interacting with the virtual person through voice. Figure 6 shown.

[0089] Furthermore, if Figure 7As shown, the ways to interact with virtual people through voice include:

[0090] Wake-up: When the user says the wake-up word, the wake-up SDK throws a wake-up event, and the client generates a wake-up id (wakeid). The wake-up id must be unique and non-duplicate. Then the client sends the wake-up event to the speech analysis service, which must carry the device id, login information, user information, etc. Then the client calls the pixel push service to play the virtual person's response (for example: Hello, I'm here);

[0091] Send the voice stream. After the hololens plays the wake-up response, it sends the voice stream to the voice analysis service. When sending the voice stream, the voice stream ID (asrid) must be carried. The ID must be unique and non-repeated. It is continuously sent until the backend responds to the command to stop the voice stream.

[0092] Respond to the instruction set. After receiving the voice stream from HoloLens, the voice analysis service will perform voice recognition, semantic understanding, dialogue processing, etc., and finally respond to the instruction set to HoloLens. The instruction set includes displaying the recognition result, stopping the voice stream, playing the speech, sending the voice stream, opening the browser, opening the file, opening the audio and video, opening a link, etc. HoloLens executes the instruction set in sequence;

[0093] Specifically, the specific explanation of each instruction is as follows: Display recognition results: display the speech recognition results at the appropriate position of HoloLens; Stop voice stream: HoloLens stops inputting voice stream to the speech analysis service; Play speech: HoloLens calls the pixel stream service collection speech synthesis engine to play the virtual human audio and video stream speech; Send voice stream: The speech analysis service returns an instruction set, which will contain multiple instructions. If it is a multi-round conversation, the returned instruction set will first stop the voice stream, play the speech, and then continue to send the voice stream to the speech analysis service, and then interact with the speech analysis service for multiple rounds. When executing this instruction, a new voice stream id needs to be regenerated; Open browser: open the HoloLens local browser; Open file: open a specific file; Open audio and video stream: open an audio and video file; Open a link: open a connection address, etc.

[0094] Furthermore, the method of dynamically adjusting the spatial coordinates includes:

[0095] By placing a holographic object in HoloLens and fixing its position and rotation by adding spatial anchors (so that the holographic object maintains its relative position even when the user moves their head or changes their viewing angle);

[0096] Use HoloLens’ built-in sensors and cameras to capture and analyze environmental information, and create a spatial map by scanning the surrounding environment and identifying feature points (these feature points can be walls, furniture, or other fixed objects in the room; once the spatial map is created, the precise location of the holographic objects can be determined based on the feature points in the map);

[0097] Add a spatial anchor component to the root GameObject of the holographic object, and attach a spatial anchor component with a relative position offset to its child GameObject (in HoloLens development, we implement spatial anchors by writing scripts. When a holographic object is placed in a scene, its position and rotation state will be anchored to a specific spatial point). When the user leaves the scene or closes the application, the hologram in the scene will be saved at the location where it was placed (and thus, when the user re-enters the scene or reopens the application, the holographic content in the previous scene can be accurately restored);

[0098] Use HoloLens to capture and identify QR codes in the scene, and use the camera and image processing system to detect and decode the information in the QR code in real time. After identifying the QR code, the location information can be obtained (HoloLens corrects the spatial coordinates through spatial calculation and coordinate conversion based on the extracted location information);

[0099] Subsequently, the position and orientation of the virtual content (model and interface) in the mixed reality environment are adjusted according to the position information of the QR code to make it consistent with the real environment;

[0100] Specifically, HoloLens uses its built-in multiple cameras and sensors to capture and analyze information about the surrounding environment. These cameras can capture depth information, RGB images, and infrared data, while sensors are used to detect movements such as movement, rotation, and tilt of the device. Based on the collected environmental data, HoloLens creates a virtual spatial map, which not only contains the location of objects, but also their shape, size, and position relative to the user. Once the spatial map is created, HoloLens begins to track the user's head and hand movements in real time, which is usually done through sensors such as accelerometers, gyroscopes, and magnetometers. Through these data, HoloLens can Update the user's position and orientation in the virtual space; when the user moves or changes their head posture, HoloLens will dynamically calibrate the spatial map based on its real-time tracking data, which means that the virtual content will be adjusted in real time according to the user's actual position and head orientation to ensure that they are always consistent with the user's perspective; HoloLens also includes a user feedback mechanism that allows users to fine-tune spatial correction through gestures or voice commands. The interactive feedback loop helps improve the accuracy of spatial correction and user experience; furthermore, by adopting real-time correction technology for spatial coordinates by scanning QR codes in real time, the synchronization between virtual content and the real environment in the project is improved, providing users with a more natural and realistic experience;

[0101] It should be noted that the above content is continuously detected and recognized by the HoloLens device during operation, and the spatial coordinates are dynamically adjusted according to the location information of the QR code, so as to ensure the accurate alignment of the virtual content with the real environment. It can automatically adjust the position of the virtual content when the user moves or the environment changes to ensure its synchronization with the real world; and when it comes to process operations and interaction with the real cockpit and helicopter, it can maximize the accuracy of the system's virtual and real superposition.

[0102] Furthermore, the method of establishing the pixel streaming service on the Ubuntu system includes:

[0103] Build a server environment that supports pixel streaming (the server environment includes installing and configuring the necessary software and hardware resources), and deploy the applications and operations that need to be streamed on the server;

[0104] After that, the corresponding version of Unreal Engine is deployed on the Ubuntu system, and the video is transmitted to the terminal device (client) in the form of video stream through the pixel streaming plug-in. The client device can be any device that supports a web browser, such as a computer, tablet or mobile phone. The client accesses the web page on the web server through the browser and receives the video stream from the server. The client currently used is hololens. On the client device, you can use HTML5 <video>Tags or other related technologies to display the received video stream) for display and operation (the Pixel Streaming Plugin is part of the Unreal Engine. The Pixel Streaming Plugin is responsible for capturing the rendering output of the application and encoding it into a video stream; further, the Pixel Streaming Plugin runs on the server side and can send the real-time image of the application to the client with minimal delay through efficient encoding and decoding algorithms), which also includes a signaling server deployed on the Ubuntu system, which uses the WebSocket communication protocol;

[0105] Specifically, through the collaborative work of the above components, the pixel streaming technology architecture based on the Ubuntu system can transmit the virtual human application screen on the server to the remote client device in real time, providing users with a smooth and efficient remote interaction experience.

[0106] Furthermore, if Figure 8 As shown in the figure, the pre-class preparation includes the learning situation analysis unit, teaching activity classification & intelligent test composition & auxiliary review unit, knowledge base and intelligent question and answer unit, resource library and resource retrieval unit and word cloud analysis unit:

[0107] Among them, the learning situation analysis unit is used to collect students' test data, evaluate students' knowledge mastery and track students' weaknesses. Specifically, by collecting and analyzing students' test data in real time, teachers can save time. Through evaluation, students' real-time knowledge mastery can be accurately captured. Through weak point tracking, weak knowledge points of each class can be summarized and updated to support effect verification and improvement.

[0108] The teaching activity classification & intelligent test-taking & auxiliary review unit is used to distinguish the types of learning activities and monitor the learning progress and duration in real time. The objective questions set are used to obtain the students' mastery of knowledge points. The automatic test-taking is used to intelligently select questions, which reduces the burden of teachers in test-taking and ensures comprehensive coverage of knowledge points. The intelligent auxiliary review is used to intelligently compare the key points of students' answers with the standard answers and give AI scores.

[0109] The knowledge base and intelligent question-answering unit allows administrators to upload teaching materials, and the intelligent teacher module automatically learns and generates a structured knowledge base. When students ask questions through voice, the intelligent teacher answers based on the learned knowledge base content. This flexible and convenient learning method can enhance students' problem-raising and problem-solving experience.

[0110] In the resource library and resource search unit, teachers and students can quickly find specific resources through voice commands; and it also includes resource classification management of the resource library, so as to enhance the search function and improve the utilization rate of teaching resources;

[0111] In the word cloud analysis unit, students share their questions and ideas in real time through word clouds; this makes it easier for students to express their doubts, and teachers can understand students' thoughts and needs before, during and after class in real time. It can also help optimize teaching content and interaction methods and improve teaching effectiveness.

[0112] Classroom auxiliary teaching includes virtual-reality integrated scene generation unit, MR & tablet simulation training unit, large-screen queuing and calling unit, MR mixed reality training unit and large-screen step counting unit;

[0113] In the virtual-reality scene generation unit, the instructor's perspective is projected in real time, so that students can participate in discussions in person, deepen their memory and sense of interaction, and enhance the display of teaching content through MR, which promotes students' in-depth understanding and application, triggers in-depth discussions, and thus transcends traditional spatial limitations. Students can learn in different places, enriching their teaching experience and scene applications;

[0114] In the MR & flat panel simulation training unit, students use simulation technology to simulate various flight environments and unexpected situations, thereby enhancing their emergency handling capabilities, allowing them to learn in a safe environment, reduce operational risks, and reduce personal injuries and equipment damage; by using customized software and general hardware, they can reduce dependence on complex equipment and simplify system updates and maintenance;

[0115] In the large-screen queuing and calling unit, students can automatically arrange the best time and equipment through the intelligent queuing system. The automatic calling is combined with the instructor's designation, thereby avoiding resource waste and time conflicts, eliminating the inefficiency and confusion of traditional scheduling, ensuring the effective use of scarce resources and classroom time, and optimizing training efficiency;

[0116] In the MR mixed reality training unit, trainees can reduce equipment operation errors and improve training effects and efficiency through MR glasses combined with digital human guidance. By combining real equipment with virtual elements, it can provide training guidance for in-depth understanding of equipment operation and enhance trainees' training experience. Through trainees' contact with and use of MR technology, they can cultivate innovative thinking and the ability to adapt to new technologies. Furthermore, MR technology can help trainees adapt to high-tech warfare in advance and enhance their ability to respond to future battlefields.

[0117] In the large-screen step statistics unit, real-time operation feedback, data visualization analysis, and automatic recording and archiving are performed on the students' immediate operation results. As a result, students can obtain intuitive and immediate operation results, promote error learning and skill improvement, and instructors optimize training through data analysis to improve teaching effectiveness and student skills. All operation records are automatically saved and available for review and analysis at any time, supporting personalized training and skill development.

[0118] After-class tutoring includes a class content summary unit, a personalized resource recommendation unit, a class quality analysis unit, an electronic teaching plan unit, and a statistical analysis unit;

[0119] In the course content summary unit, by quickly summarizing the key points of the course, time and energy can be saved, and the tediousness of reviewing the entire video can be avoided. The knowledge structure can deepen understanding and memory, and help grasp the core concepts.

[0120] In the personalized resource recommendation unit, relevant resources are recommended according to the learning situation, thereby realizing personalized resource recommendation, and through intelligent matching of video clips and exercises, accurate weak point review is carried out, so that knowledge gaps can be accurately supplemented and learning effects can be improved; by optimizing the learning path, ineffective learning time can be reduced, a complete knowledge system can be built, and students' confidence and academic performance can be improved;

[0121] In the classroom quality analysis unit, by providing detailed reports, it can accurately record and analyze teaching details, support precise adjustment of teaching strategies, monitor the listening status in real time, find the attention trough, guide teachers to optimize classroom content and methods, improve teaching based on data, enhance student participation and learning effects, and ensure that teaching goals are achieved;

[0122] In the electronic teaching plan unit, the teaching process is recorded and transcribed through multiple lenses for automatic recording and transcription, which can save time for sorting and ensure the accuracy of the teaching plan. Through rapid positioning and adjustment, knowledge points are automatically switched and positioned, so that the explanation can be optimized in combination with classroom analysis to improve teaching results. Through one-click download and update, the teaching plan update process can be simplified, accurately meeting the needs of teachers and improving teaching efficiency.

[0123] In the statistical analysis unit, by collecting statistics on course resources, student performance and classroom quality, teachers can gain a deeper understanding of learning progress and teaching effectiveness. By discovering and intervening in abnormal situations in learning and teaching, they can improve teaching quality and efficiency, so as to provide accurate feedback and adjustment suggestions to ensure that teaching reaches the best level at the end of the semester;

[0124] Specifically, the smart teacher module is designed around theoretical courses, experimental courses, and equipment practice courses. It conducts small-scale pilot teaching on helicopter aviation electrical engineering, helicopter fire control equipment, and helicopter equipment support practice. It integrates the whole process of pre-class preparation and preview, classroom teaching training, and post-class review and review, realizing seamless connection and intelligent assistance of teaching activities; in the pre-class preparation and preview stage, teachers use learning points to understand the students' mastery of historical knowledge points, so as to make targeted decisions on whether to repeat certain weak knowledge points in the next class. Through classified teaching activities, teachers can issue preview tasks and supervise students' preview progress to ensure that they complete reading and learning efficiently. The test release of intelligent composition and its effect analysis further help teachers understand the knowledge mastery of students after preview and review, so as to adjust the teaching plan. Students can use the resource voice retrieval function, combined with their own weak knowledge points to retrieve and use high-quality courseware, videos and other materials for self-enhanced learning. When students encounter difficult problems, they can also get timely and accurate answers through intelligent Q&A. During the classroom teaching and training stage, the senseless attendance in theoretical classes saves the teacher the need to call the roll, and automatically records and links to the usual grades, improving classroom efficiency. The borderless classroom makes cross-space live broadcast possible, effectively combining theory with practice, inside and outside the campus, and promoting resource sharing and academic exchanges. When teaching, teachers can collaborate with smart teachers to retrieve cases, courseware and quickly conduct in-class tests in real time, making the classroom more flexible and interesting and improving students' attention. Intelligent supervision such as behavior analysis and random roll call helps teachers maintain classroom order and urge students to listen carefully through real-time behavior analysis. In experimental classes and equipment practice classes, advanced simulation technology and mixed reality technology provide students with a fully real and safe immersive training environment, and intelligent queuing and calling provide efficient scheduling of equipment in the preparation area and training area. Data analysis and visualization technology provide students with intuitive and real-time feedback on operation results, and archive and record data in real time to help them learn and improve from mistakes. In the post-class review stage, students can quickly review the main content of each class through smart teachers to deepen their memory effect, and get personalized recommendations based on personal review tests and learning situation analysis to strengthen the learning of weak points. Teachers can use classroom quality analysis tools to review and summarize their own and students' performance in each class period and optimize the teaching rhythm. The electronic teaching plan function has become a convenient tool for teachers to update teaching plans, ensuring the continuous evolution of teaching content. In addition, the statistical analysis function allows teachers to timely review the teaching activities, knowledge points and changing trends of classroom quality in a stage, identify problems in time and implement interventions, ensuring the continuous optimization and improvement of teaching quality.

[0125] In this embodiment, through the MR access synchronous classroom module, the teacher connects the first-person video stream in the MR device to the synchronous classroom for live broadcast, so that the time delay during the live broadcast is small, which is convenient for students to observe the teacher's live demonstration through the smart blackboard screen; through the client carrying the image of the virtual person, it is possible to control the sound receiving and playback equipment and interact with the back-end voice analysis service, thereby realizing the function of users interacting with the virtual person through voice; through the real-time spatial coordinate correction module, the position of the virtual content can be automatically adjusted when the user moves or the environment changes to ensure its synchronization with the real world. Through the collaborative work of the Unreal Engine and the application, pixel streaming plug-in, signaling server, Web server, network transmission layer and client device components, the pixel streaming based on the Ubuntu system can transmit the virtual person application screen on the server to the remote client device in real time, and provide users with a smooth and efficient remote interactive experience. Through the combination of pre-class preparation, classroom auxiliary teaching and after-class tutoring in the smart teacher module, seamless connection and intelligent assistance of teaching activities are achieved, and problems can be discovered and intervened in time, ensuring the continuous optimization and improvement of teaching quality.

[0126] Example 2

[0127] See also Figure 2 As shown, the smart teacher method for virtual human and real teacher integrated teaching described in this embodiment includes the following steps:

[0128] S1, connect the first-person video stream in the MR device to the synchronous classroom for live broadcast;

[0129] S2. Interacting with a virtual human through voice, wherein the virtual human interaction module includes a voice control SDK and client voice interaction logic;

[0130] S3, dynamically adjust the spatial coordinates;

[0131] S4. Establish pixel streaming service on Ubuntu system;

[0132] S5. Integrate teaching activities including pre-class preparation, classroom auxiliary teaching and after-class tutoring.

[0133] In this embodiment, the time delay can be small during live broadcast, which is convenient for students to observe the instructor's live demonstration through the smart blackboard screen; it can control the audio and video equipment and interact with the back-end voice analysis service, so as to realize the function of user interaction with virtual people through voice; it can automatically adjust the position of virtual content when the user moves or the environment changes to ensure its synchronization with the real world, and can transmit the virtual human application screen on the server to the remote client device in real time, and provide users with a smooth and efficient remote interaction experience, and can also realize seamless connection and intelligent assistance of teaching activities, timely discover problems and implement intervention to ensure continuous optimization and improvement of teaching quality.

[0134] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0135] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0136] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0137] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.< / video>

Claims

1. The intelligent teacher system for the integrated teaching of virtual humans and real teachers is characterized by: include: MR access synchronous classroom module, which is used to connect the first-person video stream in the MR device to the synchronous classroom for live broadcast; A virtual human interaction module, which is used to interact with a virtual human through voice, and the virtual human interaction module includes a voice control SDK and client voice interaction logic; A spatial coordinate real-time correction module, which is used to dynamically adjust the spatial coordinates; Pixel Streaming Service module, which is used to establish the Pixel Streaming service on the Ubuntu system; The smart teacher module is used to integrate pre-class preparation, classroom auxiliary teaching and after-class tutoring in teaching activities.

2. The intelligent teacher system for virtual human and real teacher integrated teaching according to claim 1 is characterized in that: The method of connecting the first-person video stream in the MR device to the synchronous classroom for live broadcast includes: The instructor conducts a live demonstration through the MR device, which starts the video stream and projects it onto the laptop. The laptop starts the synchronous classroom client and synchronizes the local video stream to the server. Then, the synchronous classroom client pulls the live video stream to observe the live broadcast, and the students watch the instructor's live demonstration through the synchronous classroom client.

3. The intelligent teacher system for virtual human and real teacher integrated teaching according to claim 1 is characterized in that: The voice control SDK includes three working states: STATE_IDLE, STATE_READY and STATE_WORKING.

4. The intelligent teacher system for virtual human and real teacher integrated teaching according to claim 1 is characterized in that: The client voice interaction logic includes: The client needs to integrate the wake-up SDK and voice control SDK; The client establishes a websocket connection with the speech analysis service and performs login authentication.

5. The intelligent teacher system for virtual human and real teacher integrated teaching according to claim 1 is characterized in that: The method of interacting with a virtual person through voice includes: When the user says the wake-up word, the wake-up SDK throws a wake-up event, the client generates a wake-up id, and then the client sends the wake-up event to the speech analysis service. Then the client calls the pixel push service to play the virtual person's answering words; After the hololens finishes playing the wake-up response, it sends a voice stream to the voice analysis service. When sending the voice stream, it needs to carry the voice stream ID until the backend responds to the command to stop the voice stream. After the speech analysis service receives the voice stream from the HoloLens, it eventually responds to the HoloLens with a set of instructions.

6. The intelligent teacher system for virtual human and real teacher integrated teaching according to claim 5 is characterized in that: The method of dynamically adjusting the spatial coordinates includes: By placing a holographic object in HoloLens and fixing the position and rotation of the holographic object by adding a spatial anchor point; Use HoloLens' built-in sensors and cameras to capture and analyze environmental information, and create a spatial map by scanning the surrounding environment and identifying feature points; Add a spatial anchor component to the root GameObject of the holographic object, and attach a spatial anchor component with a relative position offset to its child GameObject; Use HoloLens to capture and identify QR codes in the scene, and use the camera and image processing system to detect and decode the information in the QR code in real time. After identifying the QR code, the location information can be obtained. Subsequently, the position and orientation of the virtual content in the mixed reality environment are adjusted according to the position information of the QR code.

7. The intelligent teacher system for virtual human and real teacher integrated teaching according to claim 1 is characterized in that: The method of establishing the pixel streaming service on the Ubuntu system includes: Build a server environment that supports pixel streaming and deploy applications and operations that need to be streamed on the server; Then, after deploying the corresponding version of Unreal Engine on the Ubuntu system, it is transmitted to the terminal device in the form of a video stream through the pixel streaming plug-in for display and operation.

8. The intelligent teacher system for virtual human and real teacher integrated teaching according to claim 1 is characterized in that: The pre-class preparation includes a learning situation analysis unit, a teaching activity classification & intelligent test paper composition & auxiliary review unit, a knowledge base and intelligent question and answer unit, a resource library and resource retrieval unit, and a word cloud analysis unit; The classroom auxiliary teaching includes a virtual-reality integrated scene generation unit, an MR & tablet simulation training unit, a large-screen queuing and calling unit, an MR mixed reality training unit, and a large-screen step statistics unit; The after-class tutoring includes a class content summary unit, a personalized resource recommendation unit, a class quality analysis unit, an electronic teaching plan unit and a statistical analysis unit.

9. A smart teacher method for integrated teaching of virtual humans and real teachers, applied to a smart teacher system for integrated teaching of virtual humans and real teachers as claimed in any one of claims 1 to 8, characterized in that: The following steps are involved: S1, connect the first-person video stream in the MR device to the synchronous classroom for live broadcast; S2. Interacting with a virtual human through voice, wherein the virtual human interaction module includes a voice control SDK and client voice interaction logic; S3, dynamically adjust the spatial coordinates; S4. Establish pixel streaming service on Ubuntu system; S5. Integrate teaching activities including pre-class preparation, classroom auxiliary teaching and after-class tutoring.

Images