VR virtual classroom teaching system

Abstract

The invention belongs to the field of virtual reality (VR) technology, and particularly relates to a VR virtual classroom teaching system which is characterized in that a 1-N synchronous teaching path is formed by a teacher VR head-mounted display, a teacher video card, a teacher notebook computer, a gigabit Ethernet switch, a student VR terminal network card and a student VR head-mounted display. The beneficial effects of the invention are that: 1) the structure is extremely simple, a 1-to-N one-way visual angle broadcast type VR classroom teaching system is established, and the system is used for synchronously pushing the real-time visual angle of the VR head-mounted display at the teacher end to the VR head-mounted displays at all student ends in a frame-level precision manner, so that students can watch the same teaching content in an immersive manner at the position and angle of the eyes of the teacher, and the teaching efficiency is improved; respective operation or mutual interference is not needed; any external hardware is not added, repeated rendering is not carried out, only the head of a teacher is utilized to display a rendered left eye (or right eye) frame, a student end synchronously sees a picture which is watched by the teacher in the lt, 80 ms, and the student end receives data streams in the local area network, and the bandwidth occupation of the local area network streaming media broadcasting mode is extremely small.

Description

Technical Field

[0001] This invention belongs to the field of virtual reality (VR) technology, and in particular relates to a VR virtual classroom teaching system. Background Technology

[0002] Existing VR systems can be broadly categorized into two paradigms: 1) Multi-user Collaborative VR: Characterized by all participants existing as independent virtual avatars in the same scene; each user has an independent perspective and can move and interact freely; teachers cannot force students to "look" at specific locations and must rely on voice or laser pointer guidance; network load increases linearly with the number of participants, with a surge in synchronization when there are more than 30 participants, and latency >150 ms can easily cause dizziness. 2) Single-player Demo in Parallel: Characterized by teachers pre-recording panoramic videos or 360° live streams, with students passively watching; there is no interaction between students, and teachers cannot adjust the perspective in real time; if live streaming is used, the uplink bandwidth ≥20 Mbps puts a heavy burden on the campus network; teachers leave the VR environment and return to operating on a PC, losing the immersive teaching experience. The common drawbacks of these two paradigms are: 1) They cannot allow teachers to operate in VR and give zero-latency "eye-level" demonstrations to all students from a first-person perspective; 2) Students either have a free perspective which leads to distraction or passive panoramic video which loses the possibility of interaction; 3) There is no lightweight, low-bandwidth, frame-synchronized solution in the industry for "one-click broadcasting from the teacher's perspective".

[0003] Chinese invention patent application number 202211146853.7 discloses an online internship system based on the Internet of Things (IoT) and its design method. The internship system includes a 5G base station, a 5G gigabit router, an enterprise-grade 10-gigabit router, a VR panoramic camera, a content backup and storage module, a local streaming media server, a cloud streaming media server, a web platform using a B / S architecture, a post-production module, a directing and broadcasting module, a multi-camera switching module, and a user experience terminal system. The system design method involves collecting audio and video information from the internship site using a 5G+VR panoramic camera, uploading it to storage, a streaming media server, and the cloud via a router, and then forwarding it via the 5G base station and router for presentation on different terminals after post-production. This solution is a pre-recorded VR teaching resource, not a real-time broadcast teaching method, and is only suitable for self-study outside of class, not for classroom teaching with teacher guidance. Summary of the Invention

[0004] The purpose of this invention is to provide a VR virtual classroom teaching system that overcomes the shortcomings of existing technologies and establishes a "1→N" unidirectional perspective broadcast VR classroom teaching system. This system synchronously pushes the real-time perspective of the teacher's VR headset to all students' VR headsets with frame-level precision, allowing students to immerse themselves in the same teaching content from the teacher's eye position and angle without needing to operate independently or interfere with each other.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A VR virtual classroom teaching system is characterized by comprising a 1→N synchronous teaching path consisting of a teacher's VR headset, a teacher's graphics card, a teacher's laptop, a gigabit Ethernet switch, student VR terminal network cards, and student VR headsets. The video processing steps are as follows: 1) The controller inside the teacher's VR headset requests the rendered image for both eyes from the teacher's graphics card; 2) The teacher's graphics card writes the rendered images of both eyes into memory via the GPU, forming RGBA image signals for both eyes; the image is formatted by a hardware encoder and output as an ES video stream signal; 3) The teacher's laptop receives the ES video stream signal, performs UDP data packetization, adds RTP timestamps, encapsulates it in MPEG-TS over SRT format, and generates an IP multicast signal; upon receiving the "all-student teaching" instruction, it sends the media stream to the accessing student terminals via TCP / IP multicast. 4) When the Gigabit Ethernet switch receives the IP multicast signal, it copies it N times within the switching chip and forwards it to the student VR terminal, forming a star-shaped multicast data stream. 5) The student VR terminal network card receives multicast IP packets, decapsulates them to obtain ES video stream signals, sends them to the signal processor for hardware decoding to output YUV420 format video frames, renders them with zero copy, and generates IP decoded image signals. 6) When the student's VR headset receives the IP decoded image signal, it places the decoded image signal onto the full-screen Quad and positions it in the foreground of the VR compositing layer; upon receiving the teacher's "Stop All Teaching" command, it returns the viewpoint to its original position.

[0006] Furthermore, in step 1), the controller inside the teacher's VR headset is a DisplayLink / HDMI controller.

[0007] Furthermore, in step 2), the rendering resolution of the teacher's graphics card is no less than 1440×1584×2@72 Hz.

[0008] Furthermore, in step 2), the network bandwidth format of the hardware encoder is set to H.265, GOP=1 s, 8Mbps, 25 fps.

[0009] Furthermore, in step 3), the graphics card in the teacher's laptop must be of a model no lower than PCIe ×16.

[0010] Furthermore, in step 3), the UDP data packets are segmented into UDP 1500-byte segments.

[0011] Furthermore, in step 4), the link interface model of the Gigabit Ethernet switch is RJ-45.

[0012] Furthermore, in step 5), the video is rendered using GPU Texture (external OES) decoding.

[0013] Furthermore, in step 6), the Quad needs to be destroyed and the local Camera rendering restored before the view is returned.

[0014] Compared with the prior art, the beneficial effects of the present invention are: 1) Provide a minimalist architecture to establish a “1→N” unidirectional perspective broadcast VR classroom teaching system, which synchronously pushes the real-time perspective of the teacher’s VR headset to all students’ VR headsets with frame-level precision, so that students can immerse themselves in watching the same teaching content from the position and angle of the teacher’s eyes, without having to operate independently or interfere with each other. 2) Without adding any external hardware or repeating the rendering, the left and right eye frames already rendered on the teacher's headset are used. They are then hardware encoded into H.264 / 265 within 60 fps and multicast over the local area network. The student end can see the screen the teacher is looking at synchronously in less than 80 ms. The student end receives the data stream within the local area network. 3) While teachers complete operations such as writing on the blackboard, conducting experiments, and disassembling in VR, the images they see are forcibly synchronized to all students' head-mounted displays with a delay of less than 40ms. Students can learn by "following the teacher's eyes" without any operation. The bandwidth usage is minimal due to the local area network streaming media broadcast method. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the architecture of an embodiment of the present invention. Detailed Implementation

[0016] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The components of the embodiments of the present invention described and shown in the accompanying drawings can typically be arranged and designed in many different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0018] See Figure 1 This is a schematic diagram of the architecture of a VR virtual classroom teaching system according to an embodiment of the present invention. The system consists of a teacher's VR headset, a teacher's graphics card, a teacher's laptop, a gigabit Ethernet switch, student VR terminal network cards, and student VR headsets, all connected sequentially to form a 1→N synchronous teaching path. This completes the closed-loop task of "teacher's eyes displaying rendered image → local area network video broadcast → student's eyes displaying synchronously." The video processing steps are as follows: 1) The teacher's VR headset has a built-in DisplayLink / HDMI controller that requests the rendered images from the teacher's graphics card in frame mode. It is ready to push the real-time view of the teacher's VR headset to all students' VR headsets with frame-level precision, so that students can immerse themselves in watching the same teaching content from the position and angle of the teacher's eyes. 2) The teacher's graphics card writes the rendered binocular images (rendering resolution 1440×1584×2@72 Hz) into CUDA Pinned Memory via the GPU's internal Copy Engine, forming binocular RGBA image signals; this image is then processed by the on-chip hardware encoder NVENC, with the network bandwidth format of the hardware encoder set to H.265, GOP=1 s, 8 Mbps, 25 fps, and output as an ES video stream signal.

[0019] 3) The teacher's laptop receives the ES video stream signal output by NVENC via PCIe ×16, divides it into UDP 1500-byte packets, adds RTP timestamps, encapsulates it into MPEG-TS over SRT format, and generates an IP multicast signal; after receiving the "all-student teaching" instruction, it sends the media stream to the accessed student terminals via TCP / IP multicast. 4) When the Gigabit Ethernet switch receives the IP multicast signal, the link interface in the Gigabit Ethernet switch is an RJ-45. The signal is copied N times in the switching chip and forwarded to the student VR terminal to form a star multicast data stream. 5) The student VR terminal network card receives multicast IP packets, the kernel SRT-Library decapsulates and obtains ES video stream signals, sends them to the signal processor SoC Hexagon VPU for hardware decoding and outputs YUV420 format video frames, zero-copy to GPU Texture (externalOES), and generates decoded image signals; 6) When the student's VR headset receives the IP decoded image signal, it pastes the decoded image signal onto the full-screen Quad and places it in the foreground of the VR compositing layer; when it receives the teacher's "Stop all teaching" command, it destroys the Quad, restores the local Camera rendering, and completes the return of the viewpoint. Steps 1-6 are interconnected, and the signal path is as follows: Teacher's head-mounted display ← Graphics card rendering → NVENC → Motherboard packaging → Switch → Student network card → VPU → GPU → Head-mounted display. The entire process takes less than 80 ms. The 1→N synchronization ends, and the teaching task is completed.

[0020] The differences between the embodiments of the present invention and existing shared collaborative systems and multi-user single-machine display systems are shown in Table 1 below.

[0021] Dimension Shared Collaboration Multiplayer Single-Player Demonstration Type This invention View control Students' free perspective makes them easily distracted. Pre-recorded / live stream, no immersion for teachers Teacher's forced perspective, unified attention Teacher's side VR PC central control needs to be exited Leaving the VR environment Teachers operate within VR throughout the entire process. Delay 100–200 ms 3–5 seconds (live broadcast) <40 ms (LAN) Bandwidth / person 1–2 Mbps 10–20 Mbps <5 kB / s Interaction Students can disturb each other No interaction Students can regain control deploy Public server required Need to push streaming box Teacher's laptop as host Table 1 Table 1 compares the differences between this invention and existing demonstration systems. The innovation of this invention lies in its focus on classroom teaching, taking the actual teaching needs after using VR devices as its starting point, and implementing VR technology in practice. It allows students to share the same perspective and content as the teacher when using VR devices to demonstrate classroom teaching, facilitating their understanding of the key knowledge points explained by the teacher.

[0022] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A VR virtual classroom teaching system, characterized in that, A synchronous teaching path from 1 to N is constructed, consisting of a teacher's VR headset, teacher's graphics card, teacher's laptop, gigabit Ethernet switch, student VR terminal network card, and student VR headset. The video processing steps are as follows: 1) The controller inside the teacher's VR headset requests the rendered image for both eyes from the teacher's graphics card; 2) The teacher's graphics card writes the rendered images of both eyes into memory via the GPU, forming RGBA image signals for both eyes; the image is formatted by a hardware encoder and output as an ES video stream signal; 3) The teacher's laptop receives the ES video stream signal, performs UDP data packetization, adds RTP timestamps, encapsulates it in MPEG-TS over SRT format, and generates an IP multicast signal; upon receiving the "all-student teaching" instruction, it sends the media stream to the accessing student terminals via TCP / IP multicast. 4) When the Gigabit Ethernet switch receives the IP multicast signal, it copies it N times within the switching chip and forwards it to the student VR terminal, forming a star-shaped multicast data stream. 5) The student VR terminal network card receives multicast IP packets, decapsulates them to obtain ES video stream signals, sends them to the signal processor for hardware decoding to output YUV420 format video frames, renders them with zero copy, and generates IP decoded image signals. 6) When the student's VR headset receives the IP decoded image signal, it places the decoded image signal onto the full-screen Quad and positions it in the foreground of the VR compositing layer; when it receives the teacher's "Stop All Teaching" command, it returns the viewpoint.

2. The VR virtual classroom teaching system according to claim 1, characterized in that, In step 1), the internal controller of the teacher's VR headset is a DisplayLink / HDMI controller.

3. The VR virtual classroom teaching system according to claim 1, characterized in that, In step 2), the teacher's graphics card rendering resolution should be no less than 1440×1584×2@72 Hz.

4. The VR virtual classroom teaching system according to claim 1, characterized in that, In step 2), the network bandwidth format of the hardware encoder is set to H.265, GOP=1 s, 8 Mbps, 25 fps.

5. A VR virtual classroom teaching system according to claim 1, characterized in that, In step 3), the graphics card in the teacher's laptop must be of a model no lower than PCIe ×16.

6. A VR virtual classroom teaching system according to claim 1, characterized in that, In step 3), UDP data packets are segmented into UDP 1500-byte segments.

7. A VR virtual classroom teaching system according to claim 1, characterized in that, In step 4), the link interface model of the gigabit Ethernet switch is RJ-45.

8. A VR virtual classroom teaching system according to claim 1, characterized in that, In step 5), the video is rendered using GPU Texture (external OES) decoding.

9. A VR virtual classroom teaching system according to claim 1, characterized in that, Before restoring the viewpoint in step 6), the Quad needs to be destroyed and the local Camera rendering restored.

Images