An AR glasses-based physical experiment remote demonstration and real-time guidance system and method

Abstract

The application discloses a kind of physical experiment remote demonstration and real-time guidance system and method based on AR glasses, it is related to AR intelligent teaching, machine vision and remote experiment technology.System includes teacher AR acquisition glasses, student AR display glasses and local area network server.Teacher is through AR glasses macro photography acquisition first visual angle experiment video, and it is broadcasted to student AR glasses synchronization viewing through WiFi 6 local area network one to many;Student can return real operation picture, and teacher carries out real scene AR space marking, realizes virtual-real fusion accurate guidance.System adds six big functions of real-time AR early warning of dangerous operation, virtual instrument reading superposition, AR experiment step guidance, operation completion degree automatic scoring, student local reverse marking question, abnormal operation automatic interception reporting, forms whole-process safety, standardization, intelligent teaching closed loop.The application can eliminate visual angle limit, improve guiding efficiency, reduce experiment risk, applicable to various physical experiment teaching, with low delay, high stability, light weight, easy deployment and the like advantages.

Description

Technical Field

[0001] This invention relates to intelligent wearable teaching devices, augmented reality (AR) interaction, machine vision recognition, low-latency audio and video transmission over local area networks, and remote experimental teaching technology. Specifically, it relates to a remote demonstration and real-time guidance system and method for physics experiments based on AR glasses. It is suitable for physics experiment classrooms and can realize first-person perspective experiment demonstration by teachers, remote real-time observation by students, AR real-scene annotation guidance, warning of dangerous behaviors, experimental step guidance, automatic operation scoring, and closed-loop Q&A. Background Technology

[0002] The following pain points are commonly found in current physics experiment classroom teaching: 1) During the teacher's demonstration, students in the back and outer rows could not clearly observe the wiring, instrument readings, and light path adjustment, resulting in a poor demonstration effect; 2) When students are doing hands-on practice in groups, teachers cannot supervise multiple groups at the same time, resulting in untimely and inaccurate guidance; 3) Ordinary live streaming and video calls only allow viewing the screen and do not support spatial annotation, resulting in poor interactivity and low guidance efficiency; 4) Dangerous behaviors such as students' improper operation and use of instruments beyond their range are difficult to detect in real time; 5) The experimental procedures are complex and prone to errors, and there is a lack of synchronized AR guidance and auxiliary reading tools; 6) The completion of the experiment relies on manual evaluation, the process is not traceable, and the answers are not accurate; 7) It is difficult to carry out large-scale, distributed, cross-location remote experimental teaching and precise tutoring.

[0003] Current technology lacks a dedicated experimental teaching system that integrates AR first-person perspective data acquisition, real-world spatial annotation, local area network one-to-many broadcasting, hazard warning, step-by-step guidance, automatic scoring, and closed-loop help and Q&A. Summary of the Invention

[0004] 3.1 Purpose of the Invention 1) Enables teachers to perform experiments from a first-person perspective with high-definition macro photography, allowing students to observe synchronously and without obstruction through AR glasses; 2) Enables real-time transmission of student experiment process footage, allowing teachers to remotely annotate with arrows, circles, and text using AR space annotations, achieving precise "hands-on guidance from a distance"; 3) Build a low-latency one-to-many WiFi 6 LAN transmission system that operates stably without relying on the external network; 4) Add six core functions to improve the teaching and safety closed loop: real-time AR early warning of dangerous operations, virtual instrument reading overlay, AR experimental step guidance, automatic scoring of operation completion, local reverse annotation and questioning by students, and automatic recording and reporting of abnormal operations. 5) Supports students to ask for help with one click, capture / record short segments and upload them, and teachers to annotate and answer questions one-on-one, forming a complete experimental teaching ecosystem.

[0005] 3.2 System Overall Architecture The system consists of three parts: 1) AR data collection glasses for teachers; 2) AR display glasses for students; 3) Local area network real-time transmission and AR annotation server.

[0006] Overall working principle: Teachers wear AR glasses to capture experimental operation videos at close range from the human eye's perspective, and stream them to the server in real time via WiFi 6 LAN; the server distributes the videos to all students' AR glasses in a one-to-many broadcast manner, and the students see the teacher's operation screen overlaid on their real field of vision.

[0007] During hands-on practice, students' AR glasses transmit the local practice footage back to them; teachers make AR annotations, and the annotation information is simultaneously sent to the students' glasses to achieve virtual-real integration guidance.

[0008] The system has visual recognition capabilities and can provide real-time AR warnings for dangerous operations, wiring errors, and over-range use; it can overlay virtual scales, readings, and highlighted steps on real instruments; it can automatically analyze the standardization of operations and provide a completion score; students can mark questionable areas in the local real-view circle and upload them, and abnormal operations are automatically recorded and pushed to teachers, realizing safe, standardized, and intelligent guidance throughout the entire process.

[0009] 3.3 Improved System Functions 1) First-person perspective macro live streaming and commentary; 2) One-to-many low-latency LAN broadcast; 3) Teacher's AR spatial annotation (arrows / circles / text); 4) Students ask questions via voice and teachers provide answers via voice. 5) Real-time transmission of student practice footage; 6) Students can request help with one click, plus capture / upload short video clips; 7) Teachers provide one-on-one annotation and targeted delivery of answers; 8) Real-time AR warnings for dangerous operations (live operation violations, hand near light source, short circuit risk); 9) Automatic alarm for instrument exceeding range / threshold; 10) The AR virtual instrument scale is superimposed on the correct reading; 11) AR experiment steps highlighted and guided; 12) Automatic recognition and scoring of student task completion; 13) Students ask questions using local reverse annotation; 14) Abnormal operations are automatically recorded and reported to the teacher; 15) The entire experiment process is recorded, stored, and can be played back retrospectively.

[0010] 3.4 System Composition (1) AR data collection glasses for teachers Includes: macro high-definition camera module, IMU attitude perception module, noise-canceling microphone, speaker, WiFi 6 transmission module, embedded main control unit, battery module, lightweight frame, AR annotation and explanation unit, and multi-channel student screen monitoring unit. (2) Student-side AR display glasses It includes: optical waveguide perspective display module, front-facing high-definition camera, one-click help module, snapshot / short segment recording module, local reverse annotation module, WiFi 6 transmission module, main control and data processing unit, power supply module, hazard alarm prompt module, and AR step guidance module.

[0011] (3) Local Area Network Server The system includes: a video stream forwarding module, an AR annotation coordinate mapping module, a classroom management module, an H.265 encoding module, a machine vision recognition module, a dangerous behavior judgment module, a step-by-step guidance module, an automatic scoring module, an anomaly capture module, and a data storage and traceability module. The system uses a WebRTC LAN transmission architecture, with an end-to-end latency of less than 200ms. Attached Figure Description

[0012] Figure 1 System overall structure diagram Figure 1 The overall structural block diagram of the system of this invention mainly includes teacher-side AR acquisition glasses, student-side AR display glasses, and a local area network real-time transmission and AR annotation server. The server includes a video stream forwarding module, an AR annotation processing module, a visual recognition module, a hazard warning module, a step-by-step guidance module, and an automatic scoring module, used to realize one-to-many video broadcasting, AR annotation processing, intelligent recognition, and teaching management functions. The teacher-side AR acquisition glasses are used for macro video acquisition, voice intercom, and AR annotation. The student-side AR display glasses are used to receive live broadcast images, transmit practical operation videos, display annotations and warning information, and support requests for help, snapshots, and local reverse annotation. All parts achieve low-latency bidirectional data transmission via a WiFi 6 local area network, forming a complete remote experimental teaching system.

[0013] Figure 2 Diagram of the AR data collection glasses module for teachers Figure 2The diagram shows the internal modules of the AR data acquisition glasses for teachers, including a macro high-definition camera module, an IMU posture perception module, a noise-canceling microphone module, a speaker / voice intercom module, a WiFi 6 transmission module, an AR annotation and explanation unit, a multi-channel student screen monitoring unit, an embedded main control unit, and a battery power supply module. These modules are used to complete first-person perspective experimental operation data acquisition, voice explanation, data transmission, and remote annotation functions.

[0014] Figure 3 Student AR display glasses module composition diagram Figure 3 This diagram shows the internal modules of the AR display glasses for students, including a waveguide perspective display module, a front-facing high-definition camera module, a one-click help button module, a capture / recording trigger unit, a local reverse annotation module, an AR hazard warning module, an AR step guidance module, a data processing and main control unit, a WiFi 6 transmission module, and a power supply / battery module. These modules enable functions such as viewing experimental images, transmitting practical data, asking for help, AR annotation display, and safety warnings.

[0015] Figure 4 This is a complete workflow diagram of the AR remote annotation and assistance guidance of the present invention.

[0016] As shown in the figure, the process includes the following steps: Step 1: Begin; Step 2: Real-time transmission of student experimental operations and video feeds Step 3: Intelligent system monitoring (dangerous behavior / step guidance / automatic scoring); Step 4: Automatic recording of abnormalities or one-click help triggered by students; Step 5: Students provide reverse annotations of real-world scenarios (circling areas of doubt). Step 6: Video monitoring / segment upload --> LAN server; Step 7: The teacher receives the video feed, requests for help, and records clips. Step 8: Teacher's AR annotation guidance (arrows / circles / text); Step 9: Labeling information is distributed to the local area network server; Step 10: Students receive AR annotation guidance.

[0017] Finally, the closed-loop guidance was completed. Detailed Implementation Example 1: Experimental Demonstration Mode 1) Teachers wear AR glasses and enter classroom broadcast mode; 2) The macro camera module captures the entire process of experimental setup, wiring, adjustment, and reading. 3) After video encoding, it is uploaded to the server via WiFi 6 local area network; 4) The server broadcasts data to all students' AR glasses in a one-to-many manner; 5) Teacher's voice explanation, with the ability to freeze the screen, zoom in on specific areas, and add AR annotations; 6) The system overlays virtual scales, correct readings, and step-by-step instructions onto the student's AR field of view; 7) All students have a clear, unobstructed first-person perspective.

[0018] Example 2: Student hands-on practice + intelligent guidance + safety early warning mode 1) Students perform experimental operations, and the glasses continuously transmit images back; 2) The system's visual recognition module monitors in real time: whether there are unauthorized wiring or hands approaching dangerous parts; whether the instruments are over-range; and whether the voltage / current is abnormal. 3) When a risk occurs, a red AR warning box will pop up on the student's AR glasses along with a voice prompt; 4) The system highlights the next operation component according to the experimental procedure, providing AR step guidance; 5) The system analyzes the standardization of operations in real time and outputs a score for the completion of the experiment; 6) Students encountering difficulties can trigger a one-click help request, or mark the location of their question in the local real-view circle and upload it; 7) When an abnormal operation occurs, the system will automatically capture the scene / short video and send it to the teacher; 8) Teachers receive images and requests for help, and provide AR annotations, text or voice answers; 9) Labels and instructions are sent to the student's AR glasses, and displayed in a superimposed virtual and real image; 10) Students correct their operations, complete the experiment, and the entire process is automatically recorded and prepared for review.

[0019] 6. Beneficial effects 1) The demonstration has no viewing angle limitations, allowing all students to clearly observe the detailed operation, which significantly improves the teaching effect; 2) Teachers can remotely annotate images in real-world AR mode to accurately point out errors, significantly improving tutoring efficiency; 3) The local area network has low latency and high stability, making it suitable for laboratory environments and not dependent on the external network; 4) Added real-time early warning for hazardous operations, significantly reducing experimental safety risks; 5) AR virtual readings and step-by-step guidance reduce learning difficulty and improve experimental standardization; 6) Automatic scoring and process tracking reduce teachers' workload and facilitate debriefing; 7) Local reverse annotation by students + automatic recording of anomalies enables more efficient closed-loop Q&A; 8) Lightweight to wear, does not interfere with experimental operations, highly versatile, and covers the vast majority of physics experiments.

Claims

1. A remote demonstration and real-time guidance system for physical experiments based on AR glasses, characterized in that, include: AR data acquisition glasses for teachers, AR display glasses for students, and a local area network real-time transmission and AR annotation server; The teacher-side AR acquisition glasses are used for macro-level acquisition of first-person perspective experimental operation videos and real-time streaming. The server is used to receive video streams and send them to student-end AR display glasses in a one-to-many broadcast manner; The student-side AR display glasses are used to transmit local hands-on video footage and receive and display AR annotations, warnings, and step-by-step guidance information; The teacher can annotate the real-world space of the images sent back by the students, and the annotation information is simultaneously sent to the corresponding student's AR glasses for overlay display; The system also integrates a machine vision module for identifying dangerous operations, monitoring instruments for exceeding their range, automatically recording abnormal operations, and automatically scoring the completion of experiments.

2. The system according to claim 1, characterized in that, The teacher-side AR data acquisition glasses include: a macro high-definition camera module, an IMU attitude module, a WiFi 6 transmission module, a voice intercom module, an AR annotation and explanation unit, and an embedded main control and power supply module.

3. The system according to claim 1, characterized in that, The student-side AR display glasses include: a waveguide perspective display module, a front-facing camera, a one-click help module, a snapshot / short-segment recording module, a local real-scene reverse annotation module, an AR hazard warning module, an AR step-by-step guidance module, and a WiFi 6 transmission module.

4. The system according to claim 1, characterized in that the server comprises: The system includes a video forwarding module, an AR annotation coordinate mapping module, a classroom management module, an H.265 encoding module, a dangerous behavior judgment module, an experimental procedure guidance module, an automatic scoring module, and an anomaly interception and data storage module, with transmission latency controlled within 200ms.

5. The system according to claim 1, characterized in that, The student app supports one-click help requests, capturing the current scene, recording short video clips, and annotating questions with local real-view circles. The questions are then relayed to the teacher's app via the server for point-to-point annotation and answering.

6. The system according to claim 1, characterized in that, The system can provide real-time AR alarms for unauthorized wiring, hands approaching dangerous parts, and instruments exceeding their range. It also overlays virtual instrument scales, correct readings, and highlighted experimental steps in the student's field of vision.

7. A method for remote demonstration and real-time guidance of physical experiments based on AR glasses, characterized in that, include: Teachers use AR glasses to capture close-up videos of experimental operations and upload them to the server via WiFi 6 local area network; The server broadcasts to many students' AR glasses, allowing students to view the content in real time using AR perspective. Students transmit video footage during practical exercises, and the system monitors dangerous behaviors, provides step guidance, and scores completion rates in real time. When a student triggers a request for help or performs an abnormal operation, the image / segment is automatically uploaded, and the teacher adds AR annotations to the scene. Annotations, instructions, and answers are simultaneously sent to students' AR glasses and displayed overlaid on the real experimental scene.

8. The method according to claim 7, characterized in that, Also includes: Students circled and marked the areas of doubt in the local real-world scene and uploaded them; Teachers receive and push annotated answers one-on-one; the system uses automatic AR warnings for dangerous operations and exceeding the range.

Images