Method for collaborative universe classroom

Abstract

The invention relates to a collaborative universe classroom method, which belongs to the field of universe classrooms, and comprises the following steps: constructing a three-dimensional scene of a virtual classroom, including a platform, a blackboard, student seats and teaching experiment equipment, and designing virtual avatar models of teachers and students; constructing an expression and action database, collecting expression and action data of teachers and students through an action capture technology, and carrying out standardization processing and classification labeling on the data; facial expressions, actions and sound data of teachers and students are collected in real time through VR glasses, a camera, a microphone and action capture equipment; and the collected data is matched with the expression and action database, and the performance of the virtual avatar is updated, so that real-time interaction between the teacher and the students in the virtual classroom is realized, and the interaction efficiency between the teacher and the students is improved.

Description

Technical Field

[0001] The present invention belongs to the field of metaverse classrooms, and particularly relates to a method for a collaborative metaverse classroom. Background Art

[0002] With the rapid development of information technology, the application of technologies such as virtual reality (VR) and augmented reality (AR) has gradually penetrated into the field of education, especially in distance education, virtual classrooms, and online learning. Traditional teaching methods mainly rely on face-to-face teacher-student interaction. However, in the modern educational environment, especially affected by factors such as globalization and the epidemic, online education has gradually become an important teaching form. However, the traditional online education model has problems such as poor interactivity, insufficient immersion, and difficulty in guaranteeing learning effects.

[0003] Existing virtual classrooms usually rely on screen-based interaction methods. The communication between teachers and students is mainly carried out through text, voice, or video, etc. However, these methods cannot truly reflect the dynamic expressions and body languages of teachers and students, resulting in easy distraction of students' attention and a relatively single classroom atmosphere. In this case, the real interaction between teachers and students is difficult to be fully demonstrated, greatly affecting the teaching effect and learning experience.

[0004] To make up for this defect, some technical solutions attempt to introduce virtual avatars or virtual teacher roles for interaction. However, existing virtual classroom systems usually lack accurate capture and real-time feedback of the real actions and expressions of teachers and students, resulting in a large difference between the virtual avatar and the actual behavior of the user, further affecting the user's immersion.

[0005] Based on the above problems, a method for a collaborative metaverse classroom is proposed. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for a collaborative metaverse classroom, which improves the interaction efficiency between teachers and students.

[0007] To solve the above technical problems, the present invention is realized through the following technical solutions:

[0008] A method for a collaborative metaverse classroom includes the following steps:

[0009] Construct a three-dimensional scene of the virtual classroom, including a podium, a blackboard, student seats, and teaching experimental equipment, and design virtual avatar models for teachers and students;

[0010] Construct an expression and action database, collect the expression and action data of teachers and students through motion capture technology, and perform standardization processing and classification annotation on the data;

[0011] Real-time collect the facial expressions, movements and voice data of teachers and students through VR glasses, cameras, microphones and motion capture devices;

[0012] Match the collected data with the expression and action database, update the performance of the virtual avatar, and realize the real-time interaction between teachers and students in the virtual classroom.

[0013] On the other hand, a collaborative metaverse classroom system is provided, including the following modules:

[0014] A virtual scene and character modeling module for designing the three-dimensional environment of the virtual classroom and virtual avatars;

[0015] An expression and action database module for storing and managing standardized data of expressions and actions;

[0016] A real-time information collection module for collecting users' facial expressions, movements and voice data through hardware devices;

[0017] A data processing and synchronization module for matching the collected data with the database and updating the virtual avatar.

[0018] On the other hand, a computer-readable storage medium is provided, storing a computer program, characterized in that when the program is executed by a processor, the following method is implemented:

[0019] Construct a three-dimensional scene of the virtual classroom, including a podium, a blackboard, student seats and teaching experimental equipment, and design virtual avatar models of teachers and students;

[0020] Construct an expression and action database, collect the facial expressions and movement data of teachers and students through motion capture technology, and perform standardized processing and classification annotation on the data;

[0021] Real-time collect the facial expressions, movements and voice data of teachers and students through VR glasses, cameras, microphones and motion capture devices;

[0022] Match the collected data with the expression and action database, update the performance of the virtual avatar, and realize the real-time interaction between teachers and students in the virtual classroom.

[0023] Beneficial effects:

[0024] The embodiments of the present disclosure adopt the technical means of constructing a three-dimensional scene of the virtual classroom and virtual avatar models, overcome the technical problems of the lack of immersion and interactivity in traditional online education, and thus achieve the technical effect of enhancing the realism and user experience of the virtual classroom;

[0025] The embodiments of the present disclosure adopt technical means of constructing an expression and action database and collecting data through motion capture technology, combined with technical means of a real-time information collection module and a data processing and synchronization module, enabling the virtual avatar to be highly synchronized with the user's real behavior and realizing real-time and efficient interaction between teachers and students in a virtual environment.

[0026] Of course, it is not necessary for any product implementing the present invention to achieve all the above-mentioned advantages simultaneously. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0028] Figure 1 It is a flowchart of the method for the embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments of the present invention belong to the scope of protection of the present invention.

[0030] In this embodiment, a method for establishing a collaborative metaverse classroom is provided through technical means such as constructing an expression and action database, designing a virtual classroom scene, precise hardware configuration, real-time data collection and synchronization processing, etc., to realize real interaction between teachers and students in a virtual environment.

[0031] As Figure 1 shown, a method for a collaborative metaverse classroom includes the following steps;

[0032] Step 1: Construct a classroom scene and a character modeling library;

[0033] For the classroom scene design, design the three-dimensional environment of the virtual classroom, including elements such as a podium, a blackboard, student seats, teaching experiment equipment, etc. By reasonably designing the internal structure of the classroom, the virtual classroom environment has a high degree of immersion and interactivity.

[0034] Furthermore, during the design process of the virtual classroom scene, considering the teaching needs of different disciplines, different types of scenes can be customized, such as a science laboratory, a computer laboratory, a language and hearing classroom, etc.

[0035] Character Modeling: Design virtual avatar models for teachers and students respectively, ensuring that the virtual characters have realistic appearances and action performances. Consider the diversity of teachers and students, including body types, genders, clothing, facial features, etc.

[0036] Step 2: Construct an expression and action database;

[0037] First, construct an expression and action database, which is used to store various common expressions and actions of teachers and students in the virtual classroom. The specific implementation steps are as follows:

[0038] Data Collection: Collect the expression and action data of teachers and students through motion capture technology. The motion capture device can accurately record the movement trajectories and posture changes of the human body in three-dimensional space, and at the same time, cooperate with facial recognition technology to capture expression changes.

[0039] Adopt machine learning technology, and use big data analysis and automatic annotation to improve the recognition accuracy of actions and expressions, ensuring that subtle expression changes and action details can be captured.

[0040] Standardize the collected data, and convert the action and expression data into features in multiple dimensions, including facial feature points such as the positions of eyebrows, eyes, and mouths, joint angles, action trajectories of hands and feet, etc., to ensure the consistency and high precision of the data.

[0041] Classify and label the data according to expression types, such as smiling, frowning, surprised, etc.; action types, such as raising hands, nodding, walking, writing, etc., to facilitate subsequent data processing and matching.

[0042] Combine the constructed expressions and actions with character modeling, so that the character modeling can reflect the user's actions and expressions in real time. For example, the gestures of the teacher during the lecture, the actions of the student when raising a hand to speak, etc.

[0043] Furthermore, the method regularly optimizes the expression and action database through a feedback mechanism, adds new data samples, and corrects the biases of existing models, so that the database always meets the needs in actual teaching.

[0044] The optimization methods include using deep learning algorithms to train and self-adjust the collected data, so as to continuously improve the recognition accuracy and response speed of actions and expressions.

[0045] Step 3: Real-time information collection;

[0046] Teachers and students need to be equipped with VR glasses, high-definition cameras, microphones and motion capture devices to achieve an all-round interactive experience.

[0047] The VR glasses are responsible for displaying the three - dimensional scene of the virtual classroom; the camera is used to capture the facial expressions and movements of teachers and students; the microphone is used to collect sound signals to ensure smooth voice communication; and the motion capture device records the user's body movements.

[0048] Furthermore, the motion capture can be completed synchronously through image acquisition, which can save costs and reduce data traffic.

[0049] Step 4: The system uses cameras and motion capture devices installed at the teacher's end and the student's end to collect the image and motion data of teachers and students in real - time. This data includes facial expressions, eye movements, mouth openings and closings, hand and body movements, etc.

[0050] The collected image and motion data will first go through a data pre - processing stage to remove noise and perform normalization to ensure data standardization.

[0051] When performing image and motion recognition, efficient image - processing algorithms and motion - analysis models are used to reduce errors and improve the real - time response speed.

[0052] By comparing the collected motion and expression data with the standard models in the database, the system will select the most appropriate expression or motion to update the performance of the virtual avatar. For example, when the teacher smiles, the facial expression of the virtual avatar will automatically synchronize with the teacher's real expression.

[0053] At the same time, the actions of teachers and students will also be immediately reflected in the virtual avatar, achieving high - precision real - time interaction.

[0054] Furthermore, for the body motion data: the hands, wrists, elbows, head, shoulders, knees, hips, and feet are used as the main collection points to record the spatial positions and movement trajectories of these key points;

[0055] For the facial expression data: the key points on the face, such as the position changes of the eyebrows, eyes, and mouth, are collected to record the expression features;

[0056] For the sound data: the voice signal is collected through the microphone to analyze the speech content, tone, and emotional features.

[0057] The system uses efficient image - processing algorithms, motion - analysis models, and speech - recognition technologies to identify and match the pre - processed data and simultaneously determine the expressed emotional state:

[0058] Body emotion analysis: By analyzing the spatial relationships of key points such as hands, wrists, elbows, head, shoulders, knees, hips, and feet, comparing with the standard action models in the database, selecting the most similar action data, and determining the expressed emotion;

[0059] Facial emotion analysis: By analyzing the positional changes of facial key points, comparing with the standard expression models in the database, selecting the most similar expression data, and determining the expressed emotion;

[0060] Voice emotion analysis: By using speech recognition technology to analyze speech content and intonation, and combining with an emotion recognition model to judge the user's emotional state;

[0061] The said emotional states include happiness, confusion, concentration, etc.

[0062] The system fuses and processes body movements, facial expressions, and voice data, comprehensively considering the mutual influence between multi-modal data: requiring emotional consistency. For example, when the voice data detects a change in the user's emotion, the system will synchronously adjust the expression and actions of the virtual avatar to ensure the consistency of emotional expression;

[0063] In some embodiments, the method includes logging in and role selection;

[0064] Teachers and students log in to the system using account passwords, and the system automatically assigns corresponding role permissions according to the user type.

[0065] Before teachers and students enter the virtual classroom, they can select or customize their virtual avatars through the virtual avatar selection interface, including hairstyles, clothing, skin color, etc. Users can also upload their own avatars or 3D models to further enhance the personalized experience.

[0066] In the metaverse classroom established by the present disclosure, the actions and expressions of teachers and students will be presented in real time in the virtual classroom through virtual avatars. All actions are processed and data-matched in real time to ensure that the performance of the virtual avatar is as close as possible to the actions and expressions of real users.

[0067] This method supports multi-party interaction and collaboration between teachers and students, and between students and students. For example, teachers can guide students' attention through gestures, and students can raise their hands to ask questions; the system also supports group discussions and collaborative tasks, allowing students to share materials and jointly solve problems in the virtual environment.

[0068] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0069] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the present invention, so that those skilled in the art can well understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents.

Claims

1. A method for a collaborative metaverse classroom, characterized in that: The following steps are involved: Construct a 3D scene of a virtual classroom, including a podium, blackboard, student seats, and teaching experimental equipment, and design virtual avatar models of teachers and students; Construct an expression and action database, collect the expression and action data of teachers and students through motion capture technology, and standardize and classify the data; Collect teachers' and students' facial expressions, movements and sound data in real time through VR glasses, cameras, microphones and motion capture devices; The collected data is matched with the expression and action database, and the performance of the avatar is updated to enable real-time interaction between teachers and students in the virtual classroom.

2. The method according to claim 1, characterized in that The three-dimensional scene of the virtual classroom is customized into a science laboratory, a computer laboratory or a language and listening classroom according to subject requirements.

3. The method according to claim 1, characterized in that The virtual avatar model covers different body shapes, genders, clothing and facial features, and supports user-defined avatars or 3D models.

4. The method according to claim 1, characterized in that The construction of the expression and action database includes: Collect data through motion capture equipment and facial recognition technology; Use machine learning technology to automatically label and standardize data; Classify and annotate facial expressions and actions, including smiling, frowning, raising hands, and nodding.

5. The method according to claim 1, characterized in that The real-time data collection and processing steps include: Limb movement data collection: Take the hands, wrists, elbows, head, shoulders, knees, hips, and feet as the main collection points, and record the spatial position and movement trajectory of these key points; Facial expression data collection: collect key facial points, such as the position changes of eyebrows, eyes, and mouth, and record expression characteristics; Sound data collection: collect voice signals through microphones and analyze voice content, intonation and emotional characteristics; Body emotion analysis: compare with the standard action model in the database, select the most similar action data, and determine the emotion expressed; Facial emotion analysis: compare with the standard expression model in the database, select the most similar expression data, and determine the expressed emotion; Voice emotion analysis: Analyze the voice content and intonation through speech recognition technology, and combine it with the emotion recognition model to judge the user's emotional state; The body movements, facial expressions and sound data are integrated and processed, and the mutual influence between multimodal data is comprehensively considered to ensure emotional consistency; when the sound data detects changes in user emotions, the system will synchronously adjust the expressions and movements of the virtual avatar to ensure consistency in emotional expression.

6. The method according to claim 1, characterized in that The real-time interaction uses efficient image processing algorithms and motion analysis models to reduce errors and improve response speed.

7. The method according to claim 1, characterized in that It also includes login and role selection steps: teachers and students log in to the system through their accounts and select or customize their virtual images.

8. The method according to claim 1, characterized in that The method supports multi-party collaboration functions, including gesture guidance, raising hands to ask questions, group discussions, and sharing of data.

9. A collaborative metaverse classroom system, characterized in that: The system includes the following modules: Virtual scene and character modeling module, used to design the three-dimensional environment and virtual avatars of the virtual classroom; Expression and action database module, used to store and manage standardized data of expressions and actions; Real-time information collection module, which collects user's expression, action and voice data through hardware devices; The data processing and synchronization module matches the collected data with the database and updates the virtual avatar.

10. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images