Multi-scene semi-synchronous virtual classroom method based on VR head-mounted display device

Abstract

The invention discloses a multi-scene semi-synchronous virtual classroom method based on VR head display equipment, and the method comprises the steps: reading an initial interface of a virtual classroom, building courseware connection between a teacher end and a student end, and entering a preset virtual classroom scene; establishing communication connection between the teacher terminal and all the student terminals, and adjusting seats and visual angles of the corresponding student terminals according to the selected seat anchor points; obtaining a teaching file in the teacher end, converting the teaching file according to a function of a preset database, and displaying the converted teaching file at a specified position of the preset virtual classroom scene; and acquiring voice information of the teacher terminal or the current student terminal, converting the acquired voice information, and transmitting the converted voice information to other terminals to realize voice interaction of the virtual classroom. According to the invention, personalized demands of classroom scene diversification, teaching voice interaction, teaching PPT courseware display and seat visual angle switching in a virtual classroom teaching scene are realized.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality technology, and in particular to a multi-scene semi-synchronous virtual classroom method based on a VR head display device. Background Art

[0002] The existing teaching mode is mainly the traditional offline teaching method. With the development of learning equipment, online teaching has also begun to appear. The traditional online teaching method is to use mobile phones, tablets and computers for online teaching. However, with the emergence of VR glasses, immersive teaching has become possible. VR glasses allow the wearer to be completely immersed in the picture through the glasses. With the head tracker of the glasses, the picture seen by both eyes can be changed in real time as the head moves, so as to achieve the feeling of being completely immersed in the virtual world.

[0003] At present, some studies have demonstrated the teaching methods of VR virtual classrooms, but the students in this virtual classroom are not real-time students, so they cannot communicate by voice, and can only comment in text. Moreover, the teaching content is the video content played, and there is no teacher teaching in real time. Moreover, this application scenario based on virtual reality teaching only has a single classroom. For lower grade students in elementary and junior high schools, a single classroom teaching scene is too boring and cannot arouse students' interest in online learning; furthermore, this application based on virtual reality teaching does not have the function of playing PPT courseware. Because VR glasses have not been widely popularized, many teachers' teaching PPT courseware are traditional PPT files. It is costly for teachers to make and use VR-PPT. At the same time, this application based on virtual reality teaching cannot be fully interactive. It simply plays videos in virtual scenes. Students cannot adjust their seats like in real classes, so as to realize the free switching of positions and listening perspectives in teaching scenes; therefore, these existing VR virtual classrooms and teaching methods cannot meet the needs of teachers and students for classroom scene diversification, teaching voice interaction, teaching PPT courseware display and seat perspective switching.

[0004] Therefore, the prior art needs to be improved. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the defects of the existing technology, the present invention provides a multi-scene semi-synchronous virtual classroom method based on a VR head display device to solve the problem that the existing VR virtual classroom and teaching methods cannot meet the personalized needs of classroom scene diversification, teaching voice interaction, teaching PPT courseware display and seat perspective switching.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] In a first aspect, the present invention provides a multi-scene semi-synchronous virtual classroom method based on a VR head display device, comprising:

[0008] Read the initial interface of the virtual classroom, establish a courseware connection between the teacher and the student, and enter the preset virtual classroom scene;

[0009] Establishing a communication connection between the teacher terminal and all student terminals, and adjusting the seats and viewing angles of the corresponding student terminals according to the selected seat anchor points;

[0010] Acquire the teaching file in the teacher terminal, convert the teaching file according to the function of the preset database, and display the converted teaching file at the designated position of the preset virtual classroom scene;

[0011] The voice information of the teacher terminal or the current student terminal is obtained, and the obtained voice information is converted, and the converted voice information is transmitted to other terminals to realize the voice interaction of the virtual classroom.

[0012] In one implementation, the process of reading the initial interface of the virtual classroom, establishing a courseware connection between the teacher terminal and the student terminal, and entering a preset virtual classroom scene includes:

[0013] Configure and read the initial interface of the virtual classroom based on the unity3D platform;

[0014] Determine the teaching channel, preset virtual classroom scene and virtual character image selected in the initial interface of the virtual classroom; wherein the preset virtual classroom scene includes: any one of an ordinary classroom scene, a forest classroom scene, a desert classroom scene, an ocean classroom scene and a space classroom scene;

[0015] A courseware connection is established between the teacher terminal and all student terminals through a synchronization plug-in of a multiplayer game scene, and the preset virtual classroom scene is entered according to the scene option.

[0016] In one implementation, establishing a communication connection between the teacher terminal and all student terminals, and adjusting the seat and viewing angle of the corresponding student terminal according to the selected seat anchor point, includes:

[0017] A communication plug-in for calling multi-person voice;

[0018] According to the selected teaching channel, using the multi-person voice communication plug-in to establish a communication connection between the teacher terminal and all student terminals;

[0019] Acquire the position of a virtual parabola emitted by the control device, and determine the currently selected seat according to the seat anchor point touched by the virtual parabola position;

[0020] The virtual and reality interaction plug-in based on the unity3D platform controls the first-person perspective of the corresponding student end, adjusts the seat of the corresponding student end to the currently selected seat, and moves the first-person perspective of the corresponding student end to the perspective of the currently selected seat.

[0021] In one implementation, adjusting the seat of the corresponding student terminal to the currently selected seat so that the first-person perspective of the corresponding student terminal moves to the perspective of the currently selected seat includes:

[0022] Numbering all seats in the preset virtual classroom scene and obtaining the coordinates of each seat in the virtual world;

[0023] All seats are marked by preset signs, and an array of virtual characters and seat mapping is formed;

[0024] According to the virtual character and seat mapping array, detecting whether the currently selected seat has a virtual character from another student terminal seated;

[0025] If the currently selected seat is occupied by a virtual character from another student, the virtual character corresponding to the original seat will be mapped to another empty seat, and the seat of the student who selected the seat will be adjusted to the currently selected seat, so that the first-person perspective of the corresponding student will be moved to the perspective of the currently selected seat.

[0026] In one implementation, the step of obtaining the teaching file in the teacher terminal, converting the teaching file according to a function of a preset database, and displaying the converted teaching file at a designated location of the preset virtual classroom scene includes:

[0027] Obtaining the teaching files in the teacher terminal;

[0028] Importing the preset database, and using the function defined in the preset database to convert the teaching file into a plurality of pictures;

[0029] Performing dynamic interactive conversion on the converted multiple images to obtain a teaching file displayed in the preset virtual classroom scene;

[0030] The teaching files displayed in the preset virtual classroom scene are synchronized to all student terminals and displayed at the designated location of the virtual classroom scene of each student terminal.

[0031] In one implementation, the acquiring of voice information from the teacher terminal or the current student terminal, converting the acquired voice information, and transmitting the converted voice information to other terminals includes:

[0032] Acquire the voice information of the teacher terminal or the current student terminal;

[0033] The multi-person voice communication plug-in is called, and the acquired voice information is converted into an audio clip supported by the unity3D platform through the multi-person voice communication plug-in, and the audio clip is transmitted to other end VR devices for playback.

[0034] In one implementation, the transmitting the audio clip to another end VR device for playing includes:

[0035] Determine the lip shape of the virtual character that matches the audio clip based on the lip shape matching plug-in of the unity3D platform;

[0036] A virtual character lip movement model corresponding to the audio clip is generated according to the lip shape of the virtual character, and the virtual character model corresponding to the speaker is identified by a marker.

[0037] In a second aspect, the present invention provides a multi-scene semi-synchronous virtual classroom device based on a VR head display device, comprising:

[0038] The connection module is used to read the initial interface of the virtual classroom, establish a courseware connection between the teacher's terminal and the student's terminal, and enter the preset virtual classroom scene;

[0039] A communication connection module, used to establish a communication connection between the teacher terminal and all student terminals, and adjust the seat and viewing angle of the corresponding student terminal according to the selected seat anchor point;

[0040] A teaching file module, used for obtaining the teaching file in the teacher's terminal, converting the teaching file according to the function of the preset database, and displaying the converted teaching file at a designated position of the preset virtual classroom scene;

[0041] The voice interaction module is used to obtain the voice information of the teacher end or the current student end, convert the obtained voice information, and transmit the converted voice information to other ends to realize the voice interaction of the virtual classroom.

[0042] In a third aspect, the present invention provides a terminal, comprising: a processor and a memory, wherein the memory stores a multi-scene semi-synchronous virtual classroom program based on a VR head display device, and when the multi-scene semi-synchronous virtual classroom program based on a VR head display device is executed by the processor, it is used to implement the operation of the multi-scene semi-synchronous virtual classroom method based on a VR head display device as described in the first aspect.

[0043] In a fourth aspect, the present invention further provides a medium, which is a computer-readable storage medium, and which stores a multi-scene semi-synchronous virtual classroom program based on a VR head-mounted display device. When the multi-scene semi-synchronous virtual classroom program based on a VR head-mounted display device is executed by a processor, it is used to implement the operation of the multi-scene semi-synchronous virtual classroom method based on a VR head-mounted display device as described in the first aspect.

[0044] The present invention adopts the above technical solution to achieve the following effects:

[0045] (1) The present invention establishes a courseware connection between the teacher end and the student end to enter a preset virtual classroom scene, thereby realizing an ordinary classroom scene, a forest classroom scene, a desert classroom scene, an ocean classroom scene, and a space classroom scene, thereby meeting the needs of diversified classroom scenes and solving the problem of the boredom of a single teaching scene.

[0046] (2) The present invention establishes a communication connection between the teacher end and all student ends, and adjusts the seat and viewing angle of the corresponding student end according to the selected seat anchor point, so that students can fully communicate and interact with the teacher in the virtual classroom teaching scene, and switch seats and viewing angles according to their own needs.

[0047] (3) The present invention obtains the teaching files in the teacher's terminal and converts the teaching files according to the functions of the preset database, so that the teacher can play PPT courseware in the virtual classroom to give lectures, thereby reducing the teacher's learning cost.

[0048] (4) The present invention realizes voice interaction in a virtual classroom by acquiring voice information from a teacher or current student, converting the acquired voice information, and transmitting the converted voice information to other terminals, thereby enabling people in the virtual classroom to determine the object of current interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0050] Figure 1 It is a flow chart of a multi-scene semi-synchronous virtual classroom method based on a VR head display device in the present invention.

[0051] Figure 2 It is a schematic diagram of the initial interface of the virtual classroom in the present invention.

[0052] Figure 3 It is a schematic diagram of a scene of position selection in the present invention.

[0053] Figure 4 It is a schematic diagram of a scene in which a speaker is identified by a marker in the present invention.

[0054] Figure 5 It is a schematic diagram of the virtual classroom scene switching in the present invention.

[0055] Figure 6 It is a schematic diagram of seats being occupied and position switching in a virtual classroom scene in the present invention.

[0056] Figure 7 It is a functional principle diagram of a terminal in one implementation of the present invention.

[0057] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] Exemplary Methods

[0060] The existing teaching methods of VR virtual classrooms are not real-time listening students, so voice communication is not possible, only text comments can be used, and the teaching content is the video content played, and there is no real-time teacher teaching. Moreover, this application scenario based on virtual reality teaching only has a single classroom. For lower-grade students in elementary and junior high schools, a single classroom teaching scene is too boring and cannot arouse students' interest in online learning; furthermore, this application based on virtual reality teaching does not have the function of playing PPT courseware. Because VR glasses have not yet been widely popularized, many teachers' teaching PPT courseware are traditional PPT files. It is relatively costly for teachers to make and use VR-PPT. At the same time, this application based on virtual reality teaching cannot be fully interactive. It simply plays videos in virtual scenes. Students cannot adjust their seats like in real classes, so as to realize the free switching of positions and listening perspectives in teaching scenes; therefore, these existing VR virtual classrooms and teaching methods cannot meet the needs of teachers and students for classroom scene diversification, teaching voice interaction, teaching PPT courseware display, and seat perspective switching.

[0061] In view of the above technical problems, an embodiment of the present invention provides a multi-scene semi-synchronous virtual classroom method based on a VR head display device. The method enters a preset virtual classroom scene by reading the initial interface of the virtual classroom and establishing a courseware connection between the teacher end and the student end; establishes a communication connection between the teacher end and all student ends, and adjusts the seat and perspective of the corresponding student end according to the selected seat anchor point; obtains the teaching file in the teacher end, and converts the teaching file according to the function of the preset database, and displays the converted teaching file at the specified position of the preset virtual classroom scene; obtains the voice information of the teacher end or the current student end, and converts the obtained voice information, and transmits the converted voice information to other ends to realize voice interaction in the virtual classroom. The present invention realizes the personalized needs of virtual classroom teaching scenes with diversified classroom scenes, teaching voice interaction, teaching PPT courseware display, and seat perspective switching.

[0062] like Figure 1 As shown, an embodiment of the present invention provides a multi-scene semi-synchronous virtual classroom method based on a VR head display device, comprising the following steps:

[0063] Step S100, read the initial interface of the virtual classroom, establish a courseware connection between the teacher's terminal and the student's terminal, and enter the preset virtual classroom scene.

[0064] In this embodiment, development is performed based on the technology of existing VR glasses, and real-time teaching is carried out in a purely virtual classroom, so that people can be completely immersed in a virtual classroom for learning, thereby solving the problem of insufficient immersion in two-dimensional picture teaching.

[0065] Specifically, in an implementation of this embodiment, step S100 includes the following steps:

[0066] Step S101, configuring and reading the initial interface of the virtual classroom based on the unity3D platform;

[0067] Step S102, determining the teaching channel, preset virtual classroom scene and virtual character image selected in the virtual classroom initial interface;

[0068] Step S103, establishing a courseware connection between the teacher terminal and all student terminals through a synchronization plug-in of a multi-player game scene, and entering the preset virtual classroom scene according to the scene option.

[0069] In this embodiment, the unity3D platform (a real-time 3D interactive content creation and operation platform) is used to design and develop the initial interface of the virtual classroom, and the corresponding buttons (for example, setting and selecting buttons for teaching channels, virtual classroom scenes, and virtual characters, buttons for entering virtual classroom scenes, etc.) are designed on the UI interface to respond to the operation of entering the virtual classroom. By setting the compilation platform of the unity3D platform to Android, the program is installed in the VR glasses; when the program is started in the VR glasses, the initial interface of the virtual classroom is read, so as to determine the teaching channel, preset virtual classroom scene, and virtual character image selected in the initial interface of the virtual classroom according to the operation of the user (student or teacher), so as to enter the corresponding virtual classroom scene.

[0070] In this embodiment, a plurality of virtual classroom scenes are pre-set in the initial interface of the virtual classroom, including: any one of a common classroom scene, a forest classroom scene, a desert classroom scene, an ocean classroom scene, and a space classroom scene.

[0071] As an example, the initial interface of the virtual classroom in this embodiment includes: a student end and a teacher end; wherein the initial interface functions of the virtual classroom on the student end include:

[0072] 1. Select the teaching channel, that is, enter the classroom number, enter the corresponding classroom and take the class in that classroom.

[0073] 2. Classroom environment selection: You can choose your favorite classroom scene such as space, ocean, forest, desert, etc.

[0074] 3. Character image selection: you can choose your favorite virtual avatar in the VR space. 4. Character nickname input.

[0075] The initial interface functions of the virtual classroom on the teacher side include: 1. Creation of teaching channels and classroom numbers for classes.

[0076] 2. Character selection.

[0077] 3. Upload the PPT to the classroom.

[0078] This embodiment pre-sets multiple virtual classroom scenes in the initial interface of the virtual classroom, so that the teacher and all students can choose their favorite classroom scenes, thereby meeting the needs of diversified classroom scenes and solving the problem of the boredom of a single teaching scene.

[0079] In this embodiment, after completing the corresponding operation of entering the virtual classroom on the virtual classroom initial interface, a courseware connection is established between the teacher end and all student ends; specifically, the operating end (teacher end or student end) connects to the server and establishes contact with other students (teachers), thereby entering the preset virtual classroom scene.

[0080] As an example, the photon plug-in (a common programming plug-in for multiplayer game scene synchronization) is used in this embodiment to connect to the photon server before the user enters the classroom, so that a courseware connection is established between multiple users, preparing for the subsequent synchronization of teaching courseware.

[0081] It is worth mentioning that in the process of establishing the courseware connection between the teacher terminal and all student terminals in this embodiment, the scene of each student terminal can be different, but the courseware played by the teacher is the same. After the courseware connection is established, the classroom scene entered by the student terminal is a virtual classroom scene of one of the multiple preset scenes selected in the virtual classroom initial interface.

[0082] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the multi-scene semi-synchronous virtual classroom method based on the VR head display device also includes the following steps:

[0083] Step S200, establishing a communication connection between the teacher terminal and all student terminals, and adjusting the seat and viewing angle of the corresponding student terminal according to the selected seat anchor point.

[0084] Specifically, in an implementation of this embodiment, step S200 includes the following steps:

[0085] Step S201, calling a multi-person voice communication plug-in;

[0086] Step S202, according to the selected teaching channel, using the multi-person voice communication plug-in to establish a communication connection between the teacher terminal and all student terminals;

[0087] Step S203, obtaining the position of the virtual parabola emitted by the control device, and determining the currently selected seat according to the seat anchor point touched by the virtual parabola position;

[0088] Step S204, based on the virtual and reality interaction plug-in of the unity3D platform, controls the first-person perspective of the corresponding student terminal, adjusts the seat of the corresponding student terminal to the currently selected seat, and moves the first-person perspective of the corresponding student terminal to the perspective of the currently selected seat.

[0089] Specifically, in an implementation of the present embodiment, the seat of the corresponding student terminal is adjusted to the currently selected seat so that the first-person perspective of the corresponding student terminal moves to the perspective of the currently selected seat, including: numbering all seats in the preset virtual classroom scene and obtaining the coordinates of each seat in the virtual world; marking all seats with preset marks and forming a virtual character and seat mapping array; according to the virtual character and seat mapping array, detecting whether there are virtual characters from other student terminals sitting in the currently selected seat; if there are virtual characters from other student terminals sitting in the currently selected seat, mapping the virtual character corresponding to the original seat to other empty seats, and adjusting the seat of the student terminal that selects the seat to the currently selected seat, so that the first-person perspective of the corresponding student terminal moves to the perspective of the currently selected seat.

[0090] In this embodiment, after establishing the courseware connection between the teacher terminal and the student terminal, it is also necessary to establish a communication connection between the teacher terminal and all the student terminals.

[0091] As an example, you can use the Agora plug-in (a common programming plug-in for multi-person voice communication) to achieve communication connection, that is, use the Agora SDK (Software Development Kit of the Agora plug-in) to programmatically establish voice communication between users; specifically: first, import the downloaded SDK (Software Development Kit) into the unity3D platform. Then, copy the AppID (software name) and Token (logo) from the newly created project and paste them into the corresponding position of the project; finally, programmatically set up the channel interface, connect with the Agora SDK, and when running the program, enter the channel number through human-computer interaction, and then automatically call the voice communication established by the Agora plug-in.

[0092] In this embodiment, after establishing the communication connection between the teacher end and all the student ends, the seat selection is performed using the handle (i.e., control device) provided with the VR glasses, and a corresponding anchor point is set for each seat in the unity3D platform, and the seat selection is performed by making the virtual parabola emitted by the handle contact the anchor point. At the same time, for the student end that selects the seat, its viewing angle in the virtual teacher scene is adjusted to the corresponding position where it is sitting.

[0093] As an example, the LocomotionSystem (a control script for virtual objects) in the XR.Interaction.Toolkit plug-in (a plug-in for developing interactive functions of augmented reality and virtual reality applications) in the unity3D platform is used to control the first-person position of the student, move the student to the selected position, and move the first-person perspective to the corresponding position. On the corresponding student side, when the coordinates of the glasses change in the virtual space, the picture will automatically re-render and adjust the perspective to the current position.

[0094] This embodiment establishes a communication connection between the teacher end and all student ends, and adjusts the seat and viewing angle of the corresponding student end according to the selected seat anchor point, so that students can fully communicate and interact with the teacher in the virtual classroom teaching scenario, and switch seats and viewing angles according to their own needs.

[0095] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the multi-scene semi-synchronous virtual classroom method based on the VR head display device also includes the following steps:

[0096] Step S300, obtaining the teaching file in the teacher's terminal, converting the teaching file according to the function of the preset database, and displaying the converted teaching file at a designated position of the preset virtual classroom scene.

[0097] Specifically, in an implementation of this embodiment, step S300 includes the following steps:

[0098] Step S301, obtaining the teaching file in the teacher terminal;

[0099] Step S302, importing the preset database, and using the function defined in the preset database to convert the teaching file into multiple pictures;

[0100] Step S303, converting the converted multiple pictures into a dynamic interactive form to obtain a teaching file displayed in the preset virtual classroom scene;

[0101] Step S304: synchronizing the teaching file displayed in the preset virtual classroom scene to all student terminals, and displaying it at a designated position of the virtual classroom scene of each student terminal.

[0102] In this embodiment, in addition to the above-mentioned arbitrary switching of virtual classroom scenes, voice interaction between teachers and students, and arbitrary switching of student seat perspectives, the teaching file (PPT) on the teacher's side can also be synchronized to a specified location of the virtual classroom scene (for example, the location of the blackboard); specifically, the process of synchronizing the teacher's PPT courseware to the virtual blackboard is as follows:

[0103] 1. The teacher first imports the teaching courseware files in .pptx format that are commonly used in normal teaching.

[0104] 2. By introducing the Microsoft.Office.Interop.PowerPoint library (that is, the preset PPT database), and using the functions defined in the library, convert the teaching courseware in .pptx format into png format images.

[0105] 3. Read the png format image as a Texture2D object, save the Texture2D object with the newly created Sprite (a control for drawing atlases in the unity3D platform), thereby converting the png format file into Sprite, and finally displaying the Sprite format teaching courseware in VR.

[0106] 2. Synchronize PPT files through the photon server. The photon plug-in is a commonly used screen synchronization server in the unity3D platform, which can synchronize the PPT screen on the teacher's side to the PPT screen of other students.

[0107] This embodiment obtains the teaching files in the teacher's terminal and converts the teaching files according to the functions of the preset database, so that the teacher can play PPT courseware in the virtual classroom for teaching, thereby reducing the teacher's learning cost.

[0108] like Figure 1 As shown, in one implementation of the embodiment of the present invention, the multi-scene semi-synchronous virtual classroom method based on the VR head display device also includes the following steps:

[0109] Step S400, obtaining voice information of the teacher terminal or the current student terminal, converting the obtained voice information, and transmitting the converted voice information to other terminals to realize voice interaction in the virtual classroom.

[0110] Specifically, in an implementation of this embodiment, step S400 includes the following steps:

[0111] Step S401, obtaining voice information of the teacher terminal or the current student terminal;

[0112] Step S402, calling a multi-person voice communication plug-in, converting the acquired voice information into an audio clip supported by the unity3D platform through the multi-person voice communication plug-in, and transmitting the audio clip to other end VR devices for playback.

[0113] Specifically, in an implementation of this embodiment, the audio clip is transmitted to other VR devices for playback, including: determining the lip shape of a virtual character matching the audio clip based on a lip shape matching plug-in of the unity3D platform; generating a virtual character lip movement model corresponding to the audio clip according to the lip shape of the virtual character, and identifying the virtual character model corresponding to the speaker through a marker.

[0114] In this embodiment, during the lecture in the virtual classroom, the voice communication function provided in the SDK of the Agora plug-in is used to push the local voice to other terminals. At the same time, the local terminal obtains the original audio data of the teacher and students and converts it into the AudioClip type (an audio clip) used in unity3D. Finally, the audio is played on the VR glasses to realize the function of voice interaction.

[0115] During the voice interaction, a marker is displayed to identify the speaker and generate the lip movements of the virtual character. Specifically, the lip movements of the virtual character are generated in real time by using the SALSA With RandomEyes plug-in (a lightweight lip matching plug-in based on unity3D that can match the lip shape according to the voice data), making the conversation more realistic. At the same time, a marker is displayed above the head of the speaking student to identify the speaker, such as Figure 4 As shown, the blue diamond on the character’s head is the marker.

[0116] During the entire virtual classroom teaching process, based on the student's operation, it is determined whether the student wants to reselect a seat. If the student wants to reselect a seat, the student reselects the seat. If the student does not want to reselect a seat, it is determined based on the student's operation whether the student wants to exit the class. If so, the program ends. Otherwise, the program returns to the step of determining whether someone is speaking and repeats the cycle.

[0117] The following describes the above solution by taking specific application scenarios as examples:

[0118] Application scenario 1:

[0119] A. Enter the initial interface of the virtual classroom: Use the unity3D platform to design and develop the initial interface, and design buttons on the UI interface to respond to the operation of entering the virtual classroom. Set the compilation platform of unity3D to Android and install the program in the VR glasses.

[0120] B. Connect to the server, establish connections with other students, and enter the teaching scene: Use the photon plug-in to connect to the photon server before the user enters the classroom, so that connections can be established between multiple users, preparing for the synchronization of subsequent teaching courseware.

[0121] C. Establish a voice communication connection: Use Agora SDK programming to establish voice communication between users.

[0122] D. Select location: Figure 3 As shown, the position is selected using the handle that comes with the VR glasses, a corresponding anchor point is set for each seat in Unity3D, and the position is selected by the virtual parabola emitted by the handle contacting the anchor point.

[0123] E. Adjust the perspective to the corresponding sitting position: Use the LocomotionSystem in the XR.Interaction.Toolkit plug-in in unity3D to control the student's first-person position, move the student to the selected position, and move the first-person perspective to the corresponding position.

[0124] F. Get the voice of the teacher and student who are speaking: Use the voice communication function provided in the SDK of the Agora plug-in to push the local voice to other terminals, and determine whether there are teachers and students speaking at other terminals. If there are, the local terminal obtains the original audio data of the teacher and student who are speaking, and then converts it to the AudioClip type used in unity3D, and finally plays the audio on the VR glasses. If not, continue to determine whether the student wants to leave the class. If so, end the program. If not, return to determine whether there is a speaker, and continue the cycle.

[0125] Application scenario 2:

[0126] A. Enter the initial interface of the virtual classroom: Use the unity3D platform to design and develop the initial interface, and design buttons on the UI interface to respond to the operation of entering the virtual classroom. Set the compilation platform of unity3D to Android and install the program in the VR glasses.

[0127] G. Select the teaching scene and fill in your nickname: interact with the virtual interface by controlling the handle's ray, and respond to the click event of the handle's ray by using the XRUIInputModule (XRUI interface input module) in the XR.Interaction.Toolkit plug-in in unity3D, such as Figure 2 As shown, click on the scene to select it, click on the input box to call out the virtual keyboard, and input by clicking on the letters on the virtual keyboard. Among them, the teaching scenes can include ordinary classrooms, forests, deserts, oceans, space and other scenes. The teaching scene is 3D, and the designer can design preset scenes for students to choose their favorite scenes, such as forest scenes, for teaching.

[0128] like Figure 5 As shown in the figure, the scene switching seen in VR glasses. Use the buttons on the handle to call out the scene selection interface at the top, and then select other scenes. The top of the left picture is the scene selection interface, and the right picture is the new scene switched after selecting other scenes.

[0129] B. Connect to the server, establish connections with other students, and enter the teaching scene: Use the photon plug-in to connect to the photon server before the user enters the classroom, so that connections can be established between multiple users, preparing for the synchronization of subsequent teaching courseware.

[0130] C. Establish voice communication connection: Use Agora SDK programming to establish voice communication between users.

[0131] D. Select the position: Use the handle that comes with the VR glasses to select the position. Set a corresponding anchor point for each seat in Unity3D. The virtual parabola emitted by the handle contacts the anchor point to select the position.

[0132] H. Map the original student at the selected position to other empty seats: Determine whether there are other students at the position selected by the current student. If so, move the original student at the selected position to the empty seat so that the current student can sit in the position. If not, proceed to the next step.

[0133] like Figure 6 As shown, Figure 6 The middle is the position of the seated person as seen through the VR glasses. The position of the person in the red frame on the left is where the current student wants to sit. The right picture shows that the student has achieved the position. The red frame is the student who was originally in the position and has been moved to another position.

[0134] E. Adjust the perspective to the corresponding sitting position: Use the LocomotionSystem in the XR.Interaction.Toolkit plug-in in unity3D to control the student's first-person position, move the student to the selected position, and move the first-person perspective to the corresponding position.

[0135] F. Get the voice of the teacher or student: Use the voice communication function provided in the SDK of the Agora plug-in to push the local voice to other terminals, and determine whether there are teachers or students speaking at other terminals. If so, the local terminal obtains the original audio data of the teacher or student speaking and converts it into the AudioClip type used in unity3D, and finally plays the audio on the VR glasses.

[0136] I. Display markers to identify the speaker and generate lip movements of the virtual character: Use the SALSAWithRandomEyes plug-in to generate the lip movements of the virtual character in real time, making the conversation more realistic. At the same time, display markers above the head of the speaking student to identify the speaker, such as Figure 4 As shown, the blue diamond on the head of the character is the marker. Afterwards, it is determined whether the student wants to reselect a position. If so, it returns to step D to select a position. If not, it is determined whether to exit the class. If so, the program ends. Otherwise, it returns to the step of determining whether someone is speaking and repeats the cycle.

[0137] It can be understood that application scenario 2 is a further technical solution of application scenario 1, which includes step G, selecting the teaching scenario and filling in your own nickname; step H, mapping the original students in the selected position to other empty positions; step I, displaying markers to identify the speaker and generating lip movements of the virtual character.

[0138] Application Scenario 3

[0139] Embodiment 3:

[0140] Application scenario 3 is a further technical solution for application scenario 2: Step H is the core innovation point. For step H, mapping the original students at the selected positions to other empty positions is further described.

[0141] H1: Number all positions in preparation for subsequent processing.

[0142] H2: Get the coordinates of each seat in the virtual world.

[0143] H3: Set a flag f for each seat to record whether there is someone sitting in the seat. If the flag f is true, there is someone sitting in the seat, and if it is false, there is no one sitting in the seat.

[0144] H4: Initialize the scene and set the seat flag f that is occupied to true.

[0145] H5: Define and initialize the person-seat mapping array to store who is sitting in which seat;

[0146] The specific definition and process of mapping arrays between people and seats is as follows: using an array, for example, if there are 18 seats, the seats are numbered 1-18, and a one-dimensional array with a length of 18 is established. First, all values ​​in the array are initialized to false. When the first person enters the classroom, the seat number of his position is set to true. When the second person enters, his array is initialized to all seat numbers from 1-18 except the position of the previous person, which is true, and all others are false. When the third person enters, his array is initialized to all seat numbers from 1-18 except the position of the previous two people, which is true, and all others are false. Initialization is performed in this way, and a new array is maintained for each additional person. For the first person who enters first, two people enter the classroom behind him. In the corresponding array and perspective, the two people will be randomly assigned to different unoccupied seats, and the corresponding array will also change, and the array for the seat number of someone is true. For the second person, one person enters the classroom behind him, and in his array and perspective, this person will be randomly assigned to an unoccupied seat, and the corresponding array will also change. There is a one-to-one correspondence between whether there is someone in the seat and whether the seat number is true. An array is used to maintain this mapping relationship. Firstly, in programming, whether there is someone in the seat needs to be abstracted into an array that the computer can understand. Secondly, by maintaining the array, it is possible to know whether there is someone in the seat by traversing the array, which makes it convenient to change seats.

[0147] H6: Check the seat number n selected by the current student, and then determine whether there is someone in the seat number n. If yes, execute H7-H9, otherwise execute H10 directly.

[0148] H7: Based on the person-seat mapping array generated by E5, get the student r sitting in seat number n.

[0149] H8: Loop through the flag f of each seat and find the first seat where f is false, that is, the empty seat s.

[0150] H9: Move student r to seat s and set flag f of seat s to true.

[0151] H10: Adjust the current student's position to the selected seat n, and adjust the viewing angle to the viewing angle of the selected seat n.

[0152] H11: Set the original seat of the current student to false. If the student is choosing a seat for the first time, this step is not required because the student has not sat down yet and the original seat does not exist.

[0153] After that, the process loops to determine whether to reselect a seat. If yes, execute E6. Otherwise, the process continues to determine whether to exit the class. If yes, the process ends. Otherwise, the process returns to determining whether to reselect a seat and continues the loop until the student exits the class.

[0154] It is worth mentioning that the real-time voice communication technology in this embodiment can be implemented using not only the Agora plug-in but also third-party plug-ins such as the uSpeak plug-in and Mouxun Voice.

[0155] The teaching courseware synchronization technology in this embodiment can be implemented by using the UnityMultiplayer plug-in (a function in the unity3D engine for creating multiplayer games) or a self-written server in addition to using the photon plug-in.

[0156] In addition to using the SALSA With RandomEyes plug-in, the virtual human lip movement technology in this embodiment can also be implemented using the SALSA LipSync Suite plug-in (a tool widely used in animation production and game development, its main function is to synchronize the lips of two-dimensional or three-dimensional characters) and the OVRLipSync plug-in (a lip synchronization plug-in developed by Oculus for Unity).

[0157] This embodiment achieves the following technical effects through the above technical solution:

[0158] (1) This embodiment establishes a courseware connection between the teacher end and the student end to enter the preset virtual classroom scene, realizing ordinary classroom scenes, forest classroom scenes, desert classroom scenes, ocean classroom scenes and space classroom scenes, thereby meeting the needs of diversified classroom scenes and solving the problem of the boredom of a single teaching scene.

[0159] (2) This embodiment establishes a communication connection between the teacher end and all student ends, and adjusts the seat and viewing angle of the corresponding student end according to the selected seat anchor point, so that students can fully communicate and interact with the teacher in the virtual classroom teaching scene, and switch seats and viewing angles according to their own needs.

[0160] (3) This embodiment obtains the teaching files in the teacher's terminal and converts the teaching files according to the functions of the preset database, so that the teacher can play PPT courseware in the virtual classroom to teach, thereby reducing the teacher's learning cost.

[0161] (4) This embodiment obtains voice information from the teacher or the current student, converts the obtained voice information, and transmits the converted voice information to other terminals to achieve voice interaction in the virtual classroom, so that people in the virtual classroom can determine the current interactive object.

[0162] Exemplary Devices

[0163] Based on the above embodiments, the present invention further provides a multi-scene semi-synchronous virtual classroom device based on a VR head display device, comprising:

[0164] The connection module is used to read the initial interface of the virtual classroom, establish a courseware connection between the teacher's terminal and the student's terminal, and enter the preset virtual classroom scene;

[0165] A communication connection module, used to establish a communication connection between the teacher terminal and all student terminals, and adjust the seat and viewing angle of the corresponding student terminal according to the selected seat anchor point;

[0166] A teaching file module, used for obtaining the teaching file in the teacher's terminal, converting the teaching file according to the function of the preset database, and displaying the converted teaching file at a designated position of the preset virtual classroom scene;

[0167] The voice interaction module is used to obtain the voice information of the teacher end or the current student end, convert the obtained voice information, and transmit the converted voice information to other ends to realize the voice interaction of the virtual classroom.

[0168] This embodiment achieves the following technical effects through the above technical solution:

[0169] (1) This embodiment establishes a courseware connection between the teacher end and the student end to enter the preset virtual classroom scene, realizing ordinary classroom scenes, forest classroom scenes, desert classroom scenes, ocean classroom scenes and space classroom scenes, thereby meeting the needs of diversified classroom scenes and solving the problem of the boredom of a single teaching scene.

[0170] (2) This embodiment establishes a communication connection between the teacher end and all student ends, and adjusts the seat and viewing angle of the corresponding student end according to the selected seat anchor point, so that students can fully communicate and interact with the teacher in the virtual classroom teaching scene, and switch seats and viewing angles according to their own needs.

[0171] (3) This embodiment obtains the teaching files in the teacher's terminal and converts the teaching files according to the functions of the preset database, so that the teacher can play PPT courseware in the virtual classroom to teach, thereby reducing the teacher's learning cost.

[0172] (4) This embodiment obtains voice information from the teacher or the current student, converts the obtained voice information, and transmits the converted voice information to other terminals to achieve voice interaction in the virtual classroom, so that people in the virtual classroom can determine the current interactive object.

[0173] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be as follows: Figure 7 shown.

[0174] The terminal includes: a processor, a memory, an interface, a display screen and a communication module connected through a system bus; wherein the processor of the terminal is used to provide computing and control capabilities; the memory of the terminal includes a storage medium and an internal memory; the storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the storage medium; the interface is used to connect to external devices; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or other devices.

[0175] When the computer program is executed by a processor, it is used to implement the operation of a multi-scene semi-synchronous virtual classroom method based on a VR head display device.

[0176] It can be understood by those skilled in the art that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0177] In one embodiment, a terminal is provided, which includes: a processor and a memory, the memory storing a multi-scene semi-synchronous virtual classroom program based on a VR head display device, and the multi-scene semi-synchronous virtual classroom program based on the VR head display device is used to implement the operation of the multi-scene semi-synchronous virtual classroom method based on the VR head display device when executed by the processor.

[0178] In one embodiment, a storage medium is provided, wherein the storage medium stores a multi-scene semi-synchronous virtual classroom program based on a VR head display device. When the multi-scene semi-synchronous virtual classroom program based on the VR head display device is executed by a processor, it is used to implement the operation of the multi-scene semi-synchronous virtual classroom method based on the VR head display device as described above.

[0179] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a non-volatile storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and volatile memory.

[0180] In summary, the present invention provides a multi-scene semi-synchronous virtual classroom method based on a VR head display device, including: reading the initial interface of the virtual classroom, and establishing a courseware connection between the teacher end and the student end, and entering a preset virtual classroom scene; establishing a communication connection between the teacher end and all student ends, and adjusting the seat and perspective of the corresponding student end according to the selected seat anchor point; obtaining the teaching file in the teacher end, and converting the teaching file according to the function of the preset database, and displaying the converted teaching file at the specified position of the preset virtual classroom scene; obtaining the voice information of the teacher end or the current student end, and converting the obtained voice information, transmitting the converted voice information to other ends, and realizing voice interaction in the virtual classroom. The present invention realizes the personalized needs of the virtual classroom teaching scene with diversified classroom scenes, teaching voice interaction, teaching PPT courseware display and seat perspective switching.

[0181] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A multi-scene semi-synchronous virtual classroom method based on VR head display device, characterized in that: include: Read the initial interface of the virtual classroom, establish a courseware connection between the teacher and the student, and enter the preset virtual classroom scene; Establishing a communication connection between the teacher terminal and all student terminals, and adjusting the seats and viewing angles of the corresponding student terminals according to the selected seat anchor points; Acquire the teaching file in the teacher terminal, convert the teaching file according to the function of the preset database, and display the converted teaching file at the designated position of the preset virtual classroom scene; The voice information of the teacher terminal or the current student terminal is obtained, and the obtained voice information is converted, and the converted voice information is transmitted to other terminals to realize the voice interaction of the virtual classroom.

2. The multi-scene semi-synchronous virtual classroom method based on VR head display device according to claim 1 is characterized in that: The process of reading the initial interface of the virtual classroom, establishing a courseware connection between the teacher terminal and the student terminal, and entering the preset virtual classroom scene includes: Configure and read the initial interface of the virtual classroom based on the unity3D platform; Determine the teaching channel, preset virtual classroom scene and virtual character image selected in the initial interface of the virtual classroom; wherein the preset virtual classroom scene includes: any one of an ordinary classroom scene, a forest classroom scene, a desert classroom scene, an ocean classroom scene and a space classroom scene; A courseware connection is established between the teacher terminal and all student terminals through a synchronization plug-in of a multiplayer game scene, and the preset virtual classroom scene is entered according to the scene option.

3. The multi-scene semi-synchronous virtual classroom method based on VR head display device according to claim 1 is characterized in that: The establishing of the communication connection between the teacher terminal and all the student terminals, and adjusting the seats and viewing angles of the corresponding student terminals according to the selected seat anchor points, includes: A communication plug-in for calling multi-person voice; According to the selected teaching channel, using the multi-person voice communication plug-in to establish a communication connection between the teacher terminal and all student terminals; Acquire the position of a virtual parabola emitted by the control device, and determine the currently selected seat according to the seat anchor point touched by the virtual parabola position; The virtual and reality interaction plug-in based on the unity3D platform controls the first-person perspective of the corresponding student end, adjusts the seat of the corresponding student end to the currently selected seat, and moves the first-person perspective of the corresponding student end to the perspective of the currently selected seat.

4. The multi-scene semi-synchronous virtual classroom method based on VR head display device according to claim 3 is characterized in that: The step of adjusting the seat of the corresponding student terminal to the currently selected seat so that the first-person perspective of the corresponding student terminal moves to the perspective of the currently selected seat includes: Numbering all seats in the preset virtual classroom scene and obtaining the coordinates of each seat in the virtual world; All seats are marked by preset signs, and an array of virtual characters and seat mapping is formed; According to the virtual character and seat mapping array, detecting whether the currently selected seat has a virtual character from another student terminal seated; If the currently selected seat is occupied by a virtual character from another student, the virtual character corresponding to the original seat will be mapped to another empty seat, and the seat of the student who selected the seat will be adjusted to the currently selected seat, so that the first-person perspective of the corresponding student will be moved to the perspective of the currently selected seat.

5. The multi-scene semi-synchronous virtual classroom method based on VR head display device according to claim 1 is characterized in that: The step of obtaining the teaching file in the teacher terminal, converting the teaching file according to a function of a preset database, and displaying the converted teaching file at a designated location of the preset virtual classroom scene includes: Obtaining the teaching files in the teacher terminal; Importing the preset database, and using the function defined in the preset database to convert the teaching file into a plurality of pictures; Performing dynamic interactive conversion on the converted multiple images to obtain a teaching file displayed in the preset virtual classroom scene; The teaching files displayed in the preset virtual classroom scene are synchronized to all student terminals and displayed at the designated location of the virtual classroom scene of each student terminal.

6. The multi-scene semi-synchronous virtual classroom method based on VR head display device according to claim 1 is characterized in that: The acquiring of the voice information of the teacher terminal or the current student terminal, converting the acquired voice information, and transmitting the converted voice information to other terminals includes: Acquire the voice information of the teacher terminal or the current student terminal; The multi-person voice communication plug-in is called, and the acquired voice information is converted into an audio clip supported by the unity3D platform through the multi-person voice communication plug-in, and the audio clip is transmitted to other end VR devices for playback.

7. The multi-scene semi-synchronous virtual classroom method based on VR head display device according to claim 6 is characterized in that: The transmitting the audio clip to another end VR device for playing includes: Determine the lip shape of the virtual character that matches the audio clip based on the lip shape matching plug-in of the unity3D platform; A virtual character lip movement model corresponding to the audio clip is generated according to the lip shape of the virtual character, and the virtual character model corresponding to the speaker is identified by a marker.

8. A multi-scene semi-synchronous virtual classroom device based on a VR head display device, characterized in that: include: The connection module is used to read the initial interface of the virtual classroom, establish a courseware connection between the teacher's terminal and the student's terminal, and enter the preset virtual classroom scene; A communication connection module, used to establish a communication connection between the teacher terminal and all student terminals, and adjust the seat and viewing angle of the corresponding student terminal according to the selected seat anchor point; A teaching file module, used for obtaining the teaching file in the teacher's terminal, converting the teaching file according to the function of the preset database, and displaying the converted teaching file at a designated position of the preset virtual classroom scene; The voice interaction module is used to obtain the voice information of the teacher end or the current student end, convert the obtained voice information, and transmit the converted voice information to other ends to realize the voice interaction of the virtual classroom.

9. A terminal, characterized in that: include: A processor and a memory, wherein the memory stores a multi-scene semi-synchronous virtual classroom program based on a VR head display device, and when the multi-scene semi-synchronous virtual classroom program based on a VR head display device is executed by the processor, it is used to implement the operation of the multi-scene semi-synchronous virtual classroom method based on a VR head display device as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multi-scene semi-synchronous virtual classroom program based on a VR head display device. When the multi-scene semi-synchronous virtual classroom program based on a VR head display device is executed by a processor, it is used to implement the operation of the multi-scene semi-synchronous virtual classroom method based on a VR head display device as described in any one of claims 1-7.

Images