Immersive teaching scene rendering method, device, equipment, storage medium and system

Abstract

The invention provides an immersive teaching scene rendering method, device and equipment, a storage medium and a system. The immersive teaching system comprises audio and video acquisition equipment, a data processing server, a scene rendering engine and a plurality of virtual reality terminals. The audio and video acquisition equipment acquires a teaching video; the data processing server performs behavior analysis on the teaching video to obtain first behavior data, the data processing server sends a first signaling to a scene rendering engine, and the first signaling carries the first behavior data; the scene rendering engine renders a first virtual three-dimensional character according to the first behavior data in the target virtual three-dimensional scene according to the first signaling to obtain a first three-dimensional picture; the scene rendering engine sends a first three-dimensional picture to a plurality of virtual reality terminals; and the plurality of virtual reality terminals display the first three-dimensional picture. According to the technical scheme, the virtual three-dimensional character in the virtual three-dimensional scene and the character action in the real environment can be synchronized.

Description

Technical Field

[0001] The present application relates to the field of virtual reality, and in particular to an immersive teaching scene rendering method, device, equipment, storage medium and system. Background Art

[0002] Virtual reality (VR) technology is a computer-generated environment that simulates the real world. By simulating vision, hearing, touch and other senses, users can feel that they are in a virtual environment. In recent years, with the continuous development and progress of VR technology, VR technology has gradually been applied to various fields, one of which is education.

[0003] As an innovative teaching method, VR classroom provides students with a more immersive and interactive learning experience. In VR classroom, teachers can wear VR equipment with students and lead students to experience VR content. After teachers wear VR equipment, they cannot perceive students' behavior in class, which is not conducive to teaching management. Summary of the invention

[0004] The present application provides an immersive teaching scene rendering method, apparatus, device, storage medium and system to solve the technical problem that teachers cannot perceive students' behavior in class after wearing VR equipment.

[0005] In a first aspect, an immersive teaching scene rendering method is provided, which is applied to a scene rendering engine in an immersive teaching system, wherein the immersive teaching system includes an audio and video acquisition device, a data processing server, the scene rendering engine, and a plurality of virtual reality terminals; the method includes:

[0006] receiving a first signaling sent by the data processing server, the first signaling carrying first behavior data, the first behavior data being obtained by the data processing server performing behavior analysis on the teaching video collected by the audio and video collection device, the first behavior data being behavior data corresponding to a first character in the teaching video;

[0007] According to the first signaling, a first virtual three-dimensional character corresponding to the first character is rendered in a target virtual three-dimensional scene according to the first behavior data to obtain a first three-dimensional picture, wherein the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene;

[0008] The first three-dimensional picture is sent to the multiple virtual reality terminals so that the multiple virtual reality terminals display the first three-dimensional picture.

[0009] In the technical solution, the immersive education system includes an audio and video acquisition device, a data processing server, a scene rendering engine and multiple virtual reality terminals; the scene rendering engine receives a first signaling sent by the data processing server, the first signaling carries first behavior data, and the first behavior data is obtained by the data processing server performing a behavior analysis on the teaching video acquired by the audio and video acquisition device; according to the first signaling, a first virtual three-dimensional character corresponding to the first character is rendered in a target virtual three-dimensional scene according to the first behavior data to obtain a first three-dimensional picture, the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing a behavior corresponding to the first behavior data in the target virtual three-dimensional scene; finally, the first three-dimensional picture is sent to multiple virtual reality terminals; due to the first line The data is behavior data corresponding to the first character in the teaching video, and the first three-dimensional picture is a three-dimensional picture of a first virtual three-dimensional character corresponding to the first character performing a behavior corresponding to the first behavior data in a target virtual three-dimensional scene. The first three-dimensional picture is sent to multiple virtual reality terminals, which can achieve action synchronization between the virtual three-dimensional character in the target virtual three-dimensional scene and the character in the real environment. A user wearing a virtual reality terminal can determine the action performed by the character in the real environment by watching the virtual character in the target virtual three-dimensional scene, which is convenient for teachers to perceive the behavior of students in the classroom after wearing VR equipment; in addition, by transmitting signaling carrying behavior data, the user wearing the virtual reality terminal can perceive the behavior of the character in the real environment. Compared with directly transmitting the teaching video, the bandwidth resources required for transmission are less, which is easy to implement.

[0010] In combination with the first aspect, in a possible implementation, before rendering the first virtual three-dimensional character corresponding to the first character in the target virtual three-dimensional scene according to the first behavior data according to the first signaling, it also includes: receiving a target three-dimensional space model sent by the data processing server, the target three-dimensional space model is obtained by the data processing server through three-dimensional reconstruction based on the teaching scene in the teaching video; rendering the target three-dimensional space model to obtain the target virtual three-dimensional scene. Since the target three-dimensional space model is obtained through three-dimensional reconstruction based on the teaching scene in the teaching video, the virtual three-dimensional scene can be made consistent with the real space, thereby enhancing the sense of immersion.

[0011] In combination with the first aspect, in a possible implementation, the brightness of the light source corresponding to the target three-dimensional space model is obtained based on the screen brightness of the teaching video. The brightness of the light source corresponding to the target three-dimensional space model is obtained based on the screen brightness of the teaching video, which can make the brightness of the virtual three-dimensional scene the same as the real environment, thereby enhancing the sense of immersion.

[0012] In combination with the first aspect, in a possible implementation, after rendering the target three-dimensional space model to obtain the target virtual three-dimensional scene, the method further includes: receiving a virtual three-dimensional character model sent by the data processing server, wherein the virtual three-dimensional character model is generated by the data processing server according to the facial image data corresponding to the teaching video, and the facial image data is obtained by the data processing server performing facial target recognition on the teaching video; and rendering the virtual three-dimensional character model in the target virtual three-dimensional scene to obtain a virtual three-dimensional character corresponding to the teaching video. Since the virtual three-dimensional character is generated according to the facial image data obtained by performing facial target recognition on the classroom video, the face of the virtual three-dimensional character in the virtual three-dimensional scene can be consistent with the face in the real environment, thereby enhancing the sense of immersion.

[0013] In combination with the first aspect, the immersive teaching system also includes a three-dimensional asset management platform; the method also includes: in response to a first user instruction, obtaining a first three-dimensional model corresponding to the first user instruction from the three-dimensional asset management platform; rendering the first three-dimensional model in the target virtual three-dimensional scene to obtain a second three-dimensional picture; and sending the second three-dimensional picture to the multiple virtual reality terminals so that the multiple virtual reality terminals display the second three-dimensional picture.

[0014] In a second aspect, an immersive teaching scene rendering device is provided, which is applied to a scene rendering engine in an immersive teaching system, wherein the immersive teaching system includes an audio and video acquisition device, a data processing server, the scene rendering engine, and a plurality of virtual reality terminals; the device includes:

[0015] a receiving module, configured to receive a first signaling sent by the data processing server, the first signaling carrying first behavior data, the first behavior data being obtained by the data processing server performing behavior analysis on the teaching video collected by the audio and video collection device, the first behavior data being behavior data corresponding to a first character in the teaching video;

[0016] a rendering module, configured to render a first virtual three-dimensional character corresponding to the first character according to the first behavior data in a target virtual three-dimensional scene according to the first signaling, to obtain a first three-dimensional picture, wherein the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene;

[0017] The picture sending module is used to send the first three-dimensional picture to the multiple virtual reality terminals so that the multiple virtual reality terminals display the first three-dimensional picture.

[0018] In a third aspect, a computer device is provided, comprising a memory and one or more processors, wherein the memory is connected to the one or more processors, and the one or more processors are used to execute one or more computer programs stored in the memory, and when the one or more processors execute the one or more computer programs, the computer device implements the immersive teaching scene rendering method of the first aspect mentioned above.

[0019] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the immersive teaching scene rendering method of the first aspect.

[0020] In a fifth aspect, an immersive teaching system is provided, including an audio and video acquisition device, a data processing server, a scene rendering engine, and a plurality of virtual reality terminals;

[0021] The audio and video acquisition device is used to collect teaching videos;

[0022] The data processing server is used to perform behavior analysis on the teaching video to obtain first behavior data, where the first behavior data is behavior data corresponding to a first character in the teaching video;

[0023] The data processing server is further used to send a first signaling to the scene rendering engine, where the first signaling carries the first behavior data;

[0024] The scene rendering engine is used to render a first virtual three-dimensional character corresponding to the first character according to the first behavior data in a target virtual three-dimensional scene according to the first signaling to obtain a first three-dimensional picture, wherein the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene;

[0025] The scene rendering engine is further used to send the first three-dimensional picture to the multiple virtual reality terminals;

[0026] The multiple virtual reality terminals are used to display the first three-dimensional image.

[0027] In combination with the fifth aspect, in a possible implementation method, the data processing server is also used to perform three-dimensional reconstruction based on the teaching scene in the teaching video to obtain the target three-dimensional space model; the data processing server is also used to send the target three-dimensional space model to the scene rendering engine; the scene rendering engine is also used to render the target three-dimensional space model to obtain the target virtual three-dimensional scene.

[0028] In conjunction with the fifth aspect, in a possible implementation, the data processing server is further used to determine the brightness of the light source corresponding to the target three-dimensional space model according to the screen brightness of the teaching video.

[0029] In combination with the fifth aspect, in a possible implementation method, the data processing server is also used to perform facial target recognition on the teaching video to obtain facial image data; the data processing server is also used to generate a virtual three-dimensional character model based on the facial image data; the data processing server is also used to send the virtual three-dimensional character model to the scene rendering engine; the scene rendering engine is also used to render the virtual three-dimensional character model in the target virtual three-dimensional scene to obtain a virtual three-dimensional character corresponding to the teaching video.

[0030] In combination with the fifth aspect, in a possible implementation method, the immersive teaching system also includes a three-dimensional asset management platform; the three-dimensional asset management platform is used to store three-dimensional models; the scene rendering engine is also used to respond to a first user instruction, obtain a first three-dimensional model corresponding to the first user instruction from the three-dimensional asset management platform; render the first three-dimensional model in the target virtual three-dimensional scene to obtain a second three-dimensional picture; the scene rendering engine is also used to send the second three-dimensional picture to the multiple virtual reality terminals; the multiple virtual reality terminals are also used to display the second three-dimensional picture.

[0031] In combination with the fifth aspect, in a possible implementation method, the immersive teaching system also includes a blockchain storage platform; the blockchain storage platform is used to store data in the immersive teaching system.

[0032] In combination with the fifth aspect, in a possible implementation, the immersive teaching system also includes an authentication management platform; the authentication management platform is used to perform identity authentication management.

[0033] In combination with the fifth aspect, in a possible implementation, the immersive teaching system also includes a data analysis and display platform; the data analysis and display platform is used to generate and display a teaching scenario report.

[0034] The present application can achieve the following technical effects: since the first behavior data is the behavior data corresponding to the first character in the teaching video, the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character corresponding to the first character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene, and the first three-dimensional picture is sent to multiple virtual reality terminals, the action synchronization between the virtual three-dimensional character in the target virtual three-dimensional scene and the character in the real environment can be achieved, and the user wearing the virtual reality terminal can watch the virtual character in the target virtual three-dimensional scene to determine the action performed by the character in the real environment, which is convenient for teachers to perceive the behavior of students in the classroom after wearing VR equipment; in addition, by transmitting signaling carrying behavior data, the user wearing the virtual reality terminal can perceive the behavior of the character in the real environment. Compared with directly transmitting the teaching video, the bandwidth resources required for transmission are less, which is easy to implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the system architecture of the immersive teaching system provided in the embodiment of the present application;

[0036] Figure 2 A flowchart of an immersive teaching scene rendering method provided in an embodiment of the present application;

[0037] Figure 3 It is a structural schematic diagram of an immersive teaching scene rendering device provided in an embodiment of the present application;

[0038] Figure 4 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0040] The technical solution of the present application is applicable to teaching scenarios, and in particular to teaching scenarios where virtual reality teaching needs to be completed using a virtual reality terminal.

[0041] The present application aims to synchronize the actions of a virtual three-dimensional character in a virtual three-dimensional scene with those of a character in real space in real time, so that a user can perceive the behavior of the character in real space by watching the behavior of the virtual three-dimensional character in the virtual three-dimensional scene, thereby facilitating teachers to manage teaching in a virtual reality teaching scene.

[0042] In order to achieve real-time synchronization of the movements of a virtual 3D character in a virtual 3D scene and a character in a real space, this application proposes an immersive teaching system. Figure 1 , Figure 1 A schematic diagram of the system architecture of the immersive teaching system provided in the embodiment of the present application is shown in FIG. Figure 1As shown, the immersive teaching system 10 includes an audio and video acquisition device 101, a data processing server 102, a scene rendering engine 103 and multiple virtual reality terminals 104.

[0043] The audio and video acquisition device 101 is used to acquire teaching videos. The audio and video acquisition device 101 can be arranged in a classroom to acquire image data and voice data in the classroom to obtain teaching videos and teaching audio. The audio and video acquisition device 101 can include acquisition components such as a camera and a microphone. The microphone can be used to acquire voice data in the classroom, and the camera can be used to acquire image data in the classroom. There can be multiple cameras, and the multiple cameras can be respectively set at multiple positions in the classroom to capture video images in the classroom; in a specific implementation scheme, there can be two cameras, which are respectively set at the front and back positions of the classroom.

[0044] The audio and video acquisition device 101 can acquire teaching videos and teaching audios in real time, and send the acquired teaching videos and teaching audios to the data processing server 102 in real time.

[0045] The data processing server 102 is used to perform various data processing on the teaching video and teaching audio collected by the audio and video acquisition device 101, and obtain the layout information of the classroom space and the character information in the classroom space. The layout information of the classroom space may include the length, width and height of the classroom, the number of rows and columns of desks and chairs in the classroom, the distribution of student seats, the width and height of windows and rain doors, etc. The layout information of the classroom space can be used to reconstruct the classroom space in three dimensions. The layout information of the classroom space can be obtained by analyzing the teaching video based on a preset visual analysis algorithm. The character information in the classroom space may include the face information of the characters in the classroom space, the identity information of the characters in the classroom space, the position information of the characters in the classroom space, the behavior information of the characters in the classroom space, the voice information of the characters in the classroom space, etc. The character information in the classroom space can be obtained by analyzing the teaching video and the teaching audio based on a preset voice recognition algorithm and a visual analysis algorithm.

[0046] There can be one or more data processing servers 102. When there are multiple data processing servers 102, different data processing servers 102 can be used to perform different functions. For example, one of the multiple data processing servers 102 can be used to perform behavior analysis on the teaching video, obtain the behavior data of the characters in the teaching video, that is, obtain the behavior information of the characters in the classroom space; another data processing server among the multiple data processing servers 102 can be used to perform face target recognition on the teaching video, obtain the face image data of the characters in the teaching video, that is, obtain the face information of the characters in the classroom space; another data processing server among the multiple data processing servers 102 can be used to generate a virtual three-dimensional character model corresponding to the teaching video according to the face image data of the characters in the teaching video; another data processing server among the multiple data processing servers 102 can be used to perform three-dimensional reconstruction of the teaching scene (i.e., the classroom) in the teaching video, and obtain the target three-dimensional space model corresponding to the teaching scene; another data processing server among the multiple data processing servers 102 is used to interact with other devices in the immersive teaching system for data, etc., not limited to the description here. Of course, the above-mentioned functions can also be implemented by integrating on a data processing server, and this application does not impose any restrictions.

[0047] After performing three-dimensional reconstruction on the teaching scene in the teaching video and obtaining a target three-dimensional space model corresponding to the teaching scene, the data processing server 102 can send the target three-dimensional space model to the scene rendering engine 103 .

[0048] After generating the virtual three-dimensional character model corresponding to the teaching video, the data processing server 102 may send the virtual three-dimensional character model corresponding to the teaching video to the scene rendering engine 103 .

[0049] After performing behavioral analysis on the teaching video and obtaining the behavioral data of the characters in the teaching video, the data processing server 102 can send a first instruction to the scene rendering engine 103. The first instruction carries first behavioral data. The first behavioral data is the behavioral data corresponding to the first character in the teaching video. The first character can be any character in the teaching video. The first character can be a teacher or a student.

[0050] The scene rendering engine 103 is used to render the target 3D space model sent by the data processing server 102 to obtain a target virtual 3D scene, which is a virtual 3D scene corresponding to the teaching scene in the teaching video. The target virtual 3D scene can be the same as the teaching scene in the teaching video. For example, the teaching scene in the target teaching video is a classroom with a length of 9 meters, a width of 6 meters, and a height of 3 meters, and the classroom contains 5 rows and 6 columns of desks and seats. Then, the target virtual 3D scene in the virtual world is a space with a length of 9 meters, a width of 6 meters, and a height of 3 meters, and there are 5 rows and 6 columns of desks and seats in the space.

[0051] The scene rendering engine 103 is also used to render the virtual three-dimensional character model sent by the data processing server 102 in the target virtual three-dimensional scene to obtain a virtual three-dimensional character corresponding to the teaching video.

[0052] The scene rendering engine 103 is also used to render the first virtual three-dimensional character corresponding to the first character according to the first behavior data in the target virtual three-dimensional scene according to the first signaling sent by the data processing server 102, so as to obtain a first three-dimensional picture, which is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene.

[0053] The scene rendering engine 103 is also used to send the first three-dimensional picture to multiple virtual reality terminals 104.

[0054] The multiple virtual reality terminals 104 are used to display the first three-dimensional image.

[0055] Optionally, the immersive teaching system 10 may further include a 3D asset management platform 105, which is used to store 3D models. The 3D models stored in the 3D asset management platform 105 may be pre-set by a user and saved in the 3D asset management platform.

[0056] The scene rendering engine 103 can also respond to the first user instruction, obtain the first three-dimensional model corresponding to the first user instruction from the three-dimensional asset management platform 105, and render the first three-dimensional model in the target virtual three-dimensional scene to obtain a second three-dimensional picture.

[0057] The scene rendering engine 103 is also used to send the second three-dimensional picture to multiple virtual reality terminals 104.

[0058] The multiple virtual reality terminals 104 are also used to display the second three-dimensional image.

[0059] Optionally, the immersive teaching system 10 may also include a blockchain storage platform 106, which is used to store data in the immersive teaching system 10. The data in the immersive teaching system 10 includes but is not limited to teacher behavior data, student behavior data, and the various data mentioned above.

[0060] The immersive teaching system 10 may further include a data analysis and display platform 107, which is used to generate and display teaching scenario reports. The data analysis and display platform 108 may configure and manage data through big data, and generate teaching scenario reports based on various data generated in the immersive teaching system 10. The teaching scenario reports include but are not limited to learning situation reports, equipment usage reports, etc. The data analysis and display platform 108 may also include a display board for displaying various data.

[0061] Optionally, the immersive teaching system 10 may further include an authentication management platform 108, which is used to manage various teaching information and perform identity authentication management on users entering the immersive teaching system.

[0062] based on Figure 1 The immersive teaching system shown can implement the technical solution of this application. The technical solution of this application is described in detail below. Figure 2 , Figure 2 The present invention provides a flow chart of an immersive teaching scene rendering method, which can be applied to a scene rendering engine in an immersive teaching system. Figure 2 As shown, the method comprises the following steps:

[0063] S201, audio and video acquisition equipment acquires teaching videos.

[0064] Here, audio and video acquisition equipment can capture images in the classroom in real time to obtain teaching videos.

[0065] S202, the audio and video acquisition device sends the teaching video to the data processing server, and the processing server receives the teaching video.

[0066] S203, the data processing server performs three-dimensional reconstruction based on the teaching scene in the teaching video to obtain a target three-dimensional space model.

[0067] The data processing server can perform three-dimensional reconstruction of the teaching scene in the teaching video in the following two ways:

[0068] Method 1: The data processing server can determine the classroom layout information in the teaching video based on the visual depth estimation algorithm. The classroom layout information includes the size information of the classroom space and the distribution of desks and chairs in the classroom; then perform three-dimensional reconstruction based on the classroom layout information in the teaching video to obtain the target three-dimensional space model.

[0069] Method 2: The data processing server can obtain the building information modeling (BIM) drawings corresponding to the teaching video, perform three-dimensional reconstruction according to the BIM drawings, and obtain the target three-dimensional space model.

[0070] In the process of performing three-dimensional reconstruction to obtain the target three-dimensional space model, the data processing server can also determine the brightness of the light source corresponding to the target three-dimensional space model according to the screen brightness of the teaching video. In the process of performing three-dimensional reconstruction to obtain the target three-dimensional space model, the data processing server can also determine the brightness of the teaching scene in the teaching video according to the screen brightness of the teaching video, and determine the brightness of the teaching scene in the teaching video as the brightness of the light source corresponding to the target three-dimensional space model.

[0071] In a feasible implementation, the data processing server can perform color space conversion on the picture in the teaching video to obtain the brightness component information of the picture in the teaching video; then determine the initial brightness information of the picture in the teaching video based on the brightness component information of the picture in the teaching video; then obtain the contrast information and reflection information of the picture in the teaching video; finally, based on the initial brightness information, contrast information and reflection information of the picture in the teaching video, combine the three in a weighted manner to obtain the ambient brightness value corresponding to the teaching video, and the ambient brightness value corresponding to the teaching video is the brightness of the teaching scene in the teaching video.

[0072] In another feasible implementation, the data processing server may also input the picture in the teaching video into a pre-trained environment brightness recognition model to obtain the environment brightness value corresponding to the teaching video. This application does not make any limitation.

[0073] Determining the brightness of the light source corresponding to the target three-dimensional space model according to the brightness of the teaching video can make the brightness of the virtual three-dimensional scene rendered based on the target three-dimensional space model the same as the brightness in the real environment, thereby enhancing the sense of immersion.

[0074] S204: The data processing server sends the target three-dimensional space model to the scene rendering engine, and the scene rendering engine receives the target three-dimensional space model.

[0075] S205: The scene rendering engine renders the target three-dimensional space model to obtain a target virtual three-dimensional scene.

[0076] S206, the data processing server performs face target recognition on the teaching video to obtain face image data.

[0077] Here, performing face target recognition on the teaching video to obtain face image data means performing target detection on the face in the teaching video through a target detection algorithm, detecting the face in the teaching video and the position of the face in the teaching video, and the position of the face in the teaching video may refer to the seat where the face is located in the teaching video. For example, if face A is detected in the teaching video, and face A is located at the position of the seat in the first row and first column in the classroom space, then the seat corresponding to face A is determined to be the first row and first column.

[0078] The target detection algorithm may be any face target detection algorithm.

[0079] S207: The data processing server generates a virtual three-dimensional character model based on the facial image data.

[0080] The data processing server may input the face image data into a pre-trained virtual three-dimensional character model generation model to obtain a virtual three-dimensional character. The virtual three-dimensional character generation model may be, for example, a deepfake model.

[0081] Since there are usually multiple characters in the teaching scene, there are multiple facial image data obtained by the aforementioned step S206. The multiple facial image data can be respectively input into the virtual three-dimensional character generation model obtained by the pre-training to obtain multiple virtual three-dimensional character models. After obtaining multiple virtual three-dimensional character models, the corresponding position of each virtual three-dimensional character can also be determined according to the position of the facial image data corresponding to each virtual three-dimensional character model in the teaching video. For example, the facial image data corresponding to the virtual three-dimensional character model M1 is located in the first row and first column of the teaching video, then the position corresponding to the virtual three-dimensional character model M1 is determined to be the first row and first column.

[0082] S208: The data processing server sends the virtual three-dimensional character model to the scene rendering engine, and the scene rendering engine receives the virtual three-dimensional character model.

[0083] Here, the data processing server may send the generated multiple virtual three-dimensional character models and the position corresponding to each of the multiple virtual three-dimensional character models to the scene rendering engine.

[0084] S209, the scene rendering model renders the virtual three-dimensional character model in the target virtual three-dimensional scene to obtain a virtual three-dimensional character corresponding to the teaching video.

[0085] The scene rendering model can render each virtual 3D character model at the corresponding position in the target virtual 3D scene according to the corresponding position of each virtual 3D character model, so as to obtain the virtual 3D character corresponding to the teaching video. For example, if the position corresponding to the virtual 3D character model M1 is the first row and the first column, then the virtual 3D character model M1 is rendered at the first row and the first column seat in the target virtual 3D scene to obtain the virtual 3D character corresponding to the virtual 3D character model M1.

[0086] S210, the data processing server performs behavior recognition on the teaching video to obtain first behavior data.

[0087] The first behavior data is the behavior data corresponding to the first character in the teaching video, and the first character can be any character in the teaching video. Since there are usually multiple characters in the teaching video, the data processing server obtains multiple first behavior data by performing behavior recognition on the teaching video, and the multiple first behavior data are the behavior data corresponding to each character in the teaching video.

[0088] The data processing server can perform behavior recognition on the teaching video according to any behavior recognition model to obtain first behavior data.

[0089] S211, the data processing server sends a first signaling to the scene rendering engine, the first signaling carries a first behavior of data, and the scene rendering engine receives the first signaling.

[0090] S212: The scene rendering engine renders a first virtual three-dimensional character according to the first behavior data in the target virtual three-dimensional scene according to the first signaling to obtain a first three-dimensional picture.

[0091] The first virtual 3D character is a character corresponding to the first character. For example, if the first behavior data is the behavior data of a student in the first row and first column seat in the teaching video, then the first virtual 3D character is a virtual 3D character in the first row and first column seat in the target virtual 3D scene.

[0092] The first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene. For example, the first behavior data is the behavior data of the student in the first row and first column seat in the teaching video, and the behavior corresponding to the first behavior data is raising a hand, then the first three-dimensional picture is a three-dimensional picture of the virtual three-dimensional character in the first row and first column seat in the target virtual three-dimensional scene raising a hand.

[0093] S213: The scene rendering engine sends the first three-dimensional picture to the multiple virtual reality terminals, and the multiple virtual reality terminals receive the first three-dimensional picture.

[0094] S214, multiple virtual reality terminals display the first three-dimensional image.

[0095] exist Figure 2 In the corresponding technical solution, the scene rendering engine receives a first signaling sent by a data processing server, the first signaling carries first behavior data, and the first behavior data is obtained by the data processing server performing a behavior analysis on the teaching video collected by the audio and video collection device; according to the first signaling, a first virtual three-dimensional character corresponding to the first character is rendered in the target virtual three-dimensional scene according to the first behavior data to obtain a first three-dimensional picture, the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene; finally, the first three-dimensional picture is sent to multiple virtual reality terminals; since the first behavior data is the behavior data corresponding to the first character in the teaching video According to the data, the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character corresponding to the first character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene, and sending the first three-dimensional picture to multiple virtual reality terminals can achieve synchronization of the virtual three-dimensional characters in the target virtual three-dimensional scene with the characters in the real environment, so that users wearing virtual reality terminals can watch the virtual characters in the target virtual three-dimensional scene to determine the actions performed by the characters in the real environment, which is convenient for teachers to perceive the behavior of students in the classroom after wearing VR equipment; in addition, by transmitting signaling carrying behavior data, users wearing virtual reality terminals can perceive the behavior of characters in the real environment. Compared with directly transmitting teaching videos, less bandwidth resources are required for transmission, which is easy to implement.

[0096] Optionally, in some possible cases, the above Figure 2 The corresponding technical solution may further include the following steps S215 to S218:

[0097] S215: In response to the first user instruction, the scene rendering model obtains a first three-dimensional model corresponding to the first user instruction from the three-dimensional asset management platform.

[0098] The first user instruction may be an instruction issued by a teacher.

[0099] S216: The scene rendering model renders the first three-dimensional model in the target virtual three-dimensional scene to obtain a second three-dimensional picture.

[0100] S217: The scene rendering model sends the second three-dimensional picture to the multiple virtual reality terminals, and the multiple virtual reality terminals receive the second three-dimensional picture.

[0101] S218, multiple virtual reality terminals display the second three-dimensional image.

[0102] The method of the present application is introduced above, and the device of the present application is introduced below.

[0103] See also Figure 3 , Figure 3 is a structural diagram of an immersive teaching scene rendering device provided in an embodiment of the present application. The immersive teaching scene rendering device is applied to a scene rendering engine in an immersive teaching system. The immersive teaching system can be as follows Figure 1 As shown. Figure 3 As shown, the immersive teaching scene rendering device 30 includes:

[0104] The receiving module 301 is used to receive a first signaling sent by the data processing server, the first signaling carrying first behavior data, the first behavior data being obtained by the data processing server performing behavior analysis on the teaching video collected by the audio and video collection device, and the first behavior data being behavior data corresponding to a first character in the teaching video;

[0105] a rendering module 302, configured to render a first virtual three-dimensional character corresponding to the first character according to the first behavior data in a target virtual three-dimensional scene according to the first signaling, to obtain a first three-dimensional picture, wherein the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene;

[0106] The picture sending module 303 is used to send the first three-dimensional picture to the multiple virtual reality terminals, so that the multiple virtual reality terminals display the first three-dimensional picture.

[0107] In a possible design, the above-mentioned receiving module 301 is also used to: receive the target three-dimensional space model sent by the data processing server, and the target three-dimensional space model is obtained by the data processing server through three-dimensional reconstruction based on the teaching scene in the teaching video; the above-mentioned rendering module 302 is also used to: render the target three-dimensional space model to obtain the target virtual three-dimensional scene.

[0108] In a possible design, the brightness of the light source corresponding to the target three-dimensional space model is obtained based on the screen brightness of the teaching video.

[0109] In a possible design, the immersive teaching system also includes a data processing server; the above-mentioned receiving module 301 is also used to: receive a virtual three-dimensional character model sent by the data processing server, the virtual three-dimensional character model is generated by the data processing server according to the facial image data corresponding to the teaching video, and the facial image data is obtained by the data processing server performing facial target recognition on the teaching video; the above-mentioned rendering module 302 is also used to: render the virtual three-dimensional character model in the target virtual three-dimensional scene to obtain a virtual three-dimensional character corresponding to the teaching video.

[0110] In a possible design, the immersive teaching system also includes a three-dimensional asset management platform; the above-mentioned rendering module 302 is also used to: in response to a first user instruction, obtain a first three-dimensional model corresponding to the first user instruction from the three-dimensional asset management platform; render the first three-dimensional model in the target virtual three-dimensional scene to obtain a second three-dimensional picture; and send the second three-dimensional picture to the multiple virtual reality terminals so that the multiple virtual reality terminals display the second three-dimensional picture.

[0111] It should be noted that Figure 3 For the contents not mentioned in the corresponding embodiments, please refer to the description of the aforementioned method embodiments, which will not be repeated here.

[0112] The above-mentioned device receives a first signaling sent by a data processing server, the first signaling carries first behavior data, and the first behavior data is obtained by the data processing server performing a behavior analysis on the teaching video collected by the audio and video collection device; according to the first signaling, a first virtual three-dimensional character corresponding to the first character is rendered in a target virtual three-dimensional scene according to the first behavior data to obtain a first three-dimensional picture, the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene; finally, the first three-dimensional picture is sent to multiple virtual reality terminals; since the first behavior data is the behavior data corresponding to the first character in the teaching video, the first three The three-dimensional picture is a three-dimensional picture of a first virtual three-dimensional character corresponding to the first character performing a behavior corresponding to the first behavior data in a target virtual three-dimensional scene. Sending the first three-dimensional picture to multiple virtual reality terminals can achieve synchronization of the virtual three-dimensional characters in the target virtual three-dimensional scene with the characters in the real environment, so that users wearing virtual reality terminals can determine the actions performed by the characters in the real environment by watching the virtual characters in the target virtual three-dimensional scene, which is convenient for teachers to perceive the behaviors of students in the classroom after wearing VR equipment. In addition, by transmitting signaling carrying behavior data, users wearing virtual reality terminals can perceive the behaviors of characters in the real environment. Compared with directly transmitting teaching videos, less bandwidth resources are required for transmission, which is easy to implement.

[0113] See also Figure 4 , Figure 4 4 is a schematic diagram of a computer device provided in an embodiment of the present application, wherein the computer device 40 includes a processor 401 and a memory 402. The memory 402 is connected to the processor 401, for example, via a bus.

[0114] The processor 401 is configured to support the computer device 40 to perform the corresponding functions in the method in the above method embodiment. The processor 401 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0115] The memory 402 is used to store program codes, etc. The memory 402 may include a volatile memory (VM), such as a random access memory (RAM); the memory 402 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 402 may also include a combination of the above-mentioned types of memories.

[0116] The processor 401 may call the program code to perform the following operations:

[0117] receiving a first signaling sent by the data processing server, the first signaling carrying first behavior data, the first behavior data being obtained by the data processing server performing behavior analysis on the teaching video collected by the audio and video collection device, the first behavior data being behavior data corresponding to a first character in the teaching video;

[0118] According to the first signaling, a first virtual three-dimensional character corresponding to the first character is rendered in a target virtual three-dimensional scene according to the first behavior data to obtain a first three-dimensional picture, wherein the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene;

[0119] The first three-dimensional picture is sent to the multiple virtual reality terminals so that the multiple virtual reality terminals display the first three-dimensional picture.

[0120] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method described in the above embodiment.

[0121] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0122] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A method for rendering an immersive teaching scene, characterized in that: A scene rendering engine applied to an immersive teaching system, wherein the immersive teaching system comprises an audio and video acquisition device, a data processing server, the scene rendering engine and a plurality of virtual reality terminals; the method comprises: receiving a first signaling sent by the data processing server, the first signaling carrying first behavior data, the first behavior data being obtained by the data processing server performing behavior analysis on the teaching video collected by the audio and video collection device, the first behavior data being behavior data corresponding to a first character in the teaching video; According to the first signaling, a first virtual three-dimensional character corresponding to the first character is rendered in a target virtual three-dimensional scene according to the first behavior data to obtain a first three-dimensional picture, wherein the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene; The first three-dimensional picture is sent to the multiple virtual reality terminals so that the multiple virtual reality terminals display the first three-dimensional picture.

2. The method according to claim 1, characterized in that Before rendering the first virtual three-dimensional character corresponding to the first character in the target virtual three-dimensional scene according to the first behavior data according to the first signaling, the method further includes: Receiving a target three-dimensional space model sent by the data processing server, where the target three-dimensional space model is obtained by the data processing server through three-dimensional reconstruction based on the teaching scene in the teaching video; The target three-dimensional space model is rendered to obtain the target virtual three-dimensional scene.

3. According to the method of claim 2, the brightness of the light source corresponding to the target three-dimensional space model is obtained based on the screen brightness of the teaching video.

4. The method according to claim 2, characterized in that: After rendering the target three-dimensional space model to obtain the target virtual three-dimensional scene, the method further includes: Receiving a virtual three-dimensional character model sent by the data processing server, wherein the virtual three-dimensional character model is generated by the data processing server according to the facial image data corresponding to the teaching video, and the facial image data is obtained by the data processing server performing facial target recognition on the teaching video; The virtual three-dimensional character model is rendered in the target virtual three-dimensional scene to obtain a virtual three-dimensional character corresponding to the teaching video.

5. The method according to any one of claims 1 to 4, characterized in that: The immersive teaching system also includes a three-dimensional asset management platform; The method further comprises: In response to a first user instruction, acquiring a first three-dimensional model corresponding to the first user instruction from the three-dimensional asset management platform; Rendering the first three-dimensional model in the target virtual three-dimensional scene to obtain a second three-dimensional picture; The second three-dimensional picture is sent to the multiple virtual reality terminals so that the multiple virtual reality terminals display the second three-dimensional picture.

6. An immersive teaching scene rendering device, characterized in that: A scene rendering engine applied to an immersive teaching system, the immersive teaching system comprising an audio and video acquisition device, a data processing server, the scene rendering engine and a plurality of virtual reality terminals; the device comprises: a receiving module, configured to receive a first signaling sent by the data processing server, the first signaling carrying first behavior data, the first behavior data being obtained by the data processing server performing behavior analysis on the teaching video collected by the audio and video collection device, the first behavior data being behavior data corresponding to a first character in the teaching video; a rendering module, configured to render a first virtual three-dimensional character corresponding to the first character according to the first behavior data in a target virtual three-dimensional scene according to the first signaling, to obtain a first three-dimensional picture, wherein the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene; The picture sending module is used to send the first three-dimensional picture to the multiple virtual reality terminals so that the multiple virtual reality terminals display the first three-dimensional picture.

7. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory is connected to the processor, and the processor is used to execute one or more computer programs stored in the memory. When the processor executes the one or more computer programs, the computer device implements the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 5.

9. An immersive teaching system, characterized in that: Includes audio and video acquisition equipment, data processing servers, scene rendering engines and multiple virtual reality terminals; The audio and video acquisition device is used to collect teaching videos; The data processing server is used to perform behavior analysis on the teaching video to obtain first behavior data, where the first behavior data is behavior data corresponding to a first character in the teaching video; The data processing server is further used to send a first signaling to the scene rendering engine, where the first signaling carries the first behavior data; The scene rendering engine is used to render a first virtual three-dimensional character corresponding to the first character according to the first behavior data in a target virtual three-dimensional scene according to the first signaling to obtain a first three-dimensional picture, wherein the target virtual three-dimensional scene is a virtual three-dimensional scene corresponding to the teaching scene in the teaching video, and the first three-dimensional picture is a three-dimensional picture of the first virtual three-dimensional character performing the behavior corresponding to the first behavior data in the target virtual three-dimensional scene; The scene rendering engine is further used to send the first three-dimensional picture to the multiple virtual reality terminals; The multiple virtual reality terminals are used to display the first three-dimensional image.

10. The immersive teaching system according to claim 9, characterized in that: The data processing server is also used to perform three-dimensional reconstruction based on the teaching scene in the teaching video to obtain a target three-dimensional space model; The data processing server is also used to send the target three-dimensional space model to the scene rendering engine; The scene rendering engine is also used to render the target three-dimensional space model to obtain the target virtual three-dimensional scene.

11. The immersive teaching system according to claim 10, characterized in that: The data processing server is also used to determine the brightness of the light source corresponding to the target three-dimensional space model according to the screen brightness of the teaching video.

12. The immersive teaching system according to claim 10, characterized in that: The data processing server is also used to perform face target recognition on the teaching video to obtain face image data; The data processing server is also used to generate a virtual three-dimensional character model based on the face image data; The data processing server is also used to send the virtual three-dimensional character model to the scene rendering engine; The scene rendering engine is also used to render the virtual three-dimensional character model in the target virtual three-dimensional scene to obtain the virtual three-dimensional character corresponding to the teaching video.

13. The immersive teaching system according to any one of claims 9 to 12, characterized in that: The immersive teaching system also includes a three-dimensional asset management platform; The three-dimensional asset management platform is used to store three-dimensional models; The scene rendering engine is further used to respond to a first user instruction, obtain a first three-dimensional model corresponding to the first user instruction from the three-dimensional asset management platform; render the first three-dimensional model in the target virtual three-dimensional scene to obtain a second three-dimensional picture; The scene rendering engine is further used to send the second three-dimensional picture to the multiple virtual reality terminals; The multiple virtual reality terminals are also used to display the second three-dimensional picture.

14. The immersive teaching system according to any one of claims 9 to 12, characterized in that: The immersive teaching system also includes a blockchain storage platform; the blockchain storage platform is used to store data in the immersive teaching system.

15. The immersive teaching system according to any one of claims 9 to 12, characterized in that: The immersive teaching system also includes an authentication management platform; the authentication management platform is used for performing identity authentication management.

16. The immersive teaching system according to any one of claims 9 to 12, characterized in that: The immersive teaching system also includes a data analysis and display platform; the data analysis and display platform is used to generate and display teaching scenario reports.

Images