Method and system for managing virtual classroom
Patent Information
- Application Number
- CN202311790548.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-26
- Filing Date
- 2023-12-25
- Publication Date
- 2024-11-26
AI Technical Summary
In virtual classrooms of distance education, lecturers have difficulty monitoring the responses of a large number of students in real time, especially during lectures where they are unable to focus on each student’s monitor to observe their responses, resulting in a timely understanding of students’ understanding and emotional state.
Generate 3D avatars to enhance the interactive experience by analyzing students’ audio/video data streams on the server side, calculating attention scores, and establishing a direct communication interface for sentiment analysis and interactions, if necessary.
Real-time monitoring and immediate help to students' responses and emotional states is achieved, and the teaching effect of distance education and students' learning experience are improved.
Smart Images

Figure CN119027283A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method and a system for managing a virtual classroom. Background Art
[0002] Recently, due to various reasons such as epidemics, geopolitical issues, the use of online communication has greatly increased. One of its related applications is distance education, which provides a learning experience for students who cannot attend classes in person. In order to carry out distance education, a virtual classroom is adopted. In the virtual classroom, the lecturer operates an electronic device (such as a personal computer) and speaks in front of a camera and a microphone. The camera continuously captures the image of the lecturer and / or classroom materials (such as blackboards, presentation slides, etc.), and the microphone continuously collects audio data from the lecturer. Then, the electronic device generates a data stream including image and audio data, and sends the data stream to the cloud server. One or more students can each operate another electronic device to receive a data stream from the cloud server, and the received data stream can be played on each student's electronic device. In some cases, students with a camera and / or microphone can also interact with the lecturer in a manner similar to that of a lecturer giving a lecture. Then, the lecturer's electronic device can receive one or more data streams including the image and / or audio data of the student.
[0003] For teaching purposes, lecturers need to ensure that students understand the material they are teaching. This can be a challenge for distance education, where observing students' real-time reactions may not be feasible when a large number of students are attending a lecture online at the same time. In addition, the lecturer may not be able to focus on his / her monitor during the lecture to monitor each student's reaction. For example, in situations where the lecturer is writing on the blackboard and thus has his / her back to the monitor or performing other tasks that prevent him / her from looking at the monitor, student reactions may not be observed. Summary of the invention
[0004] An object of the present invention is to provide a method for managing a virtual classroom which is capable of analyzing the reaction of at least one student during a lecture.
[0005] According to one aspect of the present invention, a method for hosting a metaverse virtual classroom (VC) is provided. The method is implemented using a system, the system comprising a server, a lecturer device associated with a lecturer and communicating with the server, and a student device associated with a student and communicating with the server. The method comprises: the lecturer device records a first audio / video (A / V) data stream associated with the lecturer and transmits the first A / V data stream to the server; the server transmits the first A / V data stream to the student device in response to receiving the first A / V data stream from the lecturer device; the student device presents the first A / V data stream to the student in response to receiving the first A / V data stream from the server; the student device records a second A / V data stream associated with the student and transmits the second A / V data stream to the server; the server performs calculation of an attention score associated with the student's attention in response to the first A / V data stream based on the second A / V data stream in response to receiving the second A / V data stream; and the server transmits the attention score to the lecturer device.
[0006] In some embodiments, the method further includes: in response to receiving an interactive command issued based on the attention score from the instructor device, the server establishing a direct communication interface between the instructor device and the student device.
[0007] In some embodiments, the method further includes: while holding the direct communication interface, the server performing an emotional analysis on the audio data included in the second A / V data stream to determine an emotional state of the student.
[0008] In some embodiments, the method further includes: the server generating, on the direct communication interface, a first 3D avatar based on the lecturer and a second 3D avatar based on the student.
[0009] In certain embodiments, the second A / V data stream includes a plurality of images of the student, and the attention score is calculated based on images included in the images using bimodal emotion classification.
[0010] In some embodiments, the step of calculating the attention score is performed periodically.
[0011] In certain embodiments, the system includes a plurality of student devices, each associated with a student and communicating with the server; the server transmits the first A / V data stream to each student device; each student device records a unique second A / V data stream associated with the corresponding student and transmits the unique second A / V data stream to the server; the server respectively calculates a plurality of attention scores related to the students' attention; and the method further includes the server sending an alert to the instructor device when it is determined that more than a certain percentage of the attention scores are below a threshold.
[0012] In some embodiments, the step of calculating a plurality of attention scores related to the student's attention is performed periodically; and the method further comprises the server sending an alert to the instructor device when it is determined that more than a certain percentage of the attention scores are below a threshold over a period of time.
[0013] Another object of the present invention is to provide a system which is configured to implement the above method.
[0014] According to one aspect of the present invention, a system for managing a virtual classroom is provided. The system includes a server, a lecturer device associated with a lecturer and communicating with the server, and a student device associated with a student and communicating with the server. The system is configured to perform the following operations of the above method: the lecturer device records a first audio / video (A / V) data stream associated with the lecturer and transmits the first A / V data stream to the server; the server transmits the first A / V data stream to the student device in response to receiving the first A / V data stream from the lecturer device; the student device presents the first A / V data stream to the student in response to receiving the first A / V data stream from the server; the student device records a second A / V data stream associated with the student and transmits the second A / V data stream to the server; the server performs calculation of an attention score associated with the student's attention in response to the first A / V data stream based on the second A / V data stream in response to receiving the second A / V data stream; and the server transmits the attention score to the lecturer device.
[0015] In some embodiments, the system is further configured to perform the following operations: the server establishes a direct communication interface between the instructor device and the student device in response to receiving an interactive command issued based on the attention score from the instructor device.
[0016] In some embodiments, the system is further configured to perform the following operations: when holding the direct communication interface, the server performs emotional analysis on the audio data included in the second A / V data stream to determine the emotional state of the student.
[0017] In some embodiments, the system is further configured to perform the following operations: the server generates a first 3D avatar based on the lecturer and a second 3D avatar based on the student on the direct communication interface.
[0018] In certain embodiments, the second A / V data stream includes a plurality of images of the student, and the attention score is calculated based on images included in the images using bimodal emotion classification.
[0019] In some embodiments, the server is configured to periodically perform the calculation of the attention score.
[0020] In some embodiments, the system includes multiple student devices, each associated with a student and communicating with the server; the server transmits the first A / V data stream to each student device; each student device records a unique second A / V data stream associated with the corresponding student and transmits the unique second A / V data stream to the server; the server respectively calculates multiple attention scores related to the students' attention; and the server is also configured to perform the following operations: sending an alarm to the lecturer device when it is determined that more than a certain percentage of the attention scores are below a threshold.
[0021] The beneficial effect of the present invention is that during the VC lecture, the server can capture and process the image of the student and calculate the attention score based on the image to determine whether each student is listening to the lecture. The determination can be made regularly to further analyze the student's reaction to the sub-part of the lecture and the reaction of the individual student during the lecture. In addition, the student's reaction to the individual lecture in a semester can also be analyzed. In the case where a specific student needs help based on the student's attention score, the lecturer can choose to open a direct communication interface between the lecturer device and the student device. When generating the direct communication interface, the server can perform emotional analysis on the audio data included in the second A / V data stream to determine the student's emotional state. When it is determined that the student's emotional state is negative, the lecturer can also use virtual face-to-face communication, in which the server generates a 3D avatar to be displayed by the student device. In this way, the method and system can at least improve the experience of VC in distance education, the lecturer can learn more about the student's reaction, and the student can get instant help from the lecturer. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] Other features and effects of the present invention will be clearly presented in the embodiments with reference to the accompanying drawings, in which:
[0023] Figure 1is a block diagram illustrating a system for managing a virtual classroom according to an embodiment of the present invention;
[0024] Figure 2 is a flow chart illustrating a method for managing VCs according to an embodiment of the present invention;
[0025] Figure 3 is a schematic diagram illustrating an example structure of a dual-mode emotion classification model according to an embodiment of the present invention;
[0026] Figure 4 (including images 4A to 4C) depict some typical human reactions and their associated psychological states;
[0027] Figure 5 is a graph that plots the attention scores of some students during a semester's lectures; and
[0028] Figure 6 is a flow chart illustrating the steps of an interactive process in which a lecturer interacts with a student after determining that the student's attention score is below a threshold value according to an embodiment of the present invention. DETAILED DESCRIPTION
[0029] Before the present invention is described in more detail, it should be noted that, where considered appropriate, reference numerals have been repeated among the drawings to indicate corresponding or analogous elements, which may optionally have similar characteristics.
[0030] Figure 1 1 is a block diagram of a system 100 according to an embodiment of the present invention. In this embodiment, the system 100 includes a server 110, a lecturer device 120, and a plurality of student devices ( Figure 1 The lecturer device 120 is operated by a lecturer, and the student devices 140a and 140b are each operated by a student. Each of the lecturer device 120 and the student devices 140a and 140b is configured to communicate with the server 110 via a network such as the Internet. In different embodiments, the system 100 may also include only one student device for use by one student. In use, the lecturer and the student can interact using the system 100, which provides an online teaching and learning environment, such as a virtual classroom (VC).
[0031] The server 110 may be implemented using a cloud server or other computing device, and includes a processor 112 , a memory module 114 , and a communication unit 116 .
[0032] The processor 112 may include but is not limited to a single-core processor, a multi-core processor, a dual-core mobile processor, a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable processor gate array (FPGA), an application-specific integrated circuit (ASIC) and / or a radio frequency integrated circuit (RFIC), etc.
[0033] The memory module 114 is connected to the processor 112 and can be implemented using, for example, a random access memory (RAM), a read-only memory (ROM), a programmable ROM (PROM), firmware and / or flash memory. The memory module 114 stores software applications, which include instructions that, when executed by the processor 112, enable the processor 112 to perform the following operations. Specifically, the software application may be a VC software application.
[0034] The communication unit 116 is connected to the processor 112 and may include one or more of a radio frequency integrated circuit (RFIC), a short-range wireless communication module supporting a short-range wireless communication network using wireless technologies such as Bluetooth and / or Wi-Fi, and a mobile communication module supporting telecommunications using Long Term Evolution (LTE), third generation (3G), fourth generation (4G) or fifth generation (5G) wireless mobile telecommunication technologies, etc. The server 110 is configured to communicate with the lecturer device 120 and the student devices 140a, 140b using the communication unit 116.
[0035] The instructor device 120 may be implemented using a personal computer, a laptop, a tablet computer, a smart phone or other electronic devices. The instructor device 120 includes a processor 122 , a display 124 , a memory module 126 , a communication unit 128 and an input detection unit 130 .
[0036] The processor 122 may include components similar to those of the processor 112 (e.g., a microprocessor) for performing operations as described below. The display 124 may be implemented using a liquid crystal display (LCD) or a light emitting diode (LED) display, and is connected to the processor 122. The memory module 126 is connected to the processor 122, includes components similar to those of the memory module 114, and stores software applications that include instructions that, when executed by the processor 122, cause the processor 122 to perform operations as described below. The communication unit 128 is connected to the processor 122, includes components similar to those of the communication unit 116, and enables communication between the lecturer device 120 and the server 110.
[0037] The input detection unit 130 is connected to the communication unit 128 (but not limited thereto, it may also be connected to the processor 122 in other embodiments) for detecting the movement and other inputs of the lecturer operating the lecturer device 120. Based on different models of computers embodying the lecturer device 120, the input detection unit 130 may include an audio collection component (hereinafter, a microphone is used as an example) and a video collection component (hereinafter, a camera is used as an example). The microphone is used to collect audio data from the lecturer, and the camera is placed facing the lecturer and is configured to continuously capture images.
[0038] Each student device 140a / 140b can be implemented using a personal computer, a laptop, a tablet computer, a smart phone, or other electronic devices. Each student device 140a / 140b includes a processor 142, a display 144, a memory module 146, a communication unit 148, and an input detection unit 150. The components included in each student device 140a / 140b will be further described below.
[0039] The processor 142 may include components similar to those of the processor 122 (e.g., a microprocessor) for performing operations as described below. The display 144 may be implemented using an LCD display or an LED display and is connected to the processor 142. The memory module 146 is connected to the processor 142 and includes components similar to those of the memory module 126 and stores software applications including instructions that, when executed by the processor 142, cause the processor 142 to perform operations as described below. The communication unit 148 is connected to the processor 142 and includes components similar to those of the communication unit 128 and enables communication between the student device 140a / 140b and the server 110. The input detection unit 150, for example, connected to the processor 142 (but not limited thereto, in other embodiments, it may also be connected to the communication unit 148) includes an audio collection component (hereinafter, a microphone is used as an example) and a video collection component (hereinafter, a camera is used as an example). The microphone is used to collect audio data from the student. The camera is placed facing the student and is configured to continuously capture images.
[0040] It is worth noting that, for the sake of simplicity, in the embodiment where there are multiple student devices 140a, 140b, it is assumed that each student device 140a / 140b is identical to each other, as described above, but not limited thereto.
[0041] It is worth noting that in some embodiments, one or both of the lecturer device 120 and the student devices 140a, 140b can be implemented using a television device. The television device includes a processor, a television screen, an input detection unit, a memory module, and a communication unit. In some embodiments, the processor, the memory module, and the communication unit can be integrated in a set-top box (STB) connected to the television screen. The processor may include components similar to those of the processor 112 for performing operations as described below. The television screen is connected to the processor. The memory module includes components similar to those of the memory module 114, and stores software applications, which include instructions that cause the processor to perform operations as described below when executed by the processor. The communication unit includes components similar to the communication unit 116, and enables communication between the television device and the server 110. The input detection unit may include an image capture unit and an audio collection component (hereinafter, a microphone is used as an example). The image capture unit is implemented using a digital camera or a video recorder such as a webcam, a depth camera, etc., is set on a television screen to face the viewer of the television device, and is configured to continuously capture images. The microphone may be integrated with the STB or may be a separate component, and is used to collect audio data from the lecturer or students.
[0042] When in use, the lecturer and the student can use the server 110 as the host of the VC to interact in the online classroom where the lecturer is to give a lecture. In this way, the lecturer can first operate the interface of the lecturer device 120 (not shown) to communicate with the server 110 to transmit a signal to the server 110 to start the VC.
[0043] Figure 2 is a flow chart illustrating a method for managing VC according to an embodiment of the present invention. In this embodiment, the method comprises: Figure 1 It should be noted that, for simplicity of description, it is assumed that only one student is present and uses one student device, such as the student device 140a, but in other embodiments, other students may also be naturally present.
[0044] In step 202, the processor 112 of the server 110 executes the VC software application in response to receiving a signal from the lecturer device 120, and generates a VC session. In various embodiments, the VC session may enable one or more students to join by operating the student devices 140a, 140b. Specifically, the student may operate an interface (not shown) of the student devices 140a, 140b to communicate with the server 110. The VC session may be provided to the student in the form of a link that can be accessed using a web browser or software application. In some embodiments, a set of room numbers and passwords may be required in order to join the VC session.
[0045] When the lecturer starts giving a lecture, in step 204 , the lecturer device 120 starts recording a first audio / video (A / V) data stream associated with the lecturer, and transmits the first A / V data stream to the server 110 .
[0046] Specifically, after the lecturer operates the interface of the lecturer device 120, the input detection unit 130 is activated to transmit the signal to the server 110; in this way, the microphone of the input detection unit 130 starts to collect the sound data around the lecturer, and the camera of the input detection unit 130 starts to capture the image around the lecturer. The lecturer can also operate the interface to share and edit images (such as handouts, presentation slides, etc.) for students. The processor 122 is configured to receive the data collected by the input detection unit 130 as described above, and generate the first A / V data stream including the image and audio data obtained by the input detection unit 130. The processor 122 is configured to control the communication unit 128 to continuously transmit the first A / V data stream to the server 110.
[0047] In step 206, the server 110 is configured to transmit the first A / V data stream to the student device 140a that has joined the VC session in response to receiving the first A / V data stream from the lecturer device 120. Please note that if there are multiple student devices 140a, 140b in the VC session, the server 110 is configured to transmit the first A / V data stream to each student device 140a / 140b.
[0048] In step 206, the student device 140a presents the first A / V data stream to the student in response to receiving the first A / V data stream from the server 110. Specifically, the display 144 of the student device 140a displays the image included in the first A / V data stream, and the speaker (not shown) of the student device 140a plays the audio data included in the first A / V data stream.
[0049] In step 208 , the student device 140 a records a second A / V data stream associated with the student and transmits the second A / V data stream to the server 110 .
[0050] Specifically, during the lecture, the input detection unit 150 of the student device 140a is also activated. That is, the microphone of the input detection unit 150 starts to collect audio data from around the student, and the camera of the input detection unit 150 starts to capture images around the student. For the student device 140a, the processor 142 is configured to receive the data collected by the input detection unit 150 as described above, generate a second A / V data stream including the image and audio data obtained by the input detection unit 150, and control the communication unit 148 to continuously transmit the second A / V data stream to the server 110.
[0051] Note that in embodiments where there are multiple student devices 140a, 140b, each student device 140a / 140b is configured to generate and transmit a unique second A / V data stream to the server 110.
[0052] In step 210, the server 110 performs calculation of an attention score associated with the student's attention in response to the first A / V data stream based on the second A / V data stream in response to the second A / V data stream received from the student device 140a. Please note that in an embodiment where there are multiple students, the server 110 calculates multiple attention scores associated with the students' attention, respectively.
[0053] Specifically, the processor 112 of the server 110 is configured to execute a bimodal emotion classification model, which may be in the form of a neural network. Note that different neural networks may be used to process the images included in the second A / V data stream to determine the attention score.
[0054] Figure 3 1 is a schematic diagram illustrating an exemplary structure of a dual-mode emotion classification model 300 stored in the memory module 114 of the server 110 according to an embodiment of the present invention. It should be noted that different structures can be used to construct the dual-mode emotion classification model, and are not limited to Figure 3 The structure shown.
[0055] When in use, the image included in the second A / V data stream can be used as an input to the dual-mode emotion classification model 300. The dual-mode emotion classification model 300 includes two convolutional neural network (CNN) groups 310A, 310B, which are configured to output a plurality of facial feature vectors and a plurality of body feature vectors, respectively. Each of the CNN groups 310A, 310B includes a CNN model, a rectified linear unit (ReLU), and a plurality of pooling layers connected in series.
[0056] Figure 4 (which includes images 4A to 4C) depicts a number of exemplary human reactions and their associated psychological states. Specifically, image 4A depicts a number of exemplary upper body postures, image 4B depicts a number of exemplary eye cues and their associated mental states (e.g., focused, determined, dreamy, suspicious, etc.), and image 4C shows a number of exemplary facial cues and associated psychological states (e.g., afraid, contemptuous, angry, sad, etc.). Figure 4 The images can be used to pre-train the dual-mode emotion classification model 300.
[0057] exist Figure 3 In some embodiments, the CNN group 310A is configured to output a facial feature vector based on the images of the second A / V data stream, and the CNN group 310B is configured to output a body feature vector based on the images of the second A / V data stream. In some embodiments, the dual-mode emotion classification model 300 may include an additional CNN group configured to output multiple eye feature vectors.
[0058] Then, the facial feature vector and the body feature vector are transmitted to the long short-term memory (LSTM) 320 included in the bimodal emotion classification model 300 as input. In addition, the facial feature vector is transmitted to the facial emotion determination layer 330A to obtain a preliminary facial emotion dataset, and the body feature vector is transmitted to the body emotion determination layer 330B to obtain a preliminary body emotion dataset. Then, the output of the LSTM 320, the preliminary facial emotion dataset, and the preliminary body emotion dataset are processed together using at least one fusion method and sent to the emotion classification unit 340. Please note that the fusion method is an image fusion technique in order to integrate more than one image to produce a merged image, and in an embodiment, the fusion method implemented can be feature level fusion, decision level fusion, product fusion method (PFM), average fusion method (AFM), multi-mode compact bilinear pooling (MCB) fusion, etc. The emotion classification unit 340 is configured to determine the predicted emotion based on the above data processed using the fusion method.
[0059] The predicted emotion is then transmitted to a score calculation unit (not shown), which is configured to calculate an attention score associated with the student's attention in response to the first A / V data stream based on the predicted emotion.
[0060] In an embodiment, the attention score may be represented as a normalized value between 0 and 1. Specifically, when the predicted emotion is considered to be more relevant to a positive mental state (e.g., happy, calm, neutral (i.e., no particular emotion is detected), etc.), the resulting attention score is closer to 1. Otherwise, when the predicted emotion is considered to be more relevant to a negative mental state (e.g., angry, disgusted, fearful, sad, surprised, etc.), the resulting attention score is closer to 0. In this embodiment, the attention score is represented as a binary number [0,1]. Specifically, the determination of a positive mental state results in an attention score of 1, and the determination of a negative mental state results in an attention score of 0. In some embodiments, the score calculation unit is stored in the memory module 114 of the server 110 and is implemented by a software function that executes the aforementioned determination scheme.
[0061] Note that in other embodiments, various calculations and representations of attention scores may be employed.
[0062] After the score calculation unit calculates the attention score, in step 212, the processor 112 transmits the attention score to the lecturer device 120. In this way, the lecturer can be informed of the student's attention score.
[0063] It should be noted that in some embodiments, the calculation of the attention score included in step 210 can be performed periodically during the lecture. Specifically, in one example, in a lecture of five students, the attention score can be calculated every five minutes. The lecture is 60 minutes long, which is divided into four 15-minute sub-parts. Table 1 below includes the attention scores recorded during the lecture.
[0064] Table 1
[0065]
[0066]
[0067] Using the above statistics, it is possible to further analyze each student’s attention. For example, Table 2 below includes the average attention scores calculated for each student for the entire lecture and each subsection.
[0068] Table 2
[0069]
[0070] In this example, the threshold is set in increments of 0.5, and when the average attention score during a sub-section of the lecture or the entire lecture is below the threshold (as shown in bold in Table 2), it can be determined that the student may not be paying enough attention during a sub-section of the lecture (e.g., student 1 in sub-sections 1, 3, 4) or the entire lecture. In addition, the overall acceptance of each sub-section of the lecture and the entire lecture can also be determined by calculating the average of all students. According to Table 2, sub-section 2 may be more popular with students (and therefore may be easier to understand).
[0071] In another example, the attention scores calculated during another lecture are shown in Table 3 below.
[0072] Table 3
[0073]
[0074]
[0075] Table 4 below shows the average attention score for each student calculated for the entire lecture and for each subsection.
[0076] Table 4
[0077]
[0078] It can be inferred that most students were not paying attention. Such reactions may be attributed to the difficulty of the material, potential Internet connection issues, etc.; therefore, the lecturer may need to introduce the material again or ask students to provide feedback on this particular lecture. In some cases, when it is determined that more than a certain percentage of attention scores are below a threshold, the processor 112 of the server 110 is configured to send an alert to the lecturer device 120. In the example of Table 4, when subsection 1 is completed and it is determined that more than half of the students (a certain percentage) have an attention score below 0.5 (threshold), the processor 112 of the server 110 sends an alert to the lecturer device 120 to notify the lecturer that most students did not respond well to this part of the lecture. Therefore, the lecturer can adjust the rest of the lecture in real time by slowing down the lecture, repeating previously taught material, etc.
[0079] In another example, the attention score of each student is calculated for each lecture during a semester, and is shown in Table 5. In this example, a attention score can be determined every five minutes in a 60-minute long lecture, and the resulting 12 attention scores are used to calculate the average attention score for each lecture.
[0080] Table 5
[0081]
[0082] Using the calculated attention scores, the instructor can be informed of each student's progress in the course throughout the semester, which can be determined by e.g. Figure 5 The graph shown in Table 5 indicates that. In the example of Table 5, Student 5 may need more attention because his / her attention span has been low throughout the semester. Figure 5 The graph shown shows that Student 1 and Student 2 adjusted well as the semester progressed (their concentration showed an upward trend), while Student 3 may not have caught up (his / her concentration showed a downward trend).
[0083] The actual performance of each student can be further incorporated into the above findings. For example, when Student 3 and Student 5 perform poorly, one may conclude that Student 3 and Student 5 are struggling to learn the material taught and may need more help. Otherwise, when Student 3 or 5 actually performs well, one may infer that the material taught may be too easy for Student 3 or 5.
[0084] Please note that in different embodiments, after calculating the attention score, further processing of the attention score (eg, calculating an average) may be performed by the server 110 or the instructor device 120 .
[0085] According to some embodiments, during a lecture, whenever a new attention score or average attention score is calculated, the resulting attention score may be immediately presented to the lecturer. For example, in response to receiving the attention score from the server 110, when it is determined that a particular student has a low attention score for a period of time (e.g., 15 minutes, such as student 4 in the example of Table 1), the processor 122 of the lecturer device 120 may control the display 124 to display an alert element thereon (e.g., a pop-up notification on the display 124), indicating that student 4 is not paying attention.
[0086] In response, the lecturer may interact with the alert element (e.g., by clicking the alert element using a mouse, or by touching the alert element if the display 124 is a touch screen) to send an interactive command to the server 110. The processor 112 of the server 110 is configured to establish a direct communication interface between the lecturer device 120 and the student device 140a in response to receiving an interactive command issued based on the attention score from the lecturer device 120. In an embodiment, the direct communication interface may enable the lecturer and the student to text and / or speak to each other. In some embodiments, the direct communication interface may be implemented using an external communication application such as Web Real-Time Communication (WebRTC), but is not limited thereto.
[0087] In one example, lecturers can send text messages to students or speak directly to them during lectures or breaks. This can make it easier for lecturers to interact with students who are found to be inattentive.
[0088] Figure 6 1 is a flow chart illustrating the steps of an interactive process of the lecturer interacting with a student (e.g., a student using the student device 140a) after determining that the student's attention score (or average attention score) is below a threshold according to an embodiment of the present invention. In this embodiment, the interactive process consists of Figure 1 system implementation, and in Figure 2 The method is implemented after step 212. It should be noted that the interactive process can be during the lecture, during the break of the lecture, or after the lecture.
[0089] The lecturer decides whether to interact with the student who is considered to need help using text messages or voice conversations via a direct communication interface in response to the attention score presented on the display 124 of the lecturer device 120. If the lecturer determines to interact with the student using text messages or voice conversations, the lecturer device 120 may transmit the interactive command generated by the manual operation of the lecturer to the server 110. Thus, in step 214, the server 110 determines whether an interactive command from the lecturer device 120 is received. If the determination result is positive, the process proceeds to step 216. On the contrary, if the server 110 does not receive an interactive command from the lecturer device 120 because the lecturer does not want to interact with the student using text messages or voice conversations, the process proceeds to step 222.
[0090] In step 216, the server 110 establishes a direct communication interface between the instructor device 120 and the student device 140a, so that the instructor can send text messages to the student or directly chat with the student.
[0091] It should be noted that, when the lecturer is communicating with the student, the lecturer device 120 may continue to acquire image and audio data, record a first A / V data stream, and transmit the first A / V data stream to the server 110. The student device 140a may continue to acquire image and audio data, record a second A / V data stream, and transmit the second A / V data stream to the server 110.
[0092] Next, in step 218, while holding the direct communication interface, the processor 112 of the server 110 is configured to perform emotional analysis on the audio data included in the second A / V data stream to determine the emotional state of the student. Specifically, the processor 112 can execute a speech analysis program to determine the emotional state of the student. In some embodiments, the speech analysis program can be a speech emotion recognition program, which includes a neural network model that is pre-trained to detect certain emotions in speech. When used, a feature extraction function is used to extract features included in the audio component of the second A / V data stream to obtain a plurality of Mel-frequency cepstral coefficients (hereinafter referred to as MFCC), and then the MFCC can be processed by the speech emotion recognition program to determine the emotional state of the student.
[0093] In some embodiments, the speech emotion recognition program can be implemented using the open source project "Speech Emotion Analyzer", which is in the form of a CNN model that uses open source data sets such as Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS), Surrey Audio-Visual Expressed Emotion (SAVEE) databases, etc. for training. The feature extraction function can be implemented using the open source librosa library of MFCC. Alternatively, the speech emotion recognition program can be implemented using the open source project "Audio Emotion Classification from Multiple Datasets", which is in the form of a CNN model that uses open source data sets such as RAVDESS, Toronto emotional speech set (TESS), etc. for training. The feature extraction function can be implemented using the open source librosa library for MFCC and multi-channel speech separation (MCSS). Alternatively, the speech emotion recognition program can be implemented using the open source project "Emovoice", which adopts the form of a classification model based on a support vector machine classifier or a linear support vector machine classifier.
[0094] In this embodiment, the determined emotional state of the student can also be represented by an emotional score using a binary number [0, 1]. Specifically, when the determined emotional state of the student is negative (e.g., afraid, sad, angry, etc.), the obtained emotional score can be 0, and when the determined emotional state of the student is positive (e.g., happy, calm, etc.), the obtained emotional score can be 1. It should be noted that in other embodiments, the emotional score can be calculated in a different manner from the above.
[0095] Then, in step 220 , the server 110 transmits the emotion score to the lecturer device 120 to inform the lecturer.
[0096] On the other hand, the lecturer may also decide whether the student needs additional help. A situation that may occur is that the lecturer determines that a voice conversation with the student will not produce the desired results (due to the student's negative emotional state), or the lecturer determines that the student's attention score is unsatisfactory, so the student needs additional help. If the lecturer determines that the student needs additional help, the lecturer device 120 can transmit a signal about additional help generated by the lecturer's manual operation to the server 110. Thus, in step 222, the server 110 determines whether a signal about additional help is received from the lecturer device 120. If the determination result is positive, the process enters step 224. On the contrary, if the lecturer determines that the student needs additional help at the moment and the server 110 does not receive a signal from the lecturer device 120, in this case, the interactive process ends. However, the lecturer can also interact with the student later.
[0097] In step 224, the processor 112 generates a first three-dimensional (3D) avatar based on the lecturer and a second 3D avatar based on the student on the direct communication interface. Then, the first 3D avatar is transmitted to the student device 140a, and the second 3D avatar is transmitted to the lecturer device 120. The generation of the first 3D avatar can be completed by the processor 112 executing a 3D avatar generation program based on the image included in the first A / V data stream, and the generation of the second 3D avatar can be completed by the processor 112 executing a 3D avatar generation program based on the image included in the second A / V data stream.
[0098] In some embodiments, the 3D avatar generation program can be implemented using the open source project "Video avatars", which is a component for performing 3D avatar construction based on video images. Alternatively, the 3D avatar generation program can be implemented using the open source project "PIXIE: Collaborative Regression of Expressive Bodies".
[0099] In the above ways, the interaction between lecturers and students can be more authentic.
[0100] In summary, the embodiment of the present invention provides a method and system for managing VC. During the course of the VC giving a lecture, the server may capture and process the images of the students and calculate the attention score based on the images to determine whether each student is listening to the lecture. The determination may be made periodically to further analyze the student's response to the sub-parts of the lecture and the response of individual students during the lecture. In addition, the student's response to individual lectures during a semester may also be analyzed. In the case where a particular student needs help based on the student's attention score, the lecturer may choose to open a direct communication interface between the lecturer device and the student device. When generating the direct communication interface, the server may perform emotional analysis on the audio data included in the second A / V data stream to determine the student's emotional state. When it is determined that the student's emotional state is negative, the lecturer may also use virtual face-to-face communication, in which the server generates a 3D avatar to be displayed by the student device. In this way, the method and system may at least improve the experience of VC in distance education, because the lecturer may learn more about the student's response and the student may get immediate help from the lecturer.
[0101] The above are merely embodiments of the present invention and should not be used to limit the scope of the present invention. That is, any simple equivalent changes and modifications made according to the claims and description of the present invention still fall within the scope of the present invention.
Claims
1. A method for managing a virtual classroom, the method being implemented using a system comprising a server, a lecturer device associated with a lecturer and communicating with the server, and a student device associated with a student and communicating with the server, It is characterized in that include: The lecturer device records a first audio / video A / V data stream associated with the lecturer, and transmits the first A / V data stream to the server; The server transmits the first A / V data stream to the student device in response to receiving the first A / V data stream from the lecturer device; the student device presenting the first A / V data stream to the student in response to receiving the first A / V data stream from the server; The student device records a second A / V data stream associated with the student and transmits the second A / V data stream to the server; The server, in response to receiving the second A / V data stream, performs calculation of an attention score associated with the student's attention in response to the first A / V data stream based on the second A / V data stream; and The server transmits the attention score to the instructor device.
2. The method for managing a virtual classroom according to claim 1, It is characterized in that The method further comprises: The server establishes a direct communication interface between the lecturer device and the student device in response to receiving an interactive command issued based on the attention score from the lecturer device.
3. The method for managing a virtual classroom according to claim 2, It is characterized in that The method further comprises: While holding the direct communication interface, the server performs an emotional analysis on the audio data included in the second A / V data stream to determine the emotional state of the student.
4. The method for managing a virtual classroom according to claim 2, It is characterized in that The method further comprises: The server generates a first 3D avatar based on the lecturer and a second 3D avatar based on the student on the direct communication interface.
5. The method for managing a virtual classroom according to claim 1, It is characterized in that The second A / V data stream includes a plurality of images of the student, and the attention score is calculated based on images included in the images using bimodal emotion classification.
6. The method for managing a virtual classroom according to claim 1, It is characterized in that The step of calculating the attention score is performed periodically.
7. The method for managing a virtual classroom according to claim 1, Features: The system includes a plurality of student devices, each associated with a student and in communication with the server; The server transmits the first A / V data stream to each student device; Each student device records a unique second A / V data stream associated with the corresponding student and transmits the unique second A / V data stream to the server; The server calculates a plurality of attention scores related to the attention of the students respectively; and The method also includes the server sending an alert to the instructor device upon determining that more than a certain percentage of the attention score is below a threshold.
8. The method for managing a virtual classroom according to claim 7, Features: The step of calculating a plurality of attention scores related to the student's attention is performed periodically; and The method also includes the server sending an alert to the instructor device upon determining that more than a certain percentage of the attention scores over a period of time are below a threshold.
9. A system for managing a virtual classroom, the system comprising a server, a lecturer device associated with a lecturer and communicating with the server, and a student device associated with a student and communicating with the server, It is characterized in that The system is configured to perform the following operations: The lecturer device records a first audio / video A / V data stream associated with the lecturer, and transmits the first A / V data stream to the server; The server transmits the first A / V data stream to the student device in response to receiving the first A / V data stream from the lecturer device; the student device presenting the first A / V data stream to the student in response to receiving the first A / V data stream from the server; The student device records a second A / V data stream associated with the student and transmits the second A / V data stream to the server; The server, in response to receiving the second A / V data stream, performs calculation of an attention score associated with the student's attention in response to the first A / V data stream based on the second A / V data stream; and The server transmits the attention score to the instructor device.
10. The system for managing a virtual classroom according to claim 9, It is characterized in that The system is also configured to: The server establishes a direct communication interface between the lecturer device and the student device in response to receiving an interactive command issued based on the attention score from the lecturer device.
11. The system for managing a virtual classroom according to claim 10, It is characterized in that The system is also configured to: While holding the direct communication interface, the server performs an emotional analysis on the audio data included in the second A / V data stream to determine the emotional state of the student.
12. The system for managing a virtual classroom according to claim 10, It is characterized in that The system is also configured to: The server generates a first 3D avatar based on the lecturer and a second 3D avatar based on the student on the direct communication interface.
13. The system for managing a virtual classroom according to claim 9, It is characterized in that The second A / V data stream includes a plurality of images of the student, and the attention score is calculated based on images included in the images using bimodal emotion classification.
14. The system for managing a virtual classroom according to claim 9, It is characterized in that The server is configured to periodically perform the calculation of the attention score.
15. The system for managing a virtual classroom according to claim 9, Features: The system includes a plurality of student devices, each associated with a student and in communication with the server; The server transmits the first A / V data stream to each student device; Each student device records a unique second A / V data stream associated with the corresponding student and transmits the unique second A / V data stream to the server; The server calculates a plurality of attention scores related to the attention of the students respectively; and The server is further configured to perform the following operations: sending an alert to the instructor device upon determining that more than a certain percentage of the attention score is below a threshold.
