Information processing device and information processing program

Abstract

To provide an information processing apparatus and an information processing program for facilitating an online conference according to the status of participants.SOLUTION: Processing comprises: receiving emotional information of a participant whose emotional information is in a predetermined state from an online conference server, extracting a room ID of the online conference to identify the online conference in which the participant in the predetermined state is participating, and determining whether an emotional information notification function is valid; determining color according to the emotional information if it is determined that the emotional information notification function is valid, identifying the participant participating in the online conference according to the room ID, commanding display of a received participant icon and the determined color on a display unit of a PC of a speaker and commanding display of the received participant icon and the determined color on the display units of PCs of participants other than the participant who received the information. Further, if it is determined that the emotional information notification function is invalid, the process is terminated.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an information processing device and an information processing program. [Background technology]

[0002] Patent document 1 discloses a presentation system that includes a means for inputting the presenter's voice when giving a presentation, a means for analyzing the volume and intonation of the input voice, and a means for highlighting the screen during the presentation or uttering an attention-grabbing voice if the analysis reveals an emphasized part of the voice that is loud or has a strong intonation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-23716 Summary of the Invention [Problem to be solved by the invention]

[0004] In online meetings, it is more difficult to grasp the status of participants than in face-to-face meetings, and it is not possible to proceed with the meeting according to the status of the participants.

[0005] The present invention aims to provide an information processing device and a program that can progress an online conference according to the status of participants. [Means for solving the problem]

[0006] The information processing device according to the first aspect includes a processor, and when emotion information relating to the emotion of a participant participating in an online conference reaches a predetermined state, the processor notifies other participants of information indicating the participant who has reached the predetermined state.

[0007] In the information processing device according to the second aspect, the other participants are predetermined participants among participants who take part in the online conference.

[0008] In the information processing device of the third aspect, the processor notifies the other participants of the participant who has reached the predetermined state by displaying an image indicating the participant who has reached the predetermined state on the display unit of the participant terminal device used by the other participants in the online conference.

[0009] In the information processing device according to a fourth aspect, the image showing the participant in the predetermined state includes a facial photograph of the participant in the predetermined state.

[0010] In the information processing device according to a fifth aspect, the processor changes the color of the image displayed on the display unit of the participant terminal device of the other participant according to the type of emotion information.

[0011] In an information processing device according to a sixth aspect, the emotion information is calculated from a captured image of the participant acquired by the participant terminal device.

[0012] In an information processing device according to a seventh aspect, the emotion information is calculated from facial expressions of the participants in the captured images acquired by the participant terminal devices.

[0013] In an information processing device according to an eighth aspect, the emotion information is calculated from the movements of the participants in the captured images acquired by the participant terminal devices.

[0014] In an information processing device according to a ninth aspect, the emotion information is calculated from the voice of the participant acquired by the participant terminal device.

[0015] In an information processing device according to a tenth aspect, the emotion information is calculated from biometric information of the participant acquired by the participant terminal device.

[0016] In an information processing device according to an eleventh aspect, the biological information includes respiration, body temperature, blood pressure, or pulse of the participant.

[0017] An information processing program according to a twelfth aspect causes a computer to execute a process of, when emotion information relating to the emotion of a participant participating in an online conference reaches a predetermined state, notifying other participants of information indicating the participant who has reached the predetermined state. [Effects of the Invention]

[0018] According to the first aspect, there is an effect that an online conference can be progressed according to the status of the participants.

[0019] According to the second aspect, it is possible to notify only necessary participants of participants who have entered a predetermined state.

[0020] According to the third aspect, it is possible to more easily identify a participant who has reached a predetermined state, compared to when an image showing the participant is not displayed on the display unit.

[0021] According to the fourth aspect, there is an effect that participants can be identified by their facial photographs.

[0022] According to the fifth aspect, there is an effect that emotion information can be known from the color.

[0023] According to the sixth aspect, there is an effect that emotion information can be acquired from an image.

[0024] According to the seventh aspect, there is an effect that emotional information can be acquired from the facial expressions of the participants.

[0025] According to the eighth aspect, there is an effect that emotion information can be acquired from the actions of the participants.

[0026] According to the ninth aspect, there is an effect that emotion information can be acquired from the voices of the participants.

[0027] According to the tenth aspect, there is an effect that emotion information can be acquired from the biometric information of the participants.

[0028] According to the eleventh aspect, there is an effect that emotion information can be acquired from the respiration, body temperature, blood pressure, or pulse of the participant.

[0029] According to the twelfth aspect, it is possible to provide an information processing program that can progress an online conference in accordance with the status of the participants. [Brief explanation of the drawings]

[0030] [Figure 1] 1 is a schematic configuration diagram of an information processing system according to an embodiment of the present invention. [Figure 2] 1 is a schematic block diagram of a PC according to an embodiment of the present invention. [Figure 3] 2 is a schematic block diagram of a server and an online conference server according to an embodiment of the present invention; [Figure 4] 10 is a flowchart illustrating an example of a process flow for notifying other participants of information indicating a participant who has entered a predetermined state according to an embodiment of the present invention. [Figure 5] 10 is an explanatory diagram illustrating an example of a relationship between emotion information and the color of an image showing a participant according to an embodiment of the present invention. FIG. [Figure 6] 10 is a diagram showing an example of a display on a display unit of a PC when emotion information is notified according to an embodiment of the present invention. [Figure 7] 10 is a diagram showing an example of a display on a display unit of a PC when emotion information is notified according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0031] (Embodiment) An example of an embodiment of the present disclosure will be described below with reference to the drawings. The same or equivalent components and parts in each drawing are designated by the same reference numerals. The dimensional proportions in the drawings are exaggerated for the sake of explanation and may differ from the actual proportions.

[0032] An example of an information processing system 10 according to this embodiment will be described with reference to FIG.

[0033] FIG. 1 is a diagram showing an example of a schematic configuration of an information processing system 10 according to this embodiment. As shown in FIG. 1, an information processing system 10 according to this embodiment includes a plurality of PCs (Personal Computers) 20, a server 50, and an online conference server 60. The PCs 20, the server 50, and the online conference server 60 are connected via a network N. This network N may be, for example, a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet. In this embodiment, the server 50 is an example of an information processing device. The online conference server 60 may function as the server 50.

[0034] The PC 20 is an example of a participant terminal device used by a participant participating in an online conference in the information processing system 10, and includes a tablet PC or a smartphone. In this embodiment, one PC is used per participant. Each PC 20 stores information identifying the participant using it, such as a user ID, and when connecting to and logging in to the online conference server 60, the information identifying the participant is also transmitted to the server 50 or the online conference server 60. Therefore, with regard to emotion information of participants during an online conference, it is possible to determine whose emotion information it is for each PC 20.

[0035] The online conference server 60 is a server computer that provides online conference services in the information processing system 10. Here, the online conference services provided by the online conference server 60 include online meetings, online presentations, online seminars (webinars), and the like.

[0036] Furthermore, the information processing system 10 in this embodiment is a system in which, when participants use PCs 20 to connect to online conference server 60 and participate in an online conference, server 50 acquires emotion information of participants from PCs 20 or online conference server 60, and notifies other participants of participants whose emotion information has reached a predetermined state. That is, in an online conference, compared to a face-to-face conference, it is difficult to grasp the state of participants, and it is not possible to proceed with the conference according to the state of the participants. Therefore, by acquiring emotion information of participants, it becomes possible to proceed with the conference according to the state of the participants.

[0037] FIG. 2 is a block diagram showing the hardware configuration of the PC 20 according to this embodiment.

[0038] 2, the PC 20 includes a CPU (Central Processing Unit) 21, which is an example of a processor, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage unit 24, an input unit 25, a display unit 26, a camera 27, a microphone 28, a speaker 29, and a communication interface 30. Each component is connected to each other via a bus 31 so as to be able to communicate with each other.

[0039] The CPU 21 is a central processing unit that executes various programs and controls each component. That is, the CPU 21 reads programs from the ROM 22 or the storage unit 24 and executes the programs using the RAM 23 as a work area. The CPU 21 controls the above components and performs various arithmetic processing in accordance with the programs recorded in the ROM 22 or the storage unit 24. In this embodiment, the programs are stored in the ROM 22 or the storage unit 24.

[0040] The ROM 22 stores various programs and various data. The RAM 23 serves as a working area and temporarily stores programs or data. The storage unit 24 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0041] The input unit 25 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

[0042] The display unit 26 is, for example, a liquid crystal display, and displays various types of information under the control of the CPU 21.

[0043] The camera 27 captures images of participants in the online conference.

[0044] The microphone 28 picks up the speech of the participants in the online conference.

[0045] The speaker 29 uses another PC 20 connected via the network N. Outputs the comments of other participants in an online meeting.

[0046] The communication interface 30 is connected to a network N, and enables the PC 20 to communicate with other PCs 20 and other devices such as a server 50.

[0047] 3 is a block diagram showing the hardware configuration of the server 50 and the online conference server 60. Since the server 50 and the online conference server 60 basically have a general computer configuration, the server 50 will be described as a representative.

[0048] 3, the server 50 includes a CPU 51, a ROM 52, a RAM 53, a storage unit 54, and a communication interface 55. Each component is connected to each other so as to be able to communicate with each other via a bus 56. The server 50 may also include an input unit such as a keyboard and a display unit such as a display.

[0049] The CPU 51 is a central processing unit that executes various programs and controls each component. That is, the CPU 51 reads programs from the ROM 52 or the storage unit 54 and executes the programs using the RAM 53 as a work area. The CPU 51 controls the above components and performs various arithmetic processing in accordance with the programs recorded in the ROM 52 or the storage unit 54. In this embodiment, the ROM 52 or the storage unit 54 stores programs and various data.

[0050] The ROM 52 stores various programs and various data. The RAM 53 temporarily stores programs or data as a working area. The storage unit 54 is configured with an HDD or SSD and stores various programs including an operating system and various data. The storage unit 54 of the server 50 stores various data transmitted from the PC 20 and the online conference server 60.

[0051] The communication interface 55 is for communicating with other devices such as the PC 20 and the online conference server 60, and uses standards such as the Internet, intranet, Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0052] Next, the operation of the information processing system 10 will be described.

[0053] 4 is a flowchart showing an example of a process flow in which server 50 acquires emotion information of a participant and, if the emotion information is in a predetermined state, notifies other participants of information indicating the participant who has entered the predetermined state. Note that this process may be performed by online conference server 60 instead of server 50.

[0054] First, the server 50 is connected to the online conference server 60. Through this connection, the server 50 acquires emotion information of the participants estimated by the PC 20 via the online conference server 60. Note that the emotion information is not limited to being acquired from the online conference server 60, but may be acquired from the PC 20.

[0055] Here, the emotion information of the participants in the online conference calculated by the PC 20 will be described.

[0056] Emotion information is calculated from images acquired by the PC 20. Specifically, it is calculated from the facial expressions of the participants in the captured images taken by the camera 27 of the PC 20 and the movements of the participants in the captured images. Facial expressions such as frowning, wrinkles on the forehead, and smiling are calculated from the movements of the eyes, eyebrows, and lips, and emotion information is calculated from the captured images. Emotions are calculated from the movements of the participants, such as placing a hand on the mouth or tilting the head, from the captured images.

[0057] As shown in Figure 5, emotional information includes "confusion, doubt," "laughter, joy," "interested," and "disinterested, fatigued." When the emotional information is calculated as "confusion, doubt," "laughter, joy," "interested," or "disinterested, fatigued," it is notified to other participants. For the speaker, "confusion, doubt" alerts the speaker that the lecture was difficult to understand, which promotes communication by setting aside time for questions. For the attendees, it allows them to detect that other attendees are having questions at the same time, creating an atmosphere in which it is easy to ask questions. For the speaker, "laughter, joy" allows them to interactively check the reactions of the speaker when breaking the ice or making a digression, even without a camera image of the attendees. For the speaker, "interested" alerts the speaker to the timing to add additional information or detailed explanations. For the attendees, it also provides an opportunity to discover potential volunteers. For the speaker, "disinterested, fatigued" alerts the speaker to the timing to simplify or omit the explanation or to take a break.

[0058] Furthermore, in this embodiment, when a participant's facial expression or movement is a predetermined pattern, the PC 20 calculates the emotional information as shown in Figure 5 corresponding to that pattern. For example, if the participant's facial expression is a "frowning expression," the emotional information is calculated as "confusion, doubt," and if the participant's facial expression is a "smile," the emotional information is calculated as "laughter, joy." Furthermore, for example, if the participant's movement is a "movement of placing a hand on the mouth," the emotional information is calculated as "disinterested, tired."

[0059] When the emotional information of a participant calculated by PC 20 becomes a predetermined state such as "confusion, doubt," the emotional information and information identifying the participant are transmitted from PC 20 to server 50 via online conference server 60.

[0060] The emotional information of the participants is not limited to being calculated from the facial expressions and movements of the participants in the images captured by the camera 27. For example, it may be calculated from the participants' voices, such as their speech, intonation, and voice volume, acquired by the microphone 28 of the PC 20. Furthermore, although not shown, the emotional information may be calculated from the participants' biometric information acquired by a device (not shown) connected to the PC 20 that acquires vital signs such as respiratory information, body temperature information, blood pressure information, and pulse information, as well as sweating and brain wave information. Specifically, the emotional information may be calculated based on stress levels, joy, anger, sadness, and happiness, etc., acquired from respiratory information, body temperature information, blood pressure information, pulse information, sweating information, brain wave information, etc. Furthermore, the emotional information may be calculated by combining these images, voices, and biometric information.

[0061] When the emotion information of such a participant reaches a predetermined state, the emotion information is transmitted from PC 20 to server 50 via online conference server 60. Note that the case is not limited to when emotion information is transmitted from PC 20 to server 50 when it reaches a predetermined state, and all emotion information may be transmitted from PC 20 to server 50 without determining whether or not the predetermined state has been reached. In this case, whether or not the emotion information has reached the predetermined state is determined by server 50.

[0062] In addition, the administrator and speakers of the online conference access the server 50 from the PC 20 to "enable / disable functions," "set notification methods," and "set colors."

[0063] "Enable / disable function" is a setting for enabling or disabling the sensory information notification function that notifies other participants of the emotional information of a participant in an online conference. When the sensory information notification function is set to enabled, an image showing a participant whose emotional information has reached a predetermined state is displayed on the display unit 26 of the PC 20 of the other participants.

[0064] The "notification method setting" is a setting for how to notify "images representing participants" during an online conference. Examples include "face photo," "name," and "user ID." As shown in FIG. 6, "face photo" is a setting for displaying a prepared face photo G of a participant or a face image of a participant taken by camera 27 during an online conference on display unit 26 of PC 20. "Name" is a setting for displaying the name of a participant on display unit 26 of PC 20. "User ID" is a setting for displaying an ID that can identify a participant on display unit 26 of PC 20.

[0065] "Color setting" is the color setting of the "image showing the participant" for each emotional information. As shown in FIG. 5, for example, "yellow" is set for emotional information "confusion, doubt," "green" for emotional information "laughter, joy," "pink" for emotional information "interested," and "blue" for emotional information "uninterested, fatigued." When the "image showing the participant" is a facial photograph G, the color of the "image showing the participant" is displayed on the display unit 26 of the PC 20 of the other participant by changing the color of the frame F around the facial photograph G according to the type of emotional information, as shown in FIG. 7. Note that the color of the frame F around the facial photograph G is not limited to being changed, and other parts may be displayed with different colors.

[0066] Note that emotion information of participants does not necessarily have to be calculated by PC 20, but may instead be calculated by online conference server 60 or server 50, which acquires images captured by camera 27 from PC 20. When emotion information of participants is calculated by PC 20, images captured by camera 27 are not transmitted to online conference server 60 or server 50, which has the advantage of protecting personal information and reducing the amount of data transmitted. On the other hand, when emotion information is calculated by online conference server 60 or server 50, there is the advantage that a program for calculating emotion information does not need to be installed on PC 20.

[0067] After the "function enable / disable," "notification method setting," and "color setting" are performed, the process of the flowchart shown in Fig. 4 begins. Note that the "function enable / disable," "notification method setting," and "color setting" do not necessarily have to be performed before the start of the online conference, but may be performed during the online conference or may be changeable during the online conference.

[0068] 4, the CPU 51 of the server 50 receives emotion information of participants whose emotion information has reached a predetermined state from the online conference server 60. Then, the process proceeds to the next step, S101.

[0069] In step S101, the CPU 51 of the server 50 extracts the room ID of the online conference. That is, it identifies the online conference in which a participant who has reached a predetermined state is participating. Then, the process proceeds to the next step, S102.

[0070] In step S102, the CPU 51 of the server 50 determines whether the sensory information recognition function is enabled or not, and then proceeds to the next step S103.

[0071] In step S103, if it is determined in step S102 that the sensory information notifying function is enabled, the process proceeds to the next step S104, and if it is determined that the sensory information notifying function is disabled, the process ends. In step S104, CPU 51 of server 50 determines a color in accordance with the emotion information received in step S100 (see FIG. 5), and then proceeds to the next step, S105.

[0072] In step S105, the CPU 51 of the server 50 identifies the participants who are participating in the online conference from the room ID extracted in step S101, and then proceeds to the next step, S106.

[0073] In step S106, the CPU 51 of the server 50 instructs the display unit 26 of the lecturer's PC 20 to display the participant icons received in step S100 and the colors determined in step S104, and then proceeds to the next step S107.

[0074] In step S107, CPU 51 of server 50 instructs display unit 26 of PC 20 of participants other than the participant who received the emotion information in step S100 to display the icon of the participant received in step S100 and the color determined in step S104, and then ends the process.

[0075] The present invention is not limited to the above-described embodiment, and various modifications and applications are possible within the scope of the gist of the present invention.

[0076] For example, in the above-described embodiment, emotion information is notified to all participants in the online conference. However, the present invention is not limited to this. It is also possible to notify only predetermined participants among the participants in the online conference. That is, the administrator or speaker of the online conference may be able to set the "notification range" in the same way as the "color setting" and "notification method setting" described above. Examples of the "notification range setting" include "everyone," "speaker only," "group," and "designated participants." "Everyone" is a setting for notifying all participants in the online conference. "Speaker only" is a setting for notifying only the speaker of the online conference. "Group" is a setting for notifying participants who belong to the same predetermined group. "Designated participants" is a setting for notifying designated participants among the participants. Note that the "notification range setting" is not limited to the above-described example. For example, it may be participants who are participating in the same place or space, or participants whose location information is nearby.

[0077] Furthermore, the present invention is not limited to acquiring and reporting emotion information from all participants, but participants from whom emotion information is to be acquired may be set based on the attributes of the participants, such as first-time participants and repeat participants.

[0078] Furthermore, the emotional information of the participants to be notified may not be the emotional information of each individual participant, but may instead display the average emotional information or analysis results of the participants in a group. In other words, the emotional information of participants in a predetermined group may be obtained, and the average emotional information or the most common emotional state may be notified to the other participants.

[0079] Furthermore, when the number of participants having specific emotional information exceeds a predetermined number, it is not limited to displaying the participant's facial photograph G, but the overall color of the display unit 26 of the participant's PC 20 may be changed or the facial photograph G may be made to blink.

[0080] Furthermore, the method of displaying emotional information may differ for each participant. For example, for participants who are close to the location information of a "confused" participant or other participants in the same space, it is possible to make it easier for them to follow up on the confused participant by making the facial photograph G larger in size rather than just the color of the facial photograph G, making it easier for them to perceive the confused participant. Furthermore, the display unit 26 of the speaker's PC 20 may change the size and order of the facial photographs G depending on the number of times "confusion" is detected and the duration of the detection.

[0081] Furthermore, the notification is not limited to displaying an image showing the participant on the display unit 26 of the PC 20, but may be other notification methods such as sound or vibration.

[0082] Furthermore, if the facial expressions and movements of a participant match a predetermined pattern, the corresponding emotional information is calculated; however, this is not limited to this, and emotions may be determined by AI that has been trained using the parameters of facial expressions and movements related to the above-mentioned emotions and the resulting data on emotions judged by humans as training data.

[0083] In the above embodiment, the program is pre-stored (installed) in a ROM or a storage unit, but the present invention is not limited to this. The program may be provided in a form recorded on a recording medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network N.

[0084] Furthermore, in the above-described embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

[0085] Furthermore, the operations of the processors in the above-described embodiments may not only be performed by a single processor, but may also be performed by multiple processors located at physically separate locations working together. Furthermore, the order of the operations of the processors is not limited to the order described in the above-described embodiments, and may be changed as appropriate. [Explanation of symbols]

[0086] 10 Information Processing Systems 20 PC 50 servers 60 Online Conference Server

Claims

1. a processor; The processor: When emotion information relating to the emotion of a participant participating in an online conference reaches a predetermined state, information indicating the participant who has reached the predetermined state is notified to other participants; The processor: notifying the participant who has reached the predetermined state by displaying an image including a facial photograph of the participant on a display unit of a participant terminal device used by the other participant in the online conference; displaying the image on the display unit in a different display method depending on the other participants; An information processing device that displays the image on the display unit using a display method that differs depending on the other participants, and that makes it easier for participants in the same space as the participant who has reached the predetermined state to perceive the participant who has reached the predetermined state.

2. The information processing apparatus according to claim 1 , wherein the other participants are predetermined participants among participants participating in the online conference.

3. The processor:

3. The information processing apparatus according to claim 1, wherein the color of the image displayed on the display unit of the participant terminal device of the other participant is changed depending on the type of the emotion information.

4. 4. The information processing device according to claim 1, wherein the emotion information is calculated from a captured image of the participant acquired by the participant terminal device.

5. The information processing device according to claim 4 , wherein the emotion information is calculated from facial expressions of the participants in the captured images acquired by the participant terminal devices.

6. The information processing device according to claim 4 or 5, wherein the emotion information is calculated from the movement of the participant in the captured image acquired by the participant terminal device.

7. 7. The information processing device according to claim 1, wherein the emotion information is calculated from the voice of the participant acquired by the participant terminal device.

8. 8. The information processing device according to claim 1, wherein the emotion information is calculated from biometric information of the participant acquired by the participant terminal device.

9. The information processing device according to claim 8 , wherein the biological information includes the participant's respiration, body temperature, blood pressure, or pulse rate.

10. On the computer, When emotion information relating to the emotion of a participant participating in an online conference reaches a predetermined state, an image including a facial photograph of the participant who has reached the predetermined state is displayed on a display unit of a participant terminal device used by the other participants in the online conference, thereby notifying the other participants of information indicating the participant who has reached the predetermined state; displaying the image on the display unit in a different display method depending on the other participants; An information processing program for executing a process of displaying the image on the display unit using a display method that differs depending on the other participants, and that makes it easier for participants in the same space as the participant who has reached the predetermined state to perceive the participant who has reached the predetermined state.

Images