Reaction Prompt Control Device, Method, and Program

Abstract

One aspect of the present invention acquires information items indicating the respective reaction states of a plurality of listeners in response to an event, and generates reaction aggregation information by aggregating the reaction states of the plurality of listeners on the basis of the acquired information items respectively indicating the reaction states. Control information for performing variable control on vibration and temperature is generated on the basis of the generated reaction aggregation information, and the generated control information is outputted so as to be presented to a speaker in the event.

Description

Technical Field

[0001] One aspect of the present invention relates to a reaction presentation control device, method, and program for presenting a listener's reaction to a speaker at an event such as a meeting, lecture, exhibition, etc.

Background Art

[0002] For example, grasping the reaction of the listener at an event such as a meeting, lecture, exhibition, etc. is important for improving the quality of the event. As a method for grasping the reaction of the listener, for example, a questionnaire survey is common, but there is a problem that the reaction of the listener cannot be grasped in real time.

[0003] Therefore, as described in Non-Patent Document 1, for example, a method of detecting the reaction of the listener during the event using a sensor or the like and presenting the result to the speaker by an avatar, or as described in Non-Patent Document 2, for example, a method of detecting the nodding of the audience and presenting the timing to the speaker using vision or touch has been proposed.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the method described in Non-Patent Document 1, in a scenario where the speaker is already using visual information, such as when speaking while looking at materials, both the materials and the avatar must be viewed. For this reason, there is a problem that the cognitive load on the speaker increases and it becomes difficult to clearly recognize the reactions of the audience.

[0006] On the other hand, in the method described in Non-Patent Document 2, in a scenario where the speaker is already using visual information, the reaction of the listener is presented using touch, so it is possible to present the reaction of the listener to the speaker with a lower cognitive load compared to the case of using visual information. However, in the method described in Non-Patent Document 2, only the timing of the listener's nodding is presented. For this reason, even if the speaker wants to know the reaction of the listener including the content of the explanation, for example, the speaker cannot clearly recognize it.

[0007] This invention has been made paying attention to the above circumstances, and aims to provide a technique that enables a speaker to more detailedly recognize the reaction state of a listener with a small cognitive load.

Means for Solving the Problem

[0008] In order to solve the above problems, one aspect of the reaction presentation control device or reaction presentation control method according to the present invention is to obtain information representing the reaction states of a plurality of listeners with respect to the speech content of the speaker, generate the obtained information, and output the generated information so as to be presented to the speaker. Information representing body movements that reflect the reactions of multiple listeners and information representing opinions input by the listeners, the information representing the reaction states of a plurality of listeners as respectively, and the obtained generate first aggregated information that aggregates the body movements of the multiple listeners based on the information representing the body movements, and second aggregated information that aggregates the opinions of the multiple listeners based on the acquired information representing the opinions to generate. And the generated generate vibration control information that represents the body movements of the multiple listeners in terms of the intensity of vibration based on the first aggregated information, and generate temperature control information that represents the opinions of the multiple listeners in terms of the height of temperature based on the generated second aggregated information, and use the generated vibration control information and the temperature control information is output so as to be presented to the speaker.

[0009] According to one aspect of the present invention, for example, in an event such as a meeting, a lecture, an exhibition, etc., information representing the reactions of a plurality of listeners with respect to the speech content of the speaker is respectively obtained and aggregated, and the aggregated result of the information representing the reaction information body movements and opinions input by the listeners is as respectively obtained and aggregated, the above body movements the aggregated result of and the aggregation results of the above opinions respectively vibration and temperature to useIt is presented to the speaker. Therefore, compared with the case where only visual display and vibration are used, the speaker can recognize the reaction states of a plurality of listeners in an event in more detail with less cognitive load.

Advantages of the Invention

[0010] That is, according to one aspect of the present invention, it is possible to provide a technique that enables a speaker to recognize the reaction states of listeners in more detail with less cognitive load.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0013] [One Embodiment] (Configuration Example) (1) System FIG. 1 is a diagram showing an example of the configuration of an online conference system including a reaction presentation control device according to an embodiment of the present invention.

[0014] An online conference system according to one embodiment includes a server device SV having a function as a reaction presentation control device, and between this server device SV, a speaker terminal ET used by a speaker, and a plurality of listener terminals UT1 to UTn respectively used by a plurality of listeners, information data can be transmitted via a network NW.

[0015] The network NW includes, for example, a wide area network centered on the Internet and an access network for accessing this wide area network. As the access network, for example, a public communication network using wired or wireless, a LAN (Local Area Network) using wired or wireless, or a CATV (Cable Television) network is used.

[0016] (2) Device (2-1) Listener Terminals UT1 to UTn FIG. 2 is a block diagram showing an example of the hardware and software configuration of listener terminals UT1 to UTn.

[0017] The listener terminals UT1 to UTn are composed of, for example, personal computers having an online conference function. Note that as the listener terminals UT1 to UTn, other tablet-type terminals, smartphones, etc. may be used as long as they have the same function.

[0018] The listener terminals UT1 to UTn are each provided with a control unit 1A that uses a hardware processor such as a Central Processing Unit (CPU). Then, a storage unit having a program storage unit 2A and a data storage unit 3A, a communication interface (hereinafter, the interface is referred to as I / F) unit 4A, and an input / output I / F unit 5A are connected to this control unit 1A via a bus (not shown).

[0019] Under the control of the control unit 1A, the communication I / F unit 4A performs data communication with the server device SV using the communication protocol defined by the network NW.

[0020] A camera 51 and an input device 52 are connected to the input / output I / F unit 5A. In this example, the camera 51 is used to capture the face image of the listener during the meeting. The input device 52 includes, for example, a keyboard and a mouse and is used for the listener to input their opinions during the meeting. In addition, a display device (not shown) is connected to the input / output I / F unit 5A, and it is possible to display the captured image of the speaker and the meeting materials on this display device.

[0021] The program storage unit 2A is configured by combining, for example, a non-volatile memory such as an SSD (Solid State Drive) that can be written to and read from at any time as a storage medium and a non-volatile memory such as a ROM (Read Only Memory). In addition to middleware such as an OS (Operating System), it stores application programs necessary to execute various control processes according to one embodiment. Hereinafter, the OS and each application program are collectively referred to as a program.

[0022] The data storage unit 3A is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium and a volatile memory such as a RAM (Random Access Memory), and includes an action determination information storage unit 31A and an opinion determination information storage unit 32A as the main storage units necessary for implementing one embodiment.

[0023] The action determination information storage unit 31A is used to store the result determined by the control unit 1A described later regarding the actions in which the reactions shown by the listeners during the meeting to the speech content of the speaker are reflected.

[0024] The opinion determination information storage unit 32A is used to store the result determined by the control unit 1A described later regarding the opinions input by the listeners during the meeting to the speech content of the speaker.

[0025] The control unit 1A includes an action determination processing unit 11A, an opinion determination processing unit 12A, and a reaction determination information transmission processing unit 13A as the processing functions necessary for implementing one embodiment. These processing units 11A to 13A are all realized by causing the hardware processor of the control unit 1A to execute the application program stored in the program storage unit 2A. Note that the above application program may be downloaded from the server device SV when necessary and stored in the program storage unit 2A in addition to being stored in the program storage unit 2A in advance.

[0026] The action determination processing unit 11A captures the video data of the face or upper body of the listener imaged by the camera 51 during the meeting via the input / output I / F unit 5A, determines the actions in which the reactions of the listener are reflected based on the captured video data, and performs the process of storing the information representing the determination result in the action determination information storage unit 31A.

[0027] The opinion determination processing unit 12A captures information representing an opinion on the speech content input by the listener using the input device 52 during the meeting via the input / output I / F unit 5A, determines the opinion based on the captured information, and performs a process of storing information representing the determination result in the opinion determination information storage unit 32A.

[0028] The reaction determination information transmission processing unit 13A reads, at periodic or arbitrary timings, information representing the determination result of the action and the determination result of the opinion from the action determination information storage unit 31A and the opinion determination information storage unit 32A, respectively, inserts each of the read determination results into, for example, one packet, and performs a process of transmitting the packet from the communication I / F unit 4A to the server device SV.

[0029] (2-2) Server device SV FIG. 3 is a block diagram showing an example of the hardware and software configuration of the server device SV.

[0030] The server device SV is composed of, for example, a server computer arranged on the cloud, and includes a control unit 1C that uses a hardware processor such as a CPU. Then, a storage unit having a program storage unit 2C and a data storage unit 3C and a communication I / F unit 4C are connected to the control unit 1C via a bus (not shown).

[0031] Under the control of the control unit 1C, the communication I / F unit 4C performs transmission and reception of information data with the speaker terminal ET and the listener terminals UT1 to UTn using the communication protocol defined by the network NW.

[0032] The program storage unit 2C is configured by combining, for example, a non-volatile memory such as an HDD or an SSD that can be written to and read from at any time as a storage medium and a non-volatile memory such as a ROM, and stores programs necessary for executing various control processes according to an embodiment of the present invention in addition to middleware such as an OS.

[0033] The data storage unit 3C is, for example, a combination of a non-volatile memory such as an HDD or an SSD that can be written to and read from at any time as a storage medium and a volatile memory such as a RAM, and includes a reaction determination information storage unit 31C. This reaction determination information storage unit 31C is used to store the reaction determination information transmitted from a plurality of listener terminals UT1 to UTn.

[0034] The control unit 1C includes, as a processing function according to an embodiment of the present invention, a reaction determination information reception processing unit 11C, an action aggregation processing unit 12C, an opinion aggregation processing unit 13C, a vibration presentation control information generation processing unit 14C, a temperature presentation control information generation processing unit 15C, and a presentation control information transmission processing unit 16C. These processing units 11C to 16C are all realized by causing the hardware processor of the control unit 1C to execute an application program stored in the program storage unit 2C.

[0035] The reaction determination information reception processing unit 11C receives the reaction determination information transmitted from the listener terminals UT1 to UTn via the communication I / F unit 4C, and performs a process of storing each received reaction determination information in the reaction determination information storage unit 31C.

[0036] The action aggregation processing unit 12C selects and reads information representing the determination result of the action from each of the reaction determination information stored in the reaction determination information storage unit 31C. Then, based on the information representing the determination result of each of the read actions, a process of generating action aggregation information that aggregates the actions of a plurality of listeners with respect to the speech content of the speaker is performed. Note that the specific method of aggregating actions will be described in the operation example.

[0037] The opinion aggregation processing unit 13C selects and reads information representing the determination result of the opinion from each of the reaction determination information stored in the reaction determination information storage unit 31C. Then, based on the information representing the determination result of each of the read opinions, a process of generating opinion aggregation information that aggregates the opinions of a plurality of listeners with respect to the speech content of the speaker is performed. Note that the specific method of aggregating opinions will also be described in the operation example.

[0038] Based on the behavior aggregation information generated by the above-mentioned behavior aggregation processing unit 12C, the vibration prompt control information generation processing unit 14C performs a process of generating vibration control information obtained by converting the aggregation result of behaviors into the intensity of vibration.

[0039] Based on the opinion aggregation information generated by the above-mentioned opinion aggregation processing unit 13C, the temperature prompt control information generation processing unit 15C performs a process of generating temperature control information obtained by converting the aggregation result of opinions into the level of temperature.

[0040] The prompt control information transmission processing unit 16C generates prompt control information by inserting the vibration prompt control information generated by the above-mentioned vibration prompt control information generation processing unit 14C and the temperature prompt control information generated by the above-mentioned temperature prompt control information generation processing unit 15C into, for example, one packet. Then, the generated prompt control information is transmitted from the communication I / F unit 4C to the speaker terminal ET.

[0041] (2-3) Speaker Terminal ET FIG. 4 is a block diagram showing an example of the hardware and software configuration of the speaker terminal ET.

[0042] The speaker terminal ET also consists of a personal computer having an online conferencing function, similar to the listener terminals UT1 to UTn described above.

[0043] The speaker terminal ET includes a control unit 1B that uses a hardware processor such as a central processing unit (CPU). Then, a storage unit having a program storage unit 2B and a data storage unit 3B, a communication I / F unit 4B, and an input / output I / F unit 5B are connected to the control unit 1B via a bus (not shown).

[0044] Under the control of the control unit 1B, the communication I / F unit 4B performs data communication with the server device SV using the communication protocol defined by the network NW.

[0045] In the input / output I / F unit 5B, in addition to an input / output device (not shown) having, for example, a keyboard and a display, a mouse MU is connected. The mouse MU includes, in addition to an input unit that performs normal operations such as click and scroll operations, a vibration device 6 and a heat generation device 7.

[0046] The vibration device 6 includes, for example, a small motor and performs a vibration operation with an intensity specified by the vibration drive signal when the vibration drive signal is given. The heat generation device 7 includes, for example, a Peltier element and performs a heat generation or cooling operation at a temperature specified by the heat generation drive signal when the heat generation drive signal is given.

[0047] The program storage unit 2C is configured by combining, for example, a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium and a non-volatile memory such as a ROM, and stores application programs necessary for executing various control processes according to one embodiment in addition to middleware such as an OS. Hereinafter, the OS and each application program are collectively referred to as a program.

[0048] The data storage unit 3B is, for example, a combination of a non-volatile memory such as an SSD that can be written to and read from at any time as a storage medium and a volatile memory such as a RAM, and includes a presentation control information storage unit 31B as a main storage unit necessary for implementing one embodiment.

[0049] The presentation control information storage unit 31B stores the presentation control information sent from the server device SV for a period until the presentation operation is performed.

[0050] The control unit 1B includes a presentation control information reception processing unit 11B, a vibration drive signal generation processing unit 12B, and a heat generation drive signal generation processing unit 13B as the processing functions necessary to implement an embodiment. These processing units 11B to 13B are all realized by causing the hardware processor of the control unit 1B to execute the application program stored in the program storage unit 2B. Note that, in addition to storing the application program in the program storage unit 2B in advance, the application program may be downloaded from the server device SV when necessary and stored in the program storage unit 2B.

[0051] The presentation control information reception processing unit 11B receives the presentation control information transmitted from the server device SV via the communication I / F unit 4B, and performs a process of storing the received presentation control information in the presentation control information storage unit 31B.

[0052] The vibration drive signal generation processing unit 12B reads vibration presentation control information from the presentation control information stored in the presentation control information storage unit 31B. Then, based on the read vibration presentation control information, it generates a vibration drive signal and performs a process of outputting the generated vibration drive signal to the vibration device 6 of the mouse MU via the input / output I / F unit 5B.

[0053] The heat generation drive signal generation processing unit 13B reads temperature presentation control information from the presentation control information stored in the presentation control information storage unit 31B. Then, based on the read temperature presentation control information, it generates a heat generation drive signal and performs a process of outputting the generated heat generation drive signal to the heat generation device 7 of the mouse MU via the input / output I / F unit 5B.

[0054] (Operation example) Next, the reaction presentation operation executed in the online conference system configured as described above will be described.

[0055] (1) Reaction determination process for the listeners in the listener terminals UT1 to UTn FIG. 5 is a flowchart showing the processing procedure and processing content of the reaction determination process executed by the control unit 1A of the listener terminals UT1 to UTn.

[0056] The control unit 1A of the listener terminals UT1 to UTn monitors the presence or absence of participation of listeners in the online meeting in step S10. In this state, when a listener performs an access operation to participate in the online meeting, the control unit 1A executes a procedure to form a conference communication link with the server device SV under the control of the application program for the online meeting. Thus, it becomes possible to participate in the online meeting hereafter.

[0057] (1-1) Judgment of behavior When the online meeting starts, the control unit 1A of the listener terminals UT1 to UTn captures, in step S11, video data of the face or upper body of the listener captured by the camera 51 via the input / output I / F unit 5A while performing transmission / reception processing of various data related to a normal online meeting under the control of the behavior judgment processing unit 11A. Then, in step S12, based on the captured video data, a behavior in which the reaction of the listener to the speaker is reflected is judged as follows, for example.

[0058] That is, the behavior judgment processing unit 11A performs image recognition processing on the above video data using, for example, a basic image pattern for behavior judgment prepared in advance, to judge the presence or absence of the "nodding" motion of the listener. Then, the presence or absence of this "nodding" motion is replaced with numerical values such as "1" and "-1", and the replaced numerical values are temporarily stored in the behavior judgment information storage unit 31A as information representing the judgment result of the behavior representing the reaction of the listener to the speaker at this time.

[0059] In addition, as behaviors representing the reaction of the listener, other reaction motions such as "shaking the head", "raising the hand", "clapping", and "laughing" may be used, and the types of motions are not limited.

[0060] (1-2) Judgment of opinion At the same time, the control unit 1A of the listener terminals UT1 to UTn, under the control of the opinion determination processing unit 12A, imports information representing the opinion input by the listener at the input device 52 via the input / output I / F unit 5A in step S13. Then, in step S14, based on the imported input information, a process of judging the listener's opinion on the content of the speaker's remark is executed, for example, as follows.

[0061] That is, the opinion judgment processing unit 12A monitors which of two software buttons, "Agree" and "Disagree," that are displayed on the display while the speaker is speaking or after the speaker has finished speaking, is operated. When either button is operated, the input information corresponding to the operated button, that is, the input information indicating "Agree" or "Disagree," is replaced with a numerical value, for example, "1" or "-1," respectively, and the replaced numerical value is temporarily stored in the opinion judgment information storage unit 32A as information indicating the judgment result of the listener's opinion.

[0062] As a method for inputting and judging opinions, an input device having a dedicated hardware button for answering may be provided, or instead of providing an input button, the opinion of the listener may be judged by image recognition of predetermined expressive actions of "agree" or "disagree" from video data captured by the camera 51 used for the above-mentioned behavior judgment. Also, opinions are not limited to "agree" or "disagree", and other opinions such as "next" or "tell me more" may be accepted and the results may be judged.

[0063] (1-3) Transmission process of reaction determination information Next, in step S15, the control unit 1A of the listener terminals UT1 to UTn, under the control of the reaction judgment information transmission processing unit 13A, reads information representing the behavior judgment result and information representing the opinion judgment result from the behavior judgment information storage unit 31A and the opinion judgment information storage unit 32A, respectively, inserts the information representing each of the read judgment results into, for example, one packet, and transmits this packet from the communication I / F unit 4A to the server device SV as reaction judgment information.

[0064] Note that the transmission process of the reaction determination information may be performed periodically at a preset cycle, or may be performed only when a significant result is obtained as the determination result of the action or opinion.

[0065] The control units 1A of the listener terminals UT1 to UTn finally perform the determination of the end of participation in the online meeting in step S16. Then, when the participation end operation is not performed and the user continues to participate in the meeting, the process returns to step S11 and a series of reaction determination processes in steps S11 to S15 are repeatedly executed. On the other hand, when the participation end operation for the meeting is performed, the disconnection procedure of the communication link for the meeting is executed with the server device SV, and the device returns to the standby state.

[0066] (2) Reaction presentation control process in the server device SV FIG. 6 is a flowchart showing the processing procedure and processing content of the reaction presentation control process executed by the control unit 1C of the server device SV.

[0067] The control unit 1C of the server device SV monitors, in step S20, for example, a meeting start request transmitted from the terminal used by the host. When the meeting start request is received in this state, under the control of the online meeting application program, a communication link for the online meeting is set between the speaker terminal ET that has transmitted the participation request and the plurality of listener terminals UT1 to UTn, respectively, and the online meeting is enabled thereafter.

[0068] (2-1) Acquisition process of reaction determination information During the meeting period, the control unit 1C of the server device SV receives, in step S21 under the control of the reaction determination information reception processing unit 11C, the reaction determination information transmitted from each of the listener terminals UT1 to UTn via the communication I / F unit 4C, and stores each received reaction determination information in the reaction determination information storage unit 31C.

[0069] In addition, in order to associate the reactions of each listener with the speaker's speech, a fixed reception waiting period is set as the acquisition period of the above reaction determination information, and each piece of reaction determination information received within this period is stored in the reaction determination information storage unit 31C as the reaction determination information for one speech.

[0070] (2-2) Action Aggregation Processing When the acquisition of the reaction determination information in the reception waiting period is completed under the control of the control unit 1C of the server device SV, first, in step S22 under the control of the action aggregation processing unit 12C, action determination information is selected and read from the reaction determination information of a plurality of listeners stored in the reaction determination information storage unit 31C. Then, in step S23, the above-mentioned action determination information of the plurality of listeners that has been read is aggregated as follows, for example.

[0071] That is, the action aggregation processing unit 12C first calculates the sum of the numerical values "1" indicating "nodding" among each piece of action determination information, and divides the calculated sum by the number of listeners participating in the meeting to calculate the action aggregation index Aall. For example, if there are 100 listeners participating in the meeting now, and it is determined that 70 of them nodded in response to a certain speech by the speaker, the action aggregation processing unit 12C calculates the action aggregation index as Aall = 0.7.

[0072] (2-3) Generation Processing of Vibration Presentation Control Information Subsequently, under the control of the vibration presentation control information generation processing unit 14C, the control unit 1C of the server device SV generates vibration presentation control information in step S24 as follows, for example.

[0073] That is, the vibration presentation control information generation processing unit 14C sets a judgment threshold to 0.5 to judge whether the number of listeners who nodded exceeds the majority, and compares the calculated behavior summary index Aall with the judgment threshold of 0.5. Then, if the comparison result is Aall > 0.5, that is, if the number of listeners who nodded exceeds the majority, the vibration presentation control information is set to "on" to generate vibration. On the other hand, if Aall ≦ 0.5, that is, if the number of listeners who nodded does not exceed the majority, the vibration presentation control information is set to "off" to not generate vibration. As a result, for example, when the behavior summary index is Aall = 0.7 as described above, vibration presentation control information in which vibration is set to "on" is generated.

[0074] The above-mentioned determination threshold value can be set to any value other than 0.5, such as 0.3 or 0.25, depending on the purpose.

[0075] (2-4) Opinion collection process When the process of acquiring the reaction judgment information during the reception waiting period is completed, the control unit 1C of the server device SV selects and reads the opinion judgment information from the reaction judgment information of the multiple listeners stored in the reaction judgment information storage unit 31C under the control of the opinion aggregation processing unit 13C in step S25. Then, in step S26, the read opinion judgment information of the multiple listeners is aggregated, for example, as follows.

[0076] That is, the opinion aggregation processing unit 13C generates the opinion aggregation index Oall by dividing the sum of the listeners who input "agree" among the multiple listeners participating in the conference by the total number of listeners. For example, if there are 100 listeners currently participating in the conference, and among these listeners, 60 listeners indicated "agree" (numeric value "1") and 40 listeners indicated "disagree" (numeric value "-1"), the opinion aggregation index Oall is Oall = (1 x 60 - 1 x 40) / 100 =0.2 It becomes.

[0077] Note that, as the calculation formula for the above opinion aggregation index Oall, any calculation formula may be used as long as it takes into account the proportion of opinions within the range that satisfies the condition of -1 ≦ Aall ≦ 1.

[0078] (2-5) Generation Process of Temperature Presentation Control Information Subsequently, under the control of the temperature presentation control information generation processing unit 15C, the control unit 1C of the server device SV generates temperature presentation control information as follows in step S27 based on the calculated opinion aggregation index Oall.

[0079] That is, in this example, it is assumed that when there are many listeners who show "approval" among the listeners participating in the meeting, the temperature is raised, and when there are many listeners who express "opposition", the temperature is lowered for temperature presentation. In this case, assuming the reference temperature is Tbase, the maximum change temperature when warming is ΔThot, and the maximum change temperature when cooling is ΔTcold, the temperature presentation control information generation processing unit 15C When Oall > 0, T = Tbase + Oall × ΔThot When Oall < 0, T = Tbase - Oall × ΔTcold When Oall = 0, T = Tbase determines the presentation temperature as follows and uses this as the temperature presentation control information.

[0080] As a result, for example, assuming the reference temperature Tbase = 25°C, the maximum change temperature when warming ΔThot = 10°C, and the maximum change temperature when cooling ΔTcold = 5°C are set respectively. In this example, since Oall > 0, the presentation temperature T is T = 25 + 0.2 × 10 = 27°C becomes.

[0081] Note that the calculation formula for the presentation temperature T is not limited to the above example. For example, a constant k greater than 0 and less than or equal to 1 may be multiplied by Oall and ΔThot or ΔTcold. That is, When Oall > 0, Tbase + k × Oall × ΔThot When Oall < 0, Tbase - k×Oall ×ΔTcold However, k satisfies 0 < k ≦ 1, and in this example, k = 1 is assumed.

[0082] (2 - 6) Output of Presentation Control Information When the vibration presentation control information and the temperature presentation control information are generated as described above, the control unit 1C of the server device SV performs the following transmission process in step S28 under the control of the presentation control information transmission processing unit 16C.

[0083] That is, the presentation control information transmission processing unit 16C inserts the generated vibration presentation control information and temperature presentation control information into, for example, one packet in a state where they are associated with each other so as to present them simultaneously. Then, this packet is transmitted from the communication I / F unit 4C to the speaker terminal ET.

[0084] When the transmission of the presentation control information is completed, the control unit 1C of the server device SV determines in step S29, for example, whether an end operation of the online meeting has been performed by the organizer terminal. If the end operation is not detected and the meeting is in progress, the process returns to step S21, and the series of reaction presentation control processes described above in steps S21 to S28 are repeatedly executed. On the other hand, when the end of the meeting is detected, the communication link for the online meeting between the speaker terminal ET and each listener terminal UT1 to UTn is disconnected, and the device returns to the standby state.

[0085] (3) Reaction Presentation Processing in the Speaker Terminal ET FIG. 7 is a flowchart showing the processing procedure and processing content of the reaction presentation processing executed by the control unit 1B of the speaker terminal ET.

[0086] The control unit 1B of the speaker terminal ET monitors whether the speaker has performed an operation to participate in the online meeting in step S30. In this state, when the speaker performs an access operation to participate in the online meeting, the control unit 1B executes a procedure to form a conference communication link with the server device SV under the control of the application program for the online meeting. Thus, it becomes possible to participate in the online meeting hereafter.

[0087] (3-1) Reception process of presentation control information When the online meeting starts, the control unit 1B of the speaker terminal ET receives the presentation control information transmitted from the server device SV via the communication I / F unit 4B under the control of the presentation control information reception processing unit 11B while performing the transmission and reception processing of various data related to the normal online meeting. Then, the received presentation control information is temporarily stored in the presentation control information storage unit 31B.

[0088] (3-2) Generation and output of vibration drive signal When the above presentation control information is received, the control unit 1B of the speaker terminal ET first selectively reads the vibration presentation control information included in the presentation control information from the presentation control information storage unit 31B in step S32 under the control of the vibration drive signal generation processing unit 12B. Then, in step S33, the vibration drive signal generation processing unit 12B generates a vibration drive signal based on the read vibration presentation control information.

[0089] For example, if the vibration presentation control information is set to vibration "on", a vibration drive signal for generating vibration at a preset intensity is generated. Then, the vibration drive signal generation processing unit 12B outputs the generated vibration drive signal from the input / output I / F unit 5B to the vibration device 6 built in the mouse MU. As a result, the vibration device 6 vibrates for a certain period at the intensity specified by the vibration drive signal. That is, the mouse MU vibrates for a certain period at a predetermined intensity in response to the nodding motions of a plurality of aggregated listeners.

[0090] On the other hand, if the vibration presentation control information is set to vibration "off", the vibration drive signal generation processing unit 12B does not generate a vibration drive signal. Therefore, the mouse MU does not vibrate, and the speaker is not presented with the nodding motion of the listener.

[0091] (3-3) Generation and Output of Heat Generation Drive Signal The control unit 1B of the speaker terminal ET, in parallel with the generation and output processing of the vibration drive signal, selectively reads the temperature presentation control information included in the presentation control information from the presentation control information storage unit 31B under the control of the heat generation drive signal generation processing unit 13B in step S34. Then, the heat generation drive signal generation processing unit 13B generates a heat generation drive signal based on the read temperature presentation control information in step S35.

[0092] For example, if the presentation temperature T set in the temperature presentation control information designates warming, the heat generation drive signal generation processing unit 13B generates a heat generation drive signal for generating the presentation temperature T that designates the specified warming. Then, the generated heat generation drive signal is supplied to the heat generation device 7 of the mouse MU via the input / output I / F unit 5B. As a result, the heat generation device 7 generates heat at the temperature specified by the heat generation drive signal. Therefore, the speaker can recognize that the aggregation result of the opinions of each listener regarding his / her speech is generally "in favor" due to the heat generation of the mouse MU.

[0093] On the other hand, if the presentation temperature T set in the temperature presentation control information designates cooling, the heat generation drive signal generation processing unit 13B generates a heat generation drive signal for generating the presentation temperature T that designates the specified cooling. Then, the generated heat generation drive signal is supplied to the heat generation device 7 of the mouse MU via the input / output I / F unit 5B. As a result, the heat generation device 7 operates to cool the mouse MU to the temperature specified by the heat generation drive signal. Therefore, the speaker can recognize that the aggregation result of the opinions of each listener regarding his / her speech is generally "opposed" due to the cooling of the mouse MU.

[0094] The control unit 1B of the speaker terminal ET finally determines the end of participation in the online meeting in step S36. Then, if the end-of-participation operation is not performed and the speaker continues to participate in the meeting, the process returns to step S31 and a series of reaction presentation processes according to steps S31 to S35 are repeatedly executed. On the other hand, when the end-of-participation operation in the meeting is performed, a procedure for disconnecting the communication link for the meeting with the server device SV is executed, and the system returns to the standby state.

[0095] (Function and Effect) As described above, in the online meeting system according to one embodiment, the following operations are performed. That is, a plurality of listener terminals UT1 to UTn during an online meeting include the presence or absence of "nodding", which represents the reaction of the listener to the speaker's speech, and the judgment result of "approval or disapproval" input by the listener, in the reaction judgment information as action judgment information and opinion judgment information, respectively, and transmit them to the server device SV. In contrast, when the server device SV receives the reaction judgment information from the listener terminals UT1 to UTn, it aggregates the actions and opinions of a plurality of listeners during the meeting based on the action judgment information and opinion judgment information included in these reaction judgment information. Then, vibration presentation control information for representing the aggregation result of the above actions by the intensity of vibration is generated, and temperature presentation control information for representing the aggregation result of the above opinions by the level of temperature is generated, and each generated presentation control information is transmitted to the speaker terminal ET. The speaker terminal ET generates a vibration drive signal according to the received vibration presentation control information to vibrate the vibration device 6 of the mouse MU, and generates a heat generation drive signal according to the received temperature presentation control information to cause the heat generation device 7 of the mouse MU to perform a heat generation or cooling operation.

[0096] Therefore, the speaker can recognize in detail the reaction of the listener to his or her speech by the combination of the vibration and the heat generation or cooling operation presented in the mouse MU without relying on visual display, that is, without causing a new cognitive load.

[0097] FIG. 8 shows four types of reactions presented in one embodiment in a two-dimensional space consisting of vibration and temperature. As shown in FIG. 8, when vibration occurs and a heat generation operation is performed, "strong approval" is presented to the speaker. On the contrary, when vibration does not occur and a cooling operation is performed, "strong opposition" is presented to the speaker. Also, when vibration occurs but a cooling operation is performed, "partially understandable but opposed" is presented to the speaker, and when vibration does not occur but a heat generation operation is performed, "not convinced but agreed" is presented to the speaker.

[0098] [Other Embodiments] (1) In the above embodiment, the case where the presence or absence of the listener's nodding is represented by the presence or absence of vibration has been described as an example. However, the presence or absence of the above nodding may be represented by the strength of vibration, or may be represented by varying the generation time length or generation pattern (for example, intermittent pattern) of vibration. Furthermore, not limited to the presence or absence of the listener's nodding, the size of the nod may be determined in multiple stages, and the determined size of the nod may be presented by changing the intensity of vibration in multiple stages.

[0099] (2) In the above embodiment, the case where the opinions (approval and opposition) input by the listener are represented by the high and low temperatures (heat generation and cooling) has been described as an example. However, the opinions (approval and opposition) may be represented by the presence or absence of heat generation or the presence or absence of cooling. Furthermore, instead of a binary choice of approval and opposition for the listener's opinion, for example, it may be determined in more stages such as "strongly approve", "partially approve", "strongly oppose", "partially oppose", and the determined stage of the opinion may be presented by varying the high and low temperatures in multiple stages.

[0100] (3) In the above embodiment, the case where the vibration device 6 and the heat generation device 7 are provided in the mouse MU has been described as an example. However, in addition to that, for example, a vibration device and a heat generation device may be provided in a wearable terminal worn by the speaker, a wristwatch, a bracelet, glasses, or clothing.

[0101] (4) In the above-described embodiment, the case where the server device SV executes the processes of aggregating actions and opinions and generating presentation control information for vibration and temperature based on the aggregated actions and opinions was described as an example. However, the present invention is not limited thereto. For example, the reaction determination information transmitted from a plurality of listener terminals UT1 to UTn may be directly transmitted to the speaker terminal ET via a network, and the speaker terminal ET may execute each process executed by the server device SV.

[0102] (5) In the above-described embodiment, an online conference system was described as an example. However, the present invention may be implemented, for example, in an online lecture, a lecture, or an online exhibition. Furthermore, the present invention may be implemented not only online but also in an offline conference, a lecture, an exhibition, etc. held in person.

[0103] (6) In addition, the configurations, processing functions, processing procedures, processing contents, etc. of the listener terminal, the speaker terminal, and the server device can also be variously modified and implemented without departing from the gist of the present invention.

[0104] The embodiments of the present invention have been described in detail above, but the description up to this point is merely an exemplification of the present invention in every respect. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted.

[0105] In short, the present invention is not limited to the above-described embodiments as they are, and at the implementation stage, the components can be modified and embodied without departing from the gist thereof. Also, various inventions can be formed by appropriately combining a plurality of components disclosed in the above-described embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

Description of Reference Numerals

[0106] SV... Server device UT1 to UTn... listener terminals ET... speaker terminal NW... network 1A, 1B, 1C... control units 2A, 2B, 2C... program memory units 3A, 3B, 3C... data memory units 4A, 4B, 4C... communication I / F units 5A, 5B... input / output I / F units 11A... action determination processing unit 12A... opinion determination processing unit 13A... reaction determination information transmission processing unit 11B... presentation control information reception processing unit 12B... vibration drive signal generation processing unit 13B... heat generation drive signal generation processing unit 11C... reaction determination information reception processing unit 12C... action aggregation processing unit 13C... opinion aggregation processing unit 14C... vibration presentation control information generation processing unit 15C... temperature presentation control information generation processing unit 16C... presentation control information transmission processing unit 31A... action determination information memory unit 32A... opinion determination information memory unit 31B... presentation control information memory unit 31C... reaction determination information memory unit

Claims

1. An acquisition processing unit that acquires, as information representing the reaction states of the plurality of listeners, information representing body movements reflecting the reactions of the plurality of listeners to the speech content of the speaker and information representing opinions input by the listeners; A reaction aggregation information generation processing unit that generates first aggregation information aggregating the body movements of the plurality of listeners based on the acquired information representing the body movements, and generates second aggregation information aggregating the opinions of the plurality of listeners based on the acquired information representing the opinions; An output processing unit that generates vibration control information representing the body movements of the plurality of listeners in terms of the intensity of vibration based on the generated first aggregation information, generates temperature control information representing the opinions of the plurality of listeners in terms of the height of temperature based on the generated second aggregation information, and outputs the generated vibration control information and temperature control information for presentation to the speaker A reaction presentation control device comprising the above.

2. The acquisition processing unit acquires the information representing the reaction states of the plurality of listeners via a network from a plurality of listener terminals used by the plurality of listeners respectively. The output processing unit transmits the vibration control information and the temperature control information via the network to a speaker terminal used by the speaker. The reaction presentation control device according to Claim 1.

3. The reaction aggregation information generation processing unit generates the first aggregation information representing the degree of nodding of the listeners based on the information representing the degree of nodding of the listeners, The output processing unit generates and outputs the vibration control information obtained by converting the degree of nodding into the intensity of vibration. The reaction presentation control device according to Claim 1.

4. The reaction aggregation information generation processing unit generates the second aggregation information representing the degree of approval or disapproval of the listeners based on the information representing the degree of approval or disapproval of the listeners, The output processing unit generates and outputs the temperature control information obtained by converting the degree of approval or disapproval into the height or low temperature. The reaction presentation control device according to Claim 1.

5. The output processing unit generates and outputs, as the temperature control information, information for performing control to warm to a first temperature higher than a preset reference temperature or cool to a second temperature lower than the reference temperature according to the degree of approval or disapproval. The reaction presentation control device according to Claim 4.

6. A reaction presentation control method executed by an information processing device, A process of respectively obtaining information representing body movements reflecting the reactions of a plurality of listeners to the speech content of a speaker and information representing opinions input by the listeners as information representing the reaction states of the plurality of listeners; A process of generating first aggregated information obtained by aggregating the body movements of the plurality of listeners based on the obtained information representing the body movements, and generating second aggregated information obtained by aggregating the opinions of the plurality of listeners based on the obtained information representing the opinions; A process of generating vibration control information representing the body movements of the plurality of listeners in terms of the intensity of vibration based on the generated first aggregated information, and generating temperature control information representing the opinions of the plurality of listeners in terms of the height of temperature based on the generated second aggregated information, and outputting the generated vibration control information and the temperature control information for presentation to the speaker; A reaction presentation control method comprising the above steps.

7. A program for causing a processor included in the reaction presentation control device to execute the processes performed by each processing unit included in the reaction presentation control device according to any one of Claims 1 to 5.

Images