Electronic conference system, electronic conference method, and terminal device

Abstract

To smoothly advance an electronic conference.SOLUTION: An electronic conference system for using a plurality of terminal devices connected to each other through a network to perform an electronic conference between a plurality of persons is provided with emotion information acquisition part for acquiring emotion information corresponding to an emotion of a first person (PS[i]), transmits the emotion information of the first person to a terminal device (10[g]) of a second person through the network, and makes the terminal device of the second person to notify the second person (PS[g]) of the transmitted emotion information of the first person.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an electronic conference system, an electronic conference method, and a terminal device. [Background technology]

[0002] Electronic conferences (Web conferences) are widely used in which multiple people hold a conference using multiple terminal devices without meeting in the same room. In a typical electronic conference system (Web conference system), the camera image output condition can be switched on and off. When the camera image output condition is set to on in a terminal device of a certain person, a camera image including the face image of that person is displayed on other terminal devices, but when the camera image output condition is set to off, such display is not performed. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-4253 A Summary of the Invention [Problem to be solved by the invention]

[0004] Not only in electronic conferences, but in many other conferences, participants often communicate with each other while observing each other's facial expressions, which is believed to help the conference proceed smoothly. However, in electronic conferences, camera image output conditions are often turned off due to concerns about communication data volume and privacy, which can result in the smooth running of the conference being compromised.

[0005] An object of the present invention is to provide an electronic conference system, an electronic conference method, and a terminal device that contribute to the smooth progress of an electronic conference. [Means for solving the problem]

[0006] The electronic conferencing system of the present invention is an electronic conferencing system for holding an electronic conference between a plurality of people using a plurality of terminal devices connected to each other via a network, and is configured (first configuration) to include an emotion information acquisition unit that acquires emotion information corresponding to the emotion of a first person who is any one of the plurality of people, transmit the emotion information of the first person via the network to a terminal device of a second person who is included in the plurality of people and different from the first person, and notify the second person of the transmitted emotion information of the first person at the terminal device of the second person.

[0007] In the electronic conferencing system according to the first configuration, the emotion information acquisition unit may be configured to estimate the emotion of the first person based on emotion estimation information of the first person that is influenced by the emotion of the first person, and acquire emotion information of the first person based on the estimation result (second configuration).

[0008] In the electronic conferencing system according to the first configuration, the emotion information acquisition unit may be configured (third configuration) to acquire emotion information of the first person based on the content of an emotion designation operation from the first person received by a terminal device of the first person.

[0009] In the electronic conference system according to the first configuration, the emotion information acquisition unit may be capable of executing a first process of estimating an emotion of the first person based on emotion estimation information of the first person that is influenced by the emotion of the first person, and generating first candidate information relating to the emotion of the first person based on the estimation result, and a second process of generating second candidate information relating to the emotion of the first person based on the content of an emotion designation operation when an emotion designation operation is received from the first person on a terminal device of the first person, and may be configured to set the second candidate information generated by the second process as the emotion information of the first person when the emotion designation operation is performed, while setting the first candidate information generated by the first process as the emotion information of the first person when the emotion designation operation is not performed (fourth configuration).

[0010] In the electronic conference system according to the fourth configuration, the electronic conference system may further include a notification unit that notifies the first person of the input of the emotion designation operation when the estimation result by the first process satisfies a predetermined condition (fifth configuration).

[0011] In the electronic conferencing system according to any one of the first to fifth configurations above, the system may further include an image processing unit that processes a registered image associated with the first person in accordance with emotional information of the first person, and the processed registered image may be displayed on a terminal device of the second person (sixth configuration).

[0012] In the electronic conferencing system related to the sixth aspect above, the image processing unit may be configured to generate a moving image corresponding to emotional information of the first person and speech information of the first person based on the registered image associated with the first person, and the moving image may be displayed on a terminal device of the second person (seventh configuration).

[0013] In the electronic conference system according to any one of the first to seventh configurations above, the electronic conference system may further include a record management unit that records conference data relating to the electronic conference on a recording medium, the plurality of persons consisting of first to nth persons, n being an integer equal to or greater than 2, the emotion information acquisition unit acquires emotion information of the first to nth persons by acquiring emotion information corresponding to the emotion of each of the first to nth persons, and the record management unit may include emotion information of the speaker for each utterance of each person in the conference data (eighth configuration).

[0014] In the electronic conference system according to any one of the first to seventh configurations above, the system may further include a record management unit that records conference data relating to the electronic conference on a recording medium, the plurality of persons consisting of first to nth persons, n being an integer equal to or greater than 2, the emotion information acquisition unit acquires emotion information of the first to nth persons by acquiring emotion information corresponding to the emotion of each of the first to nth persons, and the record management unit may include the emotion information of the first to nth persons in the conference data (ninth configuration).

[0015] In the electronic conferencing system according to the ninth configuration, a statistical processing unit is provided that generates emotion statistical data by taking statistics on emotion information of the first to nth persons, and the record management unit may be configured to include the emotion statistical data in the conference data (tenth configuration).

[0016] The electronic conference system according to any one of the first to seventh configurations may be configured as an eleventh configuration, further comprising a statistical processing unit and a statistical notification unit, wherein the plurality of persons consists of first to nth persons, n being an integer equal to or greater than 2, the emotion information acquisition unit acquires emotion information of the first to nth persons by acquiring emotion information corresponding to the emotion for each of the first to nth persons, the statistical processing unit generates emotion statistical data by taking statistics on the emotion information of the first to nth persons, and the statistical notification unit causes any one or more of the plurality of terminal devices to make a notification corresponding to the emotion statistical data (eleventh configuration).

[0017] A conferencing method according to the present invention is an electronic conferencing method for holding an electronic conferencing among a plurality of people using a plurality of terminal devices connected to each other through a network, the conferencing method comprising: an emotion information acquisition step of acquiring emotion information corresponding to the emotion of a first person who is any one of the plurality of people; a transmission step of transmitting the emotion information of the first person through the network to a terminal device of a second person who is included in the plurality of people and different from the first person; and a notification step of notifying the second person of the transmitted emotion information of the first person at the terminal device of the second person (twelfth configuration).

[0018] A program according to the present invention has a configuration (thirteenth configuration) for causing a processor to realize the conferencing method according to the twelfth configuration.

[0019] A terminal device according to the present invention is a terminal device used for holding an electronic conference among a plurality of persons via a network, and is configured to include an emotion information acquisition unit that acquires emotion information of a first person who is included in the plurality of persons and is a user of the terminal device, and a communication processing unit that transmits emotion information of the first person to the network so that a terminal device of a second person who is included in the plurality of persons and is different from the first person can be notified of the emotion information of the first person (fourteenth configuration). Effect of the Invention

[0020] According to the present invention, it is possible to provide an electronic conference system, method, and terminal device that contribute to the smooth progress of an electronic conference. [Brief description of the drawings]

[0021] [Figure 1] 1 is an overall configuration diagram of a conference system (electronic conference system) according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating an internal configuration of a terminal device according to an embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating an internal configuration of a server device according to an embodiment of the present invention. [Figure 4] 1 is a diagram illustrating an overall configuration of a conference system according to an embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram illustrating an internal configuration of a terminal device according to an embodiment of the present invention. [Figure 6] FIG. 10 is a diagram showing how emotion information of a target person is transmitted to a plurality of recipients in the first example pertaining to an embodiment of the present invention. [Figure 7] FIG. 1 is a diagram showing a state of information transmission between a plurality of terminal devices according to a first embodiment of the present invention. [Figure 8] 4 is a flowchart showing an operation of the conference system relating to emotion information in the first example of the embodiment of the present invention. [Figure 9] FIG. 1 is a diagram showing a state in which an emotion information acquisition unit is provided in a terminal device according to a first example pertaining to an embodiment of the present invention. [Figure 10]FIG. 10 is a diagram showing a state in which a plurality of frames are set during an electronic conference according to a first embodiment of the present invention. [Figure 11] FIG. 13 is a conceptual diagram of an operation for acquiring emotion information according to a fourth embodiment of the present invention. [Figure 12] FIG. 13 is a diagram showing a state in which a manual input suggestion notifying unit is provided in a terminal device according to a fifth embodiment of the present invention. [Figure 13] 13 is a flowchart of emotion information setting and acquisition operations in a fifth example belonging to an embodiment of the present invention. [Figure 14] FIG. 13 relates to a sixth embodiment of the present invention, and is a diagram showing a state in which an image processing unit is provided in a terminal device and registered image group data and processing data are stored. [Figure 15] FIG. 13 is a diagram showing a registered image for each participant of a conference according to a sixth embodiment of the present invention. [Figure 16] FIG. 13 is a diagram showing an eighth embodiment of the present invention, in which a recording management section, a recording medium, a playback management section, a statistical processing section, and a statistical notification section are provided in a server device. [Figure 17] FIG. 13 is a diagram showing a structure of conference data according to an eighth embodiment of the present invention. [Figure 18] FIG. 13 is a diagram showing individual voice data and attribute data according to an eighth example belonging to an embodiment of the present invention. [Figure 19] FIG. 13 is a structural diagram of additional data according to a ninth embodiment of the present invention. [Figure 20] FIG. 13 is a diagram for explaining one aspect of the operation according to the ninth example belonging to the embodiment of the present invention. [Figure 21] FIG. 13 is a diagram for explaining one aspect of the operation according to the ninth example belonging to the embodiment of the present invention. [Figure 22] FIG. 13 is a diagram showing an example of a statistics notification image according to a ninth embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Hereinafter, examples of the embodiments of the present invention will be described in detail with reference to the drawings. In each of the drawings, the same parts are given the same reference numerals, and duplicated descriptions of the same parts are omitted as a rule. In this specification, for the sake of simplicity, a symbol or a code referring to information, a signal, a physical quantity, a member, or the like may be written, and the name of the information, signal, physical quantity, member, or the like corresponding to the symbol or code may be omitted or abbreviated. For example, the emotion information acquisition unit referred to by "20" described later (see FIG. 9(a)) may be written as emotion information acquisition unit 20 or abbreviated as acquisition unit 20, but they all refer to the same thing.

[0023] FIG. 1 shows the overall configuration of the system SYS according to this embodiment. The system SYS can be used for various purposes, but in this embodiment, it is assumed that the system SYS is used for holding a conference (conversation) between multiple people PS. In this case, the system SYS functions as a conference system (electronic conference system), and hereinafter, the system SYS is referred to as the conference system SYS. The conference in this embodiment is an electronic conference, and in this embodiment, the conference and the electronic conference refer to the same thing. In an electronic conference, a conversation between multiple people PS is realized through the transmission and reception of signals (including at least audio signals) between multiple terminal devices, without the multiple people PS meeting together. An electronic conference is also generally referred to as a Web conference. Each person PS corresponds to a participant in the electronic conference.

[0024] The conference system SYS is configured with a plurality of terminal devices 10 and a server device 30. Each terminal device 10 is assigned to one of the plurality of persons PS. Each terminal device 10 and server device 30 are connected to a network 50 including the Internet. In other words, the plurality of terminal devices 10 and server device 30 are connected to each other through the network 50. Each of the plurality of terminal devices 10 and server device 30 is capable of two-way communication with any device connected to the network 50.

[0025] Each terminal device 10 is an electronic device such as a personal computer, a tablet, or a smartphone. The multiple terminal devices 10 incorporated in the conference system SYS may be the same type of electronic device, or may be different types of electronic devices. However, in the following, it is considered that the multiple terminal devices 10 have the same configuration.

[0026] FIG. 2 shows the internal configuration of one terminal device 10. The terminal device 10 includes a communication processing unit 11, a memory unit 12, a terminal control unit 13, a display unit 14, an operation unit 15, a camera unit 16, a microphone 17, a speaker 18, and a sensor unit 19. The following description of the communication processing unit 11, the memory unit 12, the terminal control unit 13, the display unit 14, the operation unit 15, the camera unit 16, the microphone 17, the speaker 18, and the sensor unit 19 applies to each terminal device 10. In describing the internal configuration of the terminal device 10, a terminal device 10 that has been focused on is referred to as a focused terminal device 10 as necessary. Also, a person PS who is a user of the focused terminal device 10 may be referred to as a focused person PS. The focused terminal device 10 is a terminal device 10 that has been assigned to the focused person PS (corresponding to the focused person PS) in the conference system SYS. Each terminal device 10 incorporated in the conference system SYS can be a focused terminal device 10.

[0027] The communication processing unit 11 performs two-way communication with the other device through the network 50. In the attention terminal device 10, two-way communication includes a transmission operation of outputting any information to the network 50 and transmitting the information to the other device, and a reception operation of receiving any information output by the other device to the network 50. The other device for the attention terminal device 10 is an arbitrary device connected to the network 50 and other than the attention terminal device 10, and includes at least the server device 30. The communication processing unit 11 in the attention terminal device 10 can perform two-way communication with each terminal device 10 other than the attention terminal device 10 via the network 50 and the server device 30. The communication processing unit 11 in the attention terminal device 10 may be capable of realizing two-way communication with each terminal device 10 other than the attention terminal device 10 without going through the server device 30.

[0028] The storage unit 12 is composed of a magnetic disk, a semiconductor memory, or a combination thereof, and stores various programs and data. The programs stored in the non-volatile memory in the storage unit 12 include a terminal program PG1.

[0029] The terminal control unit 13 is composed of a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), etc., which are arithmetic processing devices, and comprehensively controls the operation of each part within the terminal device 10, and realizes various functions by executing programs stored in the memory unit 12 in the arithmetic processing device within the terminal control unit 13.

[0030] The display unit 14 is composed of a liquid crystal display panel or an organic electroluminescence panel, etc., and displays any image (in other words, a video) under the control of the terminal control unit 13. The person of interest PS can visually recognize the display content of the display unit 14 of the attention terminal device 10. In the following description, the display on the terminal device 10 means the display on the display unit 14 provided in the terminal device 10.

[0031] The operation unit 15 receives various operations from an operator. The operator of the target terminal device 10 is the target person PS. When the terminal device 10 is provided with a touch panel, the display unit 14 and the operation unit 15 may be configured by the touch panel.

[0032] The camera unit 16 captures images within its own field of view and outputs an image signal representing the captured image. The camera unit 16 has the function of capturing still images and moving images. An image represented by the output image signal from the camera unit 16 (i.e., an image captured by the camera unit 16) is referred to as a camera image. It is assumed that the field of view of the camera unit 16 of the target terminal device 10 includes the entire body or part of the target person PS. It is assumed that at least the face of the target person PS is included in the field of view of the camera unit 16 of the target terminal device 10. Therefore, the camera image representing the captured image of the camera unit 16 of the target terminal device 10 includes at least the facial image of the target person PS.

[0033] The microphone 17 converts its surrounding sounds into an audio signal and outputs it. In the conference system SYS, the main purpose of the microphone 17 in the target terminal device 10 is to pick up the speech content of the target person PS. Therefore, the microphone 17 in the target terminal device 10 picks up the speech content of the target person PS, converts the picked up sound into an audio signal, and outputs it.

[0034] The speaker 18 outputs the audio signal input thereto as sound (as sound waves) under the control of the terminal control unit 13. The person of interest PS can hear the output sound of the speaker 18 of the terminal device 10 of interest. In the following description, the output of sound or audio from the terminal device 10 means the output of sound or audio from the speaker 18 provided in the terminal device 10.

[0035] The sensor unit 19 is a sensor device that obtains information that cannot be obtained or is difficult to obtain by the camera unit 16 and the microphone 17. For example, the sensor unit 19 in the target terminal device 10 can obtain biometric information of the target person PS. However, the sensor unit 19 may not be provided in the terminal device 10.

[0036] 3 shows the internal configuration of the server device 30. The server device 30 includes a communication processing unit 31, a storage unit 32, and a server control unit 33. The server device 30 may include all or some of the display unit, operation unit, camera unit, operation unit, microphone, speaker, and sensor unit similar to those provided in the terminal device 10, but here, attention is focused only on the communication processing unit 31, the storage unit 32, and the server control unit 33, which play important roles in the operation of the conference system SYS. The server device SV may be configured with one or more computer devices. The conference system SYS may be configured using so-called cloud computing.

[0037] The communication processing unit 31 performs two-way communication with a counterpart device through the network 50. In the server device 30, two-way communication includes a transmission operation of outputting any information to the network 50 and transmitting the information to the counterpart device, and a reception operation of receiving any information output by the counterpart device to the network 50. The counterpart device for the server device 30 is any device connected to the network 50. Here, each of the multiple terminal devices 10 corresponds to a counterpart device for the server device 30.

[0038] The storage unit 32 is composed of a magnetic disk, a semiconductor memory, or a combination thereof, and stores and keeps track of various programs and data. The programs stored in the non-volatile memory in the storage unit 32 include a server program PG2.

[0039] The server control unit 33 is composed of a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), etc., which are arithmetic processing devices, and controls the overall operation of each part within the server device 30, and realizes various functions by executing programs stored in the memory unit 32 in the arithmetic processing device within the server control unit 33.

[0040] 4, for the sake of concrete explanation, it is assumed that a total of n terminal devices 10 are incorporated in the conference system SYS, and among the n terminal devices 10, the first to nth terminal devices 10 are referred to as terminal devices 10[1] to 10[n]. Also, a person PS to which a terminal device 10[i] is assigned is referred to as a person PS[i]. n represents any integer equal to or greater than 2, and i represents any natural number equal to or less than n. Furthermore, as shown in FIG. 5, the communication processing unit 11, memory unit 12, terminal control unit 13, display unit 14, operation unit 15, camera unit 16, microphone 17, speaker 18, and sensor unit 19 in terminal device 10[i] may be referred to as the communication processing unit 11[i], memory unit 12[i], terminal control unit 13[i], display unit 14[i], operation unit 15[i], camera unit 16[i], microphone 17[i], speaker 18[i], and sensor unit 19[i], respectively.

[0041] In the terminal device 10[i], two-way communication between the terminal device 10[i] and the other device is performed through the network 50 using the communication processing unit 11[i], but in the following, for the sake of simplicity, the description of the communication processing unit 11[i] and the network 50 may be omitted. The same applies to two-way communication between the server device 30 and the other device of the server device 30.

[0042] In each of the terminal devices 10[1] to 10[n], the terminal control unit 13 executes the terminal program PG1, and in the server device 30, the server control unit 33 executes the server program PG2, thereby holding an electronic conference.

[0043] The conference system SYS can execute a voice transmission process in an electronic conference, in which the speech content of any person PS is output as voice from the terminal device 10 assigned to the other person PS.

[0044] In the voice transmission process when the person PS[1] speaks, the speech of the person PS[1] is collected by the microphone 17[1] and converted into a voice signal, and the voice signal representing the speech of the person PS[1] is transmitted to the terminal devices 10[2] to 10[n] through the network 50 and the server device 30. Then, in each of the terminal devices 10[2] to 10[n], the voice signal representing the speech of the person PS[1] is output. Similarly, in the voice transmission process when the person PS[2] speaks, the speech of the person PS[2] is collected by the microphone 17[2] and converted into a voice signal, and the voice signal representing the speech of the person PS[2] is transmitted to the terminal devices 10[1] and 10[3] to 10[n] through the network 50 and the server device 30. Then, in each of the terminal devices 10[1] and 10[3] to 10[n], a voice signal representing the contents of the speech of the person PS[2] is output as a voice. The voice transmission process when the person PS[3] etc. speaks is also performed in the same manner.

[0045] To explain in detail with a focus on the operation of the device, the voice transmission process when the person PS[i] speaks is performed by a process PR A1 And the PR for that A1 The terminal device 10[i] transmits the voice signal obtained in the process PR to the server device 30. A2 And, Processing PR A2 A process PR in which the server device 30 transmits the voice signal received by the server device 30 to each terminal device 10 other than the terminal device 10[i] via the A3 And, Processing PR A3 A process PR for outputting the voice signal received by each terminal device 10 other than the terminal device 10[i] as sound from each terminal device 10 other than the terminal device 10[i] via the A4 This can be achieved by:

[0046] The conference system SYS can also execute a camera image transmission process in an electronic conference. In the camera image transmission process, a camera image showing the result of photographing each person PS is displayed on the terminal device 10 assigned to the other person PS. In the camera image transmission process, the camera image is acquired and updated in real time (however, there is a delay associated with the transmission).

[0047] In the camera image transmission process for person PS[1], the image signal of the camera image obtained by shooting with the camera unit 16[1] is transmitted to the terminal devices 10[2]-10[n] via the network 50 and the server device 30, and the transmitted camera image is displayed on the display unit 14 of each of the terminal devices 10[2]-10[n]. At this time, the camera image received by each of the terminal devices 10[2]-10[n] includes the shooting result of person PS[1] (the facial image of person PS[1]), so that persons PS[2]-PS[n] can proceed with the conference while watching the facial expression of person PS[1]. Similarly, in the camera image transmission process for person PS[2], the image signal of the camera image obtained by shooting with the camera unit 16[2] is transmitted to the terminal devices 10[1] and 10[3] to 10[n] via the network 50 and the server device 30, and the transmitted camera image is displayed on the display unit 14 in each of the terminal devices 10[1] and 10[3] to 10[n]. At this time, the camera image received by each of the terminal devices 10[1] and 10[3] to 10[n] includes the shooting result of person PS[2] (face image of person PS[2]), so that person PS[1] and PS[3] to PS[n] can proceed with the conference while watching the facial expression of person PS[2]. The camera image transmission process for person PS[3] etc. is similar.

[0048] To explain in detail with a focus on the operation of the device, the camera image transmission process for the person PS[i] is a process PR V1 And the PR for that V1 The terminal device 10[i] transmits the image signal obtained in the process PR to the server device 30. V2 And, Processing PR V2 A process PR in which the server device 30 transmits the image signal received by the server device 30 to each terminal device 10 other than the terminal device 10[i] via the V3 And, Processing PR V3 A process PR for displaying the image signal received by each terminal device 10 other than the terminal device 10[i] as an image on each terminal device 10 other than the terminal device 10[i] via the V4 This can be achieved by:

[0049] However, each person PS can specify whether or not to execute the camera image transmission process by switching the camera image output conditions on / off. That is, the person PS[i] can freely set the camera image output conditions of the terminal device 10[i] to on or off by inputting a predetermined operation to the operation unit 15[i].

[0050] When the camera image output condition in the terminal device 10[1] is set to ON, the camera image transmission process for the person PS[1] described above is executed, and the camera image captured by the camera unit 16[1] is displayed on the terminal devices 10[2] to 10[n]. When the camera image output condition in the terminal device 10[1] is set to OFF, the camera image transmission process for the person PS[1] described above is not executed, and as a result, the camera image captured by the camera unit 16[1] is not displayed on the terminal devices 10[2] to 10[n]. Similarly, when the camera image output condition in the terminal device 10[2] is set to ON, the camera image transmission process for the person PS[2] described above is executed, and the camera image captured by the camera unit 16[2] is displayed on the terminal devices 10[1] and 10[3] to 10[n]. When the camera image output condition in the terminal device 10[2] is set to off, the camera image transmission process to the person PS[2] described above is not executed, and as a result, the camera image captured by the camera unit 16[2] is not displayed on the terminal devices 10[1] and 10[3] to 10[n]. The same applies to the camera image output condition in the terminal device 10[3] etc.

[0051] Incidentally, it is believed that in many cases, not only electronic conferences, but in any conference, a conference can proceed smoothly by proceeding with the conversation while observing the facial expressions of the other party. However, in electronic conferences, the camera image output conditions are often set to off from the viewpoint of communication data volume, privacy, etc., and as a result, the smoothness of the conference may be impaired. Here, the inventors have focused on the fact that observing the facial expressions of the other party is similar to sensing the other party's emotions, and have developed a technology that allows participants in a conference to inform each other of their emotions. By utilizing this technology, it is possible to sense the emotions of the other party even if the camera image output conditions are set to off (i.e., the same effect as observing the other party's facial expressions can be obtained), and it is believed that the conference (electronic conference) can proceed smoothly.

[0052] In the following, specific operation examples (particularly operation examples related to the notification of emotions) and application techniques of the conference system SYS will be described in several examples. The matters described above in this embodiment are applied to each of the following examples unless otherwise specified and unless there is a contradiction. In the cases where there are matters in each example that contradict the matters described above, the description in each example may take precedence. Furthermore, unless there is a contradiction, matters described in any of the following examples can also be applied to any of the other examples (i.e., any two or more of the examples can be combined).

[0053] Unless otherwise specified, the processes and operations shown in each of the following embodiments are assumed to be those performed during the period when an electronic conference is held in the conference system SYS (excluding the eighth and ninth embodiments, which relate to recording or playing back conference data). In the following, the period when an electronic conference is held in the conference system SYS may be referred to as the electronic conference period. In addition, in each of the following embodiments, unless otherwise specified, it is assumed that the above-mentioned audio transmission process is executed, but the camera image output condition is set to off in each of the terminal devices 10[1] to 10[n], and therefore no camera image transmission process is executed in the conference system SYS.

[0054] <<First Example>> A first embodiment will be described. As shown in Fig. 6, the conference system SYS according to the first embodiment has a function of acquiring emotion information according to the emotion of a target person in an electronic conference, and transmitting the acquired emotion information to each terminal device 10 assigned to each recipient, thereby notifying each recipient of the emotion of the target person. When the person PS[1] is the target person, the other people PS[2] to PS[n] correspond to the recipients, and when the person PS[2] is the target person, the other people PS[1] and PS[3] to PS[n] correspond to the recipients. The same is true when the person PS[3] etc. is the target person, and each of the people PS[1] to PS[n] becomes a target person, and the emotion information of each person is notified to each other.

[0055] FIG. 7 shows the state of information transmission according to the first embodiment. FIG. 7 shows a schematic diagram of the state of information transmission when the person PS[i] functions as a target person and the person PS[g] functions as a recipient person. The terminal device 10[g] represents any terminal device 10 different from the terminal device 10[i] among the terminal devices 10[1] to 10[n], and the person PS[g] represents any person PS different from the person PS[i] among the persons PS[1] to PS[n]. Therefore, g represents any natural number different from i and equal to or less than n. As described above, the above-mentioned voice transmission process is executed in the first embodiment, and therefore the speech content of the person PS[i] is transmitted to the terminal device 10[g] and output as voice at the terminal device 10[g], and the speech content of the person PS[g] is transmitted to the terminal device 10[i] and output as voice at the terminal device 10[i] (only the former transmission state is shown in FIG. 7).

[0056] 8 shows an operation flowchart of the conference system SYS related to acquisition, transmission, and notification of emotion information. The conference system SYS sequentially executes the processes of steps S10, S20, and S30. A series of processes consisting of the processes of steps S10, S20, and S30 may be repeatedly executed at a predetermined cycle.

[0057] In step S10, terminal device 10[i] executes emotion information acquisition processing to acquire emotion information corresponding to the emotion of person PS[i].

[0058] In the following step S20, an inter-device transmission process is executed to transmit the emotion information acquired in step S10 to each terminal device 10 other than terminal device 10[i]. This transmission is performed via server device 30. That is, the inter-device transmission process is made up of a process in which terminal device 10[i] (more specifically, communication processing unit 11[i]) transmits the emotion information acquired in step S10 toward network 50 (in other words, a process of transmitting to server device 30 via network 50), a process in which server device 30 transmits the emotion information received from terminal device 10[i] from server device 30 to each terminal device 10 other than terminal device 10[i], and a process in which each terminal device 10 other than terminal device 10[i] receives the emotion information transmitted from server device 30.

[0059] In step S30 following step S20, each terminal device 10 other than the terminal device 10[i] executes an emotion information notification process to notify its own user of the emotion information received from the server device 30. By performing the emotion information notification process in the terminal device 10[g], the terminal device 10[g] can notify the emotion information of the person PS[i] to the person PS[g] who is its own user. The notification in step S30 may be any notification that appeals to the perception of the person PS[g], but since a voice notification may cause an interruption to the progress of the conference, the notification in step S30 is preferably a visual notification. That is, the notification in step S30 is preferably a notification that displays the received emotion information (the emotion information of the person PS[i] acquired in step S10) on the display unit 14[g]. Hereinafter, the emotion information notification process is considered to correspond to the display of emotion information. Specifically, in step S30, an emotion notification image corresponding to the received emotion information (the emotion information of the person PS[i] acquired in step S10) is displayed on the display unit 14[g].

[0060] 7 shows a state in which emotion information corresponding to the emotion of person PS[i] is transmitted to the terminal device 10[g] of person PS[g] and notified to person PS[g], but in reality, in parallel with this, emotion information corresponding to the emotion of person PS[g] is transmitted to the terminal device 10[i] of person PS[i] and notified to person PS[i]. That is, the processing group consisting of the above-mentioned steps S10, S20, and S30 is executed for each of persons PS[1] to PS[n] after regarding each of persons PS[1] to PS[n] as a target person individually. For this reason, emotional information of person PS[1] is transmitted to terminal devices 10[2] to 10[n] and notified to persons PS[2] to PS[n], emotional information of person PS[2] is transmitted to terminal devices 10[1] and 10[3] to 10[n] and notified to persons PS[1] and PS[3] to PS[n], ..., emotional information of person PS[n] is transmitted to terminal devices 10[1] to 10[n-1] and notified to persons PS[1] to PS[n-1].

[0061] 9(a) and 9(b), each terminal device 10 is provided with an emotion information acquisition unit 20 for acquiring emotion information of a corresponding person, and the emotion information acquisition unit 20 provided in terminal device 10[i] will be particularly referred to with the symbol "20[i]". Emotion information acquisition unit 20[1] executes emotion information acquisition processing to acquire emotion information corresponding to the emotion of person PS[1], and emotion information acquisition unit 20[2] executes emotion information acquisition processing to acquire emotion information corresponding to the emotion of person PS[2]. The same is true for emotion information acquisition unit 20[3], etc. Acquisition unit 20[i] is a functional block realized by terminal control unit 13[i] executing terminal program PG1. The explanation for acquisition unit 20[i] applies to each of acquisition units 20[1] to 20[n].

[0062] The emotion information acquisition process executed by the acquisition unit 20[i] includes at least an emotion classification process for determining which of f types of classified emotions the emotion of the person PS[i] is classified into, and the classification result is included in the emotion information of the person PS[i]. f is an arbitrary integer of 2 or more. The f types of classified emotions include, for example, a total of 10 types of classified emotions, including "joy", "acceptance", "fear", "surprise", "sadness", "disgust", "anger", "expectation", "calmness" and "others". However, as described above, as long as "f≧2", it is arbitrary, and the way of classifying the emotion of the person PS[i] is also arbitrary. Therefore, for example, the acquisition unit 20[i] may classify the emotion of the person PS[i] into either positive emotion or negative emotion in two stages (in this case, positive emotion and negative emotion are the two types of classified emotions in total).

[0063] Furthermore, the emotion information acquisition process executed by the acquisition unit 20[i] may further include an emotion level determination process for determining and judging the emotion level of the person PS[i] in addition to the emotion classification process. When the emotion information acquisition process executed by the acquisition unit 20[i] includes an emotion level determination process, the determination and judgment results of the emotion level of the person PS[i] are also included in the emotion information of the person PS[i]. The emotion level of the person PS[i] represents the degree of emotion of the person PS[i]. Here, as an example, the emotion level is determined from the first to fifth levels. For any natural number s, the degree of emotion corresponding to the (s+1)th level is stronger than the degree of emotion corresponding to the sth level. Therefore, among the first to fifth levels, the degree of emotion is strongest at the fifth level, and the degree of emotion is weakest at the first level. In addition, the (s+1)th level is considered to be higher than the sth level. For example, if it is determined that the emotion of person PS[i] belongs to the emotion of “anger”, the acquisition unit 20[i] can further execute an emotion level determination process, thereby determining and judging to which of levels 1 to 5 the degree of the emotion of “anger” of person PS[i] belongs.

[0064] The emotion notification image corresponding to the emotion information of the person PS[i] displayed on the terminal device 10[g] is arbitrary as long as the emotion of the person PS[i] indicated by the emotion information of the person PS[i] can be recognized by the person PS[g]. That is, for example, if the emotion of the person PS[i] indicated by the emotion information of the person PS[i] is "anger", a predetermined anger icon image expressing the emotion of "anger" is displayed on the display unit 14[g] as an emotion notification image expressing the emotion of the person PS[i], and if the emotion of the person PS[i] indicated by the emotion information of the person PS[i] is "joy", a predetermined joy icon image expressing the emotion of "joy" is displayed on the display unit 14[g] as an emotion notification image expressing the emotion of the person PS[i]. Each of these icon images may be stored in advance in the storage unit 12 of each terminal device 10.

[0065] When the emotion information of person PS[i] includes the emotion level of person PS[i] as a result of executing the emotion level determination process, it is preferable to change the details of the emotion notifying image displayed on display unit 14[g] according to the emotion level of person PS[i].

[0066] According to this embodiment, even if the camera image output condition is set to off, it is possible to sense the emotions of the other person (i.e., the same effect as seeing the other person's facial expression is obtained), which is believed to enable the conference (electronic conference) to proceed smoothly.

[0067] <<Second Example>> A second embodiment will be described. The second embodiment is implemented in combination with the first embodiment. The emotion information acquisition unit 20[i] according to the second embodiment estimates the emotion of the person PS[i] based on emotion estimation information influenced by the emotion of the person PS[i], and acquires emotion information of the person PS[i] based on the estimation result. This estimation realizes at least the emotion classification process described above, and may also realize emotion level determination process.

[0068] For example, the emotion estimation information of the person PS[i] may include the captured image information of the person PS[i]. The captured image information of the person PS[i] is an image signal of a camera image obtained by the camera unit 16[i] of the terminal device 10[i], and this camera image includes at least a face image of the person PS[i] (an image of the face of the person PS[i] captured by the camera unit 16[i]). A method for estimating the emotion of a person from a face image of the person is well known, so a description of the method is omitted. The acquisition unit 20[i] may use any method for estimating the emotion of the person from the face image of the person. In addition, the image signal of the camera image obtained by the camera unit 16[i] (i.e., the captured image information of the person PS[i]) also includes the gaze information of the person PS[i]. The gaze information of the person PS[i] is information that represents the gaze direction and movement of the person PS[i], and it is known that there is a correlation between the gaze information of the person PS[i] and the emotion of the person PS[i]. For this reason, the emotion of the person PS[i] may be estimated by also referring to the gaze information of the person PS[i]. Alternatively, the facial and body movements of the person PS[i] may be detected from the captured image information of the person PS[i], and the emotion of the person PS[i] may be estimated based on the detection results of the movements.

[0069] Alternatively, for example, the emotion estimation information of the person PS[i] may include speech information of the person PS[i]. The speech information of the person PS[i] is an audio signal indicating the speech content of the person PS[i], and the speech information of the person PS[i] is included in the output audio signal of the microphone 17[i] of the terminal device 10[i]. The acquisition unit 20 can estimate the emotion of the person PS[i] from the speech information of the person PS[i]. A method for estimating the emotion of a person from the speech information of the person is publicly known, so a description of the method is omitted. The acquisition unit 20[i] can use any method for estimating the emotion of the person from the speech information of the person.

[0070] Further, or for example, the emotion estimation information of the person PS[i] may include biometric information of the person PS[i]. In this case, the sensor unit 19[i] (see FIG. 5) detects and acquires the biometric information of the person PS[i]. The biometric information of the person PS[i] indicates, for example, the heart rate, brain waves, body temperature, blood flow, and sweating state of the person PS[i]. A method for estimating an emotion of a person from the biometric information of the person is well known, and therefore a description of the method is omitted. The acquisition unit 20[i] may use any method for estimating an emotion of the person from the biometric information of the person.

[0071] The emotion estimation information of the person PS[i] may include any one or any two of the captured image information, speech voice information, and biometric information of the person PS[i], or may include all of the captured image information, speech voice information, and biometric information of the person PS[i]. It is expected that the accuracy of the estimation will increase as the types of information used for estimation increase.

[0072] According to this embodiment, even if the camera image output condition is set to off, each participant in the conference can sense the other participant's emotions without bothering the other participant.

[0073] In the second embodiment, a series of processing steps S10, S20, and S30 may be repeatedly executed at a predetermined cycle. For example, as shown in FIG. 10, the electronic conference period is divided into a plurality of frames each having a length of a predetermined cycle. In practice, after the start of the electronic conference period, a new frame is set every time a predetermined cycle elapses, and thus a plurality of frames are finally set in the electronic conference period. Then, in step S10, the emotion of the person PS[i] is estimated for each frame based on the emotion estimation information in the frame, and emotion information of the person PS[i] is obtained based on the estimation result. The emotion information transmitted between the terminal devices 10[i] and 10[g] in step S20 and the emotion information of the person PS[i] displayed on the terminal device 10[g] in step S30 may be updated with the latest emotion information obtained in step S10.

[0074] <<Third Example>> A third embodiment will now be described. The third embodiment is implemented in combination with the first embodiment. Emotion information acquisition unit 20[i] according to the third embodiment receives an emotion designation operation input from person PS[i] via operation unit 15[i] of terminal device 10[i], and acquires emotion information of person PS[i] based on the content of the emotion designation operation received via operation unit 15[i].

[0075] The emotion designation operation realizes at least the emotion classification process described above. That is, the emotion designation operation includes an emotion classification designation operation that designates which of the f types of classified emotions the emotion of the person PS[i] is to be classified into, and the acquisition unit 20[i] sets and acquires emotion information of the person PS[i] (more specifically, information indicating which of the f types of classified emotions the emotion of the person PS[i] is to be classified into) in accordance with the designation content of this emotion classification designation operation.

[0076] The above-mentioned emotional level determination process may be further implemented based on the emotion designation operation. That is, the emotion designation operation may include an emotion designation operation that designates the emotional level of the person PS[i]. In this case, the acquisition unit 20[i] sets and acquires the emotion information of the person PS[i] (more specifically, information indicating the emotional level of the person PS[i]) according to the contents of the emotion designation operation.

[0077] The emotion designation operation may be performed in any manner, and may be configured, for example, by selection from a list or by text input.

[0078] According to this embodiment, it is possible to inform the other person of one's own feelings or to know the other person's feelings without the need for estimation as described in the second embodiment. If one does not want to reveal one's inner thoughts to the other person, it is also possible to inform the other person of a specified feeling (or a false feeling in some cases).

[0079] In the third embodiment, a series of processes consisting of steps S10, S20, and S30 may be repeatedly executed at a predetermined cycle. The emotion information transmitted between terminal devices 10[i] and 10[g] in step S20, and the emotion information of person PS[i] displayed on terminal device 10[g] in step S30 may be updated with the latest emotion information acquired in step S10.

[0080] <<Fourth Example>> A fourth embodiment will now be described. The fourth embodiment is implemented in combination with the first embodiment, and the techniques shown in the second embodiment and the techniques shown in the third embodiment are incorporated into the fourth embodiment.

[0081] 11 shows a conceptual diagram of setting and acquiring emotion information for person PS[i] in the fourth embodiment. Emotion information acquisition section 20[i] in the fourth embodiment can execute both an inferred information generation process corresponding to the process for generating emotion information in the second embodiment, and a manual information generation process corresponding to the process for generating emotion information in the third embodiment. However, in the fourth embodiment, the information generated by the inferred information generation process and the information generated by the manual information generation process function as candidates for emotion information to be acquired in step S10.

[0082] That is, in the estimated information generation process, the acquisition unit 20[i] estimates the emotion of the person PS[i] based on the emotion estimation information of the person PS[i] by the method shown in the second embodiment, and generates first candidate information related to the emotion of the person PS[i] based on the estimation result. The first candidate information related to the emotion of the person PS[i] is the same as the emotion information of the person PS[i] that should be acquired in the second embodiment. In addition, in the manual information generation process, the acquisition unit 20[i] receives an input of an emotion designation operation from the person PS[i] at the operation unit 15[i] of the terminal device 10[i] by the method shown in the third embodiment, and generates second candidate information related to the emotion of the person PS[i] based on the content of the emotion designation operation received at the operation unit 15[i]. The second candidate information related to the emotion of the person PS[i] is the same as the emotion information of the person PS[i] that should be acquired in the third embodiment. Then, the acquisition unit 20[i] sets the first candidate information or the second candidate information as emotion information of the person PS[i] (that is, acquires the first candidate information or the second candidate information as emotion information of the person PS[i]).

[0083] The second candidate information based on the manual information generation process is generated only when an emotion designation operation is performed. When an emotion designation operation is performed, the acquisition unit 20[i] sets the second candidate information generated by the manual information generation process as the emotion information of the person PS[i] (i.e., acquires the second candidate information as the emotion information of the person PS[i]). When the second candidate information is set as the emotion information of the person PS[i], the emotion information of the person PS[i] based on the emotion designation operation is transmitted to the terminal device 10[g] and notified to the person PS[g].

[0084] When no emotion designation operation has been performed, the acquisition unit 20[i] sets the first candidate information generated in the estimated information generation process as the emotion information of the person PS[i] (i.e., acquires the first candidate information as the emotion information of the person PS[i]). When the first candidate information is set as the emotion information of the person PS[i], the emotion information of the person PS[i] based on the estimation result is transmitted to the terminal device 10[g] and notified to the person PS[g].

[0085] With respect to the emotion information of the person PS[i], it can be said that the operation mode of the terminal device 10[i] is set to a first mode in which the first candidate information is set to the emotion information of the person PS[i] and transmitted to the terminal device 10[g], or a second mode in which the second candidate information is set to the emotion information of the person PS[i] and transmitted to the terminal device 10[g]. The operation of the terminal device 10[i] in a state in which the operation mode of the terminal device 10[i] is set to the first mode is similar to that of the second embodiment, and the operation of the terminal device 10[i] in a state in which the operation mode of the terminal device 10[i] is set to the second mode is similar to that of the third embodiment. It can be considered that a mode setting unit (not shown) that sets the operation mode of the terminal device 10[i] to the first or second mode is provided in the terminal control unit 13[i].

[0086] The operation mode of the terminal device 10[i] is set to the first mode as a general rule. That is, the operation mode of the terminal device 10[i] is maintained in the first mode unless an emotion designation operation is input to the operation unit 15[i]. Starting from a state in which the operation mode of the terminal device 10[i] is set to the first mode, when an emotion designation operation is input to the operation unit 15[i], the operation mode of the terminal device 10[i] is switched from the first mode to the second mode. Thereafter, for example, the operation mode of the terminal device 10[i] is maintained in the second mode until a predetermined release operation is input to the operation unit 15[i], and the operation mode of the terminal device 10[i] is returned to the first mode when a predetermined release operation is input to the operation unit 15[i]. Alternatively, for example, after the operation mode of the terminal device 10[i] is switched from the first mode to the second mode, when a predetermined time (e.g., 10 seconds) has elapsed, the operation mode of the terminal device 10[i] may return from the second mode to the first mode.

[0087] According to this embodiment, it is possible to obtain both the effects of the second embodiment and the third embodiment. That is, in principle, it becomes possible to sense the emotions of the other party through the estimated information generation process without bothering each participant in the conference, while it becomes possible to notify the other party of a specified emotion (or a false emotion in some cases) only when one does not want to reveal his / her inner thoughts to the other party.

[0088] <<Fifth Example>> A fifth embodiment will be described. In the fourth embodiment, when the estimation result of the emotion of the person PS[i] by the estimation information generation process satisfies a predetermined notification judgment condition, the terminal device 10[i] may notify the person PS[i] of a manual input suggestion related to the input of an emotion designation operation. This will be described as the fifth embodiment. The fifth embodiment is an embodiment in which an applied technology is added to the fourth embodiment, and is implemented in combination with the fourth embodiment.

[0089] 12(a) and (b), each terminal device 10 is provided with a manual input suggestion notification unit 22 for providing the above-mentioned manual input suggestion notification, and the manual input suggestion notification unit 22 provided in the terminal device 10[i] will be specifically referred to as "22[i]". The notification unit 22[1] provides a manual input suggestion notification to the person PS[1], and the notification unit 22[2] provides a manual input suggestion notification to the person PS[2]. The same is true for the notification unit 22[3], etc. The notification unit 22[i] is a functional block that is realized by the terminal control unit 13[i] executing the terminal program PG1. The explanation of the notification unit 22[i] applies to each of the notification units 22[1] to 22[n].

[0090] The operation of the fifth embodiment will be described with reference to Fig. 13. In the fifth embodiment, the processing of steps S11 to S18 shown in Fig. 13 is executed in step S10 of Fig. 8. The processing of steps S11 to S18 can be executed in each terminal device 10, but for the sake of concrete explanation, attention is focused on terminal device 10[i] here, and the processing of steps S11 to S18 executed in terminal device 10[i] will be explained. The processing of steps S11 to S18 is executed by terminal control unit 13[i] in terminal device 10[i] executing terminal program PG1. The operation of Fig. 13 starts from the processing of step S11.

[0091] In step S11, emotion information acquisition unit 20[i] executes an inferred information generation process to infer the emotion of person PS[i] and acquires first candidate information. In the following step S12, terminal control unit 13[i] checks whether an emotion designation operation has been input to operation unit 15[i], and if an emotion designation operation has been input (Y in step S12), the process proceeds from step S12 to step S16, whereas if an emotion designation operation has not been input (N in step S12), the process proceeds from step S12 to step S13.

[0092] In step S13, the terminal control unit 13[i] (for example, a condition success / failure determination unit (not shown) included in the terminal control unit 13[i]) determines whether a predetermined notification determination condition is satisfied based on the estimation result of the emotion of the person PS[i] in step S11. The contents of the notification determination condition will be described later. If it is determined in step S13 that the notification determination condition is satisfied (Y in step S13), the process proceeds to step S14; otherwise (N in step S13), the process proceeds to step S18.

[0093] In step S14, the manual input suggestion notification unit 22[i] issues a manual input suggestion notification to the person PS[i] regarding the input of an emotion designation operation. The manual input suggestion notification executed by the manual input suggestion notification unit 22[i] may be any notification that makes the person PS[i] aware of the presence of an emotion designation operation, and may be, for example, a first suggestion notification that prompts or recommends the person PS[i] to input an emotion designation operation, a second suggestion notification that inquires about the necessity of inputting an emotion designation operation to the person PS[i], or a third suggestion notification that informs the person PS[i] that an emotion designation operation can be input. The manual input suggestion notification may be issued by any method that appeals to the perception of the person PS[i]. However, since a manual input suggestion notification by voice may cause an interruption to the progress of the conference, the manual input suggestion notification is preferably a visual notification. That is, for example, the manual input suggestion notification by the manual input suggestion notification unit 22[i] may be a notification that displays a suggestion image related to input of an emotion designation operation (for example, a text image that realizes the above-mentioned first, second or third suggestion notification) on the display unit 14[i].

[0094] After the manual input suggestion notification is given in step S14, in step S15, the terminal control unit 13[i] monitors whether or not an emotion designation operation is input to the operation unit 15[i]. After the manual input suggestion notification is given in step S14, if an emotion designation operation is input to the operation unit 15[i] before a predetermined timeout period has elapsed (Y in step S15), the process proceeds from step S15 to step S16, otherwise (N in step S15), the process proceeds from step S15 to step S18. If a predetermined cancel operation is input to the operation unit 15[i] after the manual input suggestion notification is given in step S14, the process may immediately proceed from step S15 to step S18.

[0095] In step S16, emotion information acquisition unit 20[i] generates second candidate information for person PS[i] by executing a manual information generation process based on the emotion designation operation input to operation unit 15[i]. In step S17 following step S16, emotion information acquisition unit 20[i] sets the second candidate information generated in step S16 as the emotion information of person PS[i]. When the second candidate information is set as the emotion information of person PS[i], the emotion information of person PS[i] based on the emotion designation operation is transmitted to terminal device 10[g] and notified to person PS[g].

[0096] Meanwhile, in step S18, emotion information acquisition unit 20[i] sets the first candidate information generated in step S11 as emotion information of person PS[i]. When the first candidate information is set as emotion information of person PS[i], emotion information of person PS[i] based on the estimation result is transmitted to terminal device 10[g] and notified to person PS[g].

[0097] An example of whether or not the notification determination condition is satisfied based on the result of the estimated information generation process will be described below. In the estimated information generation process, the emotion classification process through estimation determines (estimates) which of f types of classification destination emotions the emotion of the person PS[i] is classified into, and the emotion level determination process through estimation determines (estimates) the emotion level of the person PS[i].

[0098] For example, the notification determination condition is met when the emotion of the person PS[i] is classified into a specific classified emotion by the emotion classification process in the estimated information generation process. The specific classified emotion is, for example, the emotion of "anger". In this case, the notification determination condition is met when the person PS[i] has the emotion of "anger" and this emotion is correctly estimated. At this time, if the person PS[i] does not want the emotion information of "anger" to be notified to the person PS[g], the person PS[i] can hide his / her true emotion by inputting an emotion designation operation. The specific classified emotion may be an emotion other than "anger".

[0099] Alternatively, for example, the notification determination condition is satisfied when the emotion level of the person PS[i] is determined (estimated) to be equal to or higher than a predetermined level by the emotion level determination process in the estimated information generation process. For example, the notification determination condition may be satisfied when the emotion level of the person PS[i] is determined (estimated) to be equal to or higher than the fourth level. At this time, if the person PS[i] does not want the high emotion level to be notified to the person PS[g], the person PS[i] can hide the heightened emotion level by inputting an emotion designation operation. The notification determination condition may be satisfied when the emotion of the person PS[i] is classified into a specific destination emotion by the emotion classification process in the estimated information generation process, and the emotion level of the person PS[i] is determined (estimated) to be equal to or higher than a predetermined level by the emotion level determination process in the estimated information generation process.

[0100] Further, or for example, the notification determination condition may be satisfied when there is a large change in the emotion of the person PS[i] estimated in the estimated information generation process. A large change in emotion is considered to represent the agitation of the person PS[i]. More specifically, for example, the notification determination condition may be satisfied when the amount of change in the emotion level of the person PS[i] estimated in the estimated information generation process within a predetermined time period is equal to or greater than a predetermined value. In this case, if the person PS[i] does not want the large change in emotion (agitation) to be notified to the person PS[g], the person PS[i] can hide the fact that a large change in emotion has occurred by inputting an emotion designation operation.

[0101] Also, for example, the person PS[i] may freely set the notification determination conditions in the terminal device 10[i] by inputting a predetermined setting operation to the operation unit 15[i].

[0102] By making it possible to provide a manual input suggestion notification as in this embodiment, it is possible to effectively suppress notification to the other party of emotional information that the other party does not want to know, such as a specific emotion or emotional ups and downs of person PS[i].

[0103] <<Sixth Example>> A sixth embodiment will be described. The sixth embodiment can be combined with any of the first to fifth embodiments. Referring to Figs. 14(a) and 14(b), each terminal device 10 is provided with an image processing unit 24, and the image processing unit 24 provided in the terminal device 10[i] is particularly referred to as "24[i]". The image processing unit 24[i] may be a functional block realized by the terminal control unit 13[i] executing the terminal program PG1, or may be a processor provided separately from the terminal control unit 13[i]. In addition, each terminal device 10 is provided with registered image group data 25 and processing data 26. The registered image group data 25 and processing data 26 in the terminal device 10[i] are particularly referred to as "25[i]" and "26[i]". The registered image group data 25[i] and the processing data 26[i] may be data pre-stored in the memory unit 12[i] (see FIG. 5), or may be data transmitted from the server device 30 to the terminal device 10[i] prior to the start of the electronic conference.

[0104] As shown in Fig. 15, a total of n registered images RI[1] to RI[n] corresponding to the faces of persons PS[1] to PS[n] are registered and set in advance in the conference system SYS. The registered image RI[i] is a still image corresponding to the face of the person PS[i]. The registered image RI[i] may be the image itself obtained by photographing the face of the person PS[i], or may be an animation image imitating the face of the person PS[i].

[0105] Image data of registered images RI[1] to RI[n] is included in registered image group data 25 of each terminal device 10. However, image data of registered image RI[i] of person PS[i] does not have to be included in registered image group data 25[i] of terminal device 10[i] corresponding to person PS[i].

[0106] The processing data 26 is data that is referred to in order to process each registered image according to the emotion information. This processing is performed by the image processing unit 24, and the processed registered image is displayed as an emotion notification image (see FIG. 8) in step S30. The operation of displaying an emotion notification image based on the emotion information of the person PS[i] on the terminal device 10[g] will be specifically described. The emotion information of the person PS[i] is transmitted from the terminal device 10[i] to the terminal device 10[g] in steps S10 and S20 in FIG. 8. In step S30, the image processing unit 24[g] in the terminal device 10[g] reads out the registered image RI[i] corresponding to the person PS[i] from the registered image group data 25[g], and processes the registered image RI[i] according to the emotion information of the person PS[i] based on the processing data 26[g]. The processed registered image RI[i] is displayed on the display unit 14[g] as an emotion notification image representing the emotion of the person PS[i]. For example, when the emotion of person PS[i] indicated by the emotion information of person PS[i] is "anger," the image processing unit 24[g] processes the registered image RI[i] so that the processed registered image RI[i] becomes an image imitating the face of person PS[i] expressing the emotion of "anger." In this case, if the emotion information of person PS[i] includes an emotion level, the content of the processing may be changed according to the emotion level.

[0107] According to this embodiment, even if the camera image output condition is set to off, it is possible to guess the other person's emotions by looking at the registered image after processing (i.e., an effect similar to seeing the other person's facial expression can be obtained).

[0108] <<Seventh Example>> A seventh embodiment will be described. In the seventh embodiment, an applied technology is added to the sixth embodiment. In the conference system SYS, a voice recognition process may be executed to convert the contents of speech of the persons PS[1] to PS[n] into text data. Here, it is assumed that the voice recognition process is executed by the server control unit 33. When the person PS[i] speaks during an electronic conference, the server control unit 33 receives an output voice signal of the microphone 17[i] including a voice signal of the contents of speech of the person PS[i] from the terminal device 10[i], and converts the contents of speech of the person PS[i] into text data by applying a voice recognition process to the received signal. The server device 30 transmits this text data (text data representing the contents of speech of the person PS[i]) to each terminal device 10 other than the terminal device 10[i]. In parallel with this, the server control unit 33 also transmits emotion information of the person PS[i] to each terminal device 10 other than the terminal device 10[i].

[0109] At this time, the terminal device 10[g], which is a representative of the terminal devices 10 other than the terminal device 10[i], operates as follows. In the terminal device 10[g] that has received the text data (hereinafter referred to as text data TXT[i]) representing the emotion information of the person PS[i] and the speech content of the person PS[i], the image processing unit 24[g] reads out the registered image RI[i] corresponding to the person PS[i] from the registered image group data 25[g], and generates a moving image according to the emotion information of the person PS[i] and the text data TXT[i] based on the registered image RI[i] while referring to the processing data 26[g]. This moving image is displayed on the display unit 14[g] as an emotion notification image. This moving image is an image obtained by processing the registered image RI[i] according to the emotion information of the person PS[i] and according to the text data TXT[i]. For example, when the emotion of person PS[i] indicated in the emotion information of person PS[i] is "anger," a moving image is generated in which an image imitating the face of person PS[i] expressing the emotion of "anger" moves in accordance with the content of the words of person PS[i].

[0110] According to this embodiment, even if the camera image output condition is set to off, it is possible to display a moving image that shows the other person actually speaking, including facial expressions related to emotions. This makes it possible to guess the other person's emotions (i.e., it is possible to obtain the same effect as gazing at the other person's facial expressions).

[0111] In addition, when an emotion designation operation is input in the terminal device 10[i], an audio signal processing process may be performed to process the output audio signal of the microphone 17[i] including the audio signal of the speech content of the person PS[i] based on the emotion information of the person PS[i] generated based on the emotion designation operation. In this case, the output audio signal of the microphone 17[i] after the audio signal processing process is output by the terminal device 10[g]. That is, for example, when an emotion designation operation is input in the terminal device 10[i] and the emotion designation operation specifies the emotion of "anger", the output audio signal of the microphone 17[i] representing the speech content of the person PS[i] may be processed into an audio signal including the emotion of "anger", and the processed audio signal may be transmitted to the terminal device 10[g] and output by the terminal device 10[g]. The audio signal processing process to process the output audio signal of the microphone 17[i] is performed by an audio signal processing unit (not shown) provided in the terminal device 10[i]. However, this audio signal processing may be performed by the server device 30.

[0112] <<Eighth Example>> The eighth embodiment will be described. In the eighth embodiment, an example of an operation related to recording and reproducing of conference data will be described. The eighth embodiment can be combined with any embodiment including the first to seventh embodiments described above. As shown in FIG. 16, the server device 30 is provided with a recording management unit 41, a recording medium 42, a reproduction management unit 43, a statistical processing unit 44, and a statistical notification unit 45. The recording medium 42 may be a recording medium physically separated from the server device 30. The recording management unit 41, the reproduction management unit 43, the statistical processing unit 44, and the statistical notification unit 45 are functional blocks realized by the server control unit 33 executing the terminal program PG2. Note that the statistical processing unit 44 and the statistical notification unit 45 are not necessary to realize the eighth embodiment. In the eighth embodiment, the functions of the statistical processing unit 44 and the statistical notification unit 45 will not be mentioned, and the statistical processing unit 44 and the statistical notification unit 45 will be described in other embodiments described later.

[0113] The recording management unit 41 records conference data relating to a conference (electronic conference) held in the conference system SYS onto the recording medium 42. In response to receiving a request signal from any of the terminal devices 10, the playback management unit 43 reads all or part of the conference data from the recording medium 42 and transmits the read data to the terminal device 10 that is the source of the request signal.

[0114] The structure of the conference data is shown in Fig. 17. The conference data includes conference main data, which includes conference shared image data (in other words, conference shared video data), conference shared audio data, and conference shared material data.

[0115] During the electronic conference (see FIG. 10), a shared image is set under the control of any of the terminal devices 10 or the server device 30, and the shared image is displayed on all of the terminal devices 10[1] to 10[n]. The conference shared image data is image data that indicates the shared image during the electronic conference.

[0116] During the electronic conference period (see FIG. 10), each terminal device 10 continuously transmits a voice signal based on the output voice signal of the microphone 17 to the server device 30. Therefore, during the speech period of the person PS[i], a voice signal including the speech content of the person PS[i] is transmitted from the terminal device 10[i] to the server device 30. A voice signal including all the voice signals transmitted from each of the terminal devices 10[1] to 10[n] to the server device 30 (i.e., a composite signal of the voice signals transmitted from each of the terminal devices 10[1] to 10[n] to the server device 30) is recorded as conference shared voice data in the recording medium 42 throughout the entire electronic conference period. Each terminal device 10 is provided with an echo canceller. Due to the function of the echo canceller, the signal component of the output sound of the speaker 18[i] is removed or sufficiently reduced in the voice signal transmitted from the terminal device 10[i] to the server device 30. In addition, a timestamp (time data) indicating the date and time of data generation or acquisition is added to each data recorded on the recording medium 42. When each data is played back, the data is synchronized based on the timestamp, and data from a specified time period can be selected and played back based on the timestamp.

[0117] Any of the persons PS[1] to PS[n] can register material data for the conference by operating their own terminal device 10. This registered material data is recorded in the recording medium 42 as conference shared material data. The conference shared material data can be displayed on each terminal device 10 during the period when the electronic conference is held.

[0118] Each person can use the conference data at any time for any purpose (for example, for creating minutes). After the electronic conference period ends, when any person PS[i] among the persons PS[1] to PS[n] inputs a predetermined first playback request operation to the operation unit 15[i] of the terminal device 10[i], the terminal control unit 13[i] transmits a predetermined first request signal to the server device 30. The first playback request operation may include an authentication operation such as inputting a password. When the server device 30 receives the first request signal, the playback management unit 43 reads out the conference master data from the recording medium 42 and transmits the conference master data to the terminal device 10[i]. When the terminal device 10[i] receives the conference master data, the conference master data is reproduced on the terminal device 10[i] under the control of the terminal control unit 13[i]. The reproduction of the conference main data includes displaying the shared image represented by the conference shared image data on the display unit 14[i] and outputting the audio signal represented by the conference shared audio data from the speaker 18[i], and may further include displaying the conference shared material data on the display unit 14[i]. The reproduction of the conference main data on the terminal device 10[i] reproduces the contents of the conference held in the past on the terminal device 10[i]. That is, after the electronic conference period ends, the reproduction of the conference main data on the terminal device 10[i] causes the shared images displayed on the terminal device 10[i] during the conference to be displayed again in chronological order, and the speech contents of each person during the conference to be output again in chronological order.

[0119] As shown in Fig. 17, the conference data includes one or more pairs of individual voice data and attribute data. In a conference, each person speaks. The individual voice data is voice data that represents the content of each person's utterance. The record management unit 41 creates attribute data corresponding to each utterance and includes it in the conference data.

[0120] During an electronic conference, a speech / silence period setting process is executed based on an output audio signal of the microphone 17[i]. In the speech / silence period setting process, the speech period of the person PS[i], which is a period during which the person PS[i] is speaking, and the silence period of the person PS[i], which is a period during which the person PS[i] is silent, are recognized and set. The speech / silence period setting process for setting the speech period and silence period of the person PS[1] may be executed by the terminal control unit 13[1], and the speech / silence period setting process for setting the speech period and silence period of the person PS[2] may be executed by the terminal control unit 13[2]. The same applies to the setting of the speech period and silence period of the person PS[3], etc. Alternatively, the speech / silence period setting process for setting the speech period and silence period of each of the persons PS[1] to PS[n] may be executed by the server control unit 33.

[0121] The structure of the attribute data is shown in Fig. 18. The record management unit 41 assigns a unique speech number to each speech of a participant in the electronic conference in chronological order during the conference. Then, for each speech number, the record management unit 41 associates individual voice data representing the speech content of the corresponding speaker with attribute data, and includes them in the conference data.

[0122] As a specific example, assume that person PS[1] speaks first during an electronic conference and person PS[6] speaks second. In this case, the record management unit 41 sets the speech number of "001" in response to the first speech, and stores in the conference data individual voice data SD_001 including the output voice signal of the microphone 17[1] during the speech period of person PS[1] and attribute data AD_001 in association with the speech number of "001". In response to the second speech, the record management unit 41 sets the speech number of "002", and stores in the conference data individual voice data SD_002 including the output voice signal of the microphone 17[6] during the speech period of person PS[6] and attribute data AD_002 in association with the speech number of "002". The same applies to the third and subsequent speeches.

[0123] The attribute data includes speaker information that identifies which of the persons PS[1] to PS[n] the corresponding speaker is, speech time information that identifies the start and end times of the corresponding speaker's speech, and emotion information during the corresponding speaker's speech. The record management unit 41 can use the emotion information transmitted from each terminal device 10 to the server device 30 during the electronic conference to create attribute data.

[0124] Regarding the above specific example, in the attribute data AD_001 associated with the speech number of "001", the speaker information indicates that the speaker corresponding to the individual voice data SD_001 is person PS[1], the speech time information indicates the start time and end time of the speech period of the speaker (PS[1]) corresponding to the individual voice data SD_001, and the speaker emotion information indicates the emotion information during the speech period of the speaker (PS[1]) corresponding to the individual voice data SD_001. Similarly, in the attribute data AD_002 associated with the speech number of "002", the speaker information indicates that the speaker corresponding to the individual voice data SD_002 is person PS[6], the speech time information indicates the start time and end time of the speech period of the speaker (PS[6]) corresponding to the individual voice data SD_002, and the speaker emotion information indicates the emotion information during the speech period of the speaker (PS[6]) corresponding to the individual voice data SD_002.

[0125] Furthermore, the attribute data may further include the page number of the material that was referenced as a shared image during the speaking period of the corresponding speaker.

[0126] It is possible to record the individual voice data and the conference shared voice data separately on the recording medium 42, but since the conference shared voice data includes all the individual voice data, it is also possible to record the conference shared voice data on the recording medium 42 and extract the individual voice data from the conference shared voice data as necessary.

[0127] What is noteworthy about this embodiment is that emotional information of the speaker is stored in the recorded conference data. By referring to the emotional information of the speaker in the conference data, the referrer can estimate the emotion of the speaker at the time of speaking. This is considered to be useful for interpreting the intention of the speech.

[0128] As described above, each person can use the conference data at any time for any purpose (for example, for creating minutes). After the electronic conference period ends, when any person PS[i] among the persons PS[1] to PS[n] inputs a predetermined second reproduction request operation to the operation unit 15[i] of the terminal device 10[i], the terminal control unit 13[i] transmits a predetermined second request signal to the server device 30. The second reproduction request operation may include an authentication operation such as inputting a password. Here, the second reproduction request operation is an operation requesting acquisition of attribute data, and therefore the second request signal is an operation requesting transmission of the attribute data. When the server device 30 receives the second request signal, the reproduction management unit 43 reads out each attribute data included in the conference data from the recording medium 42 and transmits each attribute data to the terminal device 10[i]. When each attribute data is received by the terminal device 10[i], each attribute data is displayed on the terminal device 10[i] under the control of the terminal control unit 13[i].

[0129] Here, when a predetermined third reproduction request operation that designates attribute data AD_001 among all attribute data displayed on the terminal device 10[i] is input to the operation unit 15[i], the terminal control unit 13[i] transmits a predetermined third request signal to the server device 30. When the server device 30 receives the third request signal, the reproduction management unit 43 reads out the individual voice data SD_001 included in the conference data from the recording medium 42 and transmits the individual voice data SD_001 to the terminal device 10[i]. When the terminal device 10[i] receives the individual voice data SD_001, the individual voice data SD_001 is reproduced and output on the terminal device 10[i] under the control of the terminal control unit 13[i]. That is, the voice signal included in the voice data SD_001 (the output voice signal of the microphone 17[1] during the speech period of the person PS[1] corresponding to the speech number 001) is output as voice on the terminal device 10[i]. The same applies when a third reproduction request operation specifying attribute data AD_002, etc. is input to operation unit 15[i].

[0130] <<Ninth Example>> A ninth embodiment will be described. In the ninth embodiment, an example of an operation relating to recording of conference data will also be described. The ninth embodiment can be combined with any embodiment including the above-mentioned first to eighth embodiments. In the ninth embodiment, the conference data shown in the eighth embodiment is also recorded on a recording medium 42 (see FIG. 16). However, the recording management unit 41 according to the ninth embodiment adds additional data as shown in FIG. 19 to the conference data recorded on the recording medium 42.

[0131] The additional data will now be described. In the record management unit 41, the electronic conference period is divided into a number of unit periods arranged in chronological order. Each unit period has a length of a predetermined unit time. The length of the unit time is arbitrary, for example, 1 minute, 10 seconds, or 5 seconds. The unit period may coincide with the frames shown in FIG. 10. However, the unit period may be made up of a number of frames.

[0132] The additional data includes speech presence / absence information. The speech presence / absence information indicates, for each participant of the conference (i.e., for each of the persons PS[1] to PS[n]), whether or not the participant spoke during each unit period during the electronic conference. The record management unit 41 can create the speech presence / absence information based on the setting result of the speech / silence period setting process described above.

[0133] The additional data includes emotion time-series information. The emotion time-series information represents, for each participant of the conference (i.e., for each of the persons PS[1] to PS[n]), a time-series change in the emotion information of the participant during the electronic conference. Therefore, the emotion time-series information includes emotion information for person PS[1] in each unit period, emotion information for person PS[2] in each unit period, ..., and emotion information for person PS[n] in each unit period. The record management unit 41 can create emotion time-series information based on emotion information transmitted from each terminal device 10 to the server device 30 during the electronic conference.

[0134] Furthermore, the additional data may further include the page number of the material that was referenced as the shared image in each unit period.

[0135] What is noteworthy in this embodiment is that the emotional information of each participant is stored in the recorded conference data. By referring to the emotional information of each participant, a viewer can estimate the situation at each time during the electronic conference and the thoughts of each participant.

[0136] The statistical processing unit 44 (FIG. 16) can be used as a functional block for analyzing the atmosphere of the place at each time during the electronic conference. As shown in FIG. 20, the statistical processing unit 44 generates emotion statistical data by taking statistics of emotion information of the persons PS[1] to PS[n] at each time during the electronic conference. The statistical processing unit 44 may generate emotion statistical data in real time during the electronic conference, or may generate emotion statistical data at any timing after the electronic conference ends.

[0137] Specifically, for example, the statistical processing unit 44 generates first emotion statistical data by taking statistics on emotion information of persons PS[1] to PS[n] in the first unit period, and generates second emotion statistical data by taking statistics on emotion information of persons PS[1] to PS[n] in the second unit period. The same applies to the third unit period. Here, it is considered that the electronic conference period is made up of the first to m-th unit periods (m is an arbitrary integer equal to or greater than 2). Then, the j-th emotion statistical data is generated by taking statistics on emotion information of persons PS[1] to PS[n] in the j-th unit period (j is an arbitrary natural number equal to or less than m).

[0138] The jth emotion statistical data is a statistic based on the emotion information of persons PS[1]~PS[n], and the jth emotion statistical data is generated from persons PS[1]~PS[n] according to a predetermined algorithm. Simply put, for example, when generating emotion information of person PS[i] by classifying whether the emotion of person PS[i] belongs to positive emotion or negative emotion in two stages, the number VAL[j] of pieces of emotion information classified as positive emotion out of the emotion information of persons PS[1]~PS[n] in the jth unit period is counted, and the count value VAL[j] is generated as the jth emotion statistical data.

[0139] The record management section 41 may include the emotion statistical data for the first to m-th unit periods in the conference data and record it on the recording medium 42. This makes it possible to check the atmosphere of the conference at any time.

[0140] It is also possible to have each terminal device 10 provide a real-time notification based on emotion statistical data during the electronic conference. This is achieved by using the statistics notification unit 45 (see FIG. 16). The statistics notification unit 45 will be described.

[0141] The statistics notification unit 45 executes a statistics notification process for causing each terminal device 10 to perform a notification based on emotion statistical data in each unit period. Notifications performed in the statistics notification process are performed in real time during the period in which the electronic conference is held. For this reason, the statistics notification process is premised on the fact that emotion statistical data is generated in real time in the statistical processing unit 44 during the period in which the electronic conference is held.

[0142] When the jth emotion statistical data for the jth unit period during which the electronic conference is held is generated by statistical processing unit 44, statistics notification unit 45 transmits statistical notification information based on the jth emotion statistical data to each terminal device 10 (see FIG. 21). When each terminal device 10 receives the statistical notification information, terminal control unit 13 causes display unit 14 to display a statistical notification image representing the received statistical notification information. As a result, the statistical notification information is notified to person PS[i] in terminal device 10[i].

[0143] Fig. 22 shows an example of the display on display unit 14 with a statistics notification image displayed. In Fig. 22, image TI is an example of a statistics notification image. For example, when the jth emotion statistical data is the above-mentioned count value VAL[j], the statistics notification image may be an image in which the count value VAL[j] itself is quantified, or it may be an image in which the count value VAL[j] is displayed as a bar, as in image TI.

[0144] By viewing the statistics notification image, an observer of the statistics notification image can sense the atmosphere (for example, the level of excitement) formed by all the participants, which is expected to promote the smooth progress of the meeting.

[0145] Note that the statistics notification section 45 may be configured to transmit statistical notification information to each terminal device 10 only when a predetermined notification condition is met. That is, for example, when the jth emotion statistical data is the above-mentioned count value VAL[j], the statistics notification section 45 may be configured to transmit statistical notification information based on the jth emotion statistical data to each terminal device 10 only when the count value VAL[j] is equal to or greater than a predetermined value. This allows, for example, each participant in a conference to recognize in real time whether the atmosphere in the room is lively or not.

[0146] Also, depending on the setting state of the terminal device 10, the statistics notification image may not be displayed on the display unit 14. For example, depending on the setting states of the terminal devices 10[1] and 10[2], the statistics notification image may be displayed on the terminal device 10[1], but may not be displayed on the terminal device 10[2]. Taking this into consideration, it can be said that in the statistics notification process, the statistics notification unit 45 causes at least one or more of the terminal devices 10[1] to 10[n] to perform a notification based on emotion statistical data.

[0147] <<Tenth Example>> A tenth embodiment will be described. In the tenth embodiment, applied techniques, modified techniques, or notes that can be applied to the above-mentioned embodiments will be described.

[0148] In each of the above-mentioned embodiments, it is assumed that the camera image output condition is set to off in each of the terminal devices 10[1] to 10[n], and mutual notification of emotional information is performed to deal with the fact that each participant cannot see the other participant's facial expression. If the camera image output condition is set to on in each of the terminal devices 10[1] to 10[n], mutual notification of emotional information may not be performed (i.e., the processing of steps S10, S20, and S30 in FIG. 8 may not be executed). However, even if the camera image output condition is set to on in each of the terminal devices 10[1] to 10[n], mutual notification of emotional information may be performed (i.e., the processing of steps S10, S20, and S30 in FIG. 8 may be executed). Furthermore, mutual notification of emotion information may also be performed when the terminal devices 10[1] to 10[n] include a mixture of terminal devices 10 for which the camera image output condition is set to on and terminal devices 10 for which the camera image output condition is set to off (i.e., the processes of steps S10, S20, and S30 in FIG. 8 may be executed).

[0149] In each of the above-described embodiments, it is assumed that an emotion information acquisition unit 20 is provided in each terminal device 10, so that a total of n emotion information acquisition units 20[1]-20[n] are provided in terminal devices 10[1]-10[n]. It can also be thought of as a single emotion state acquisition unit (first emotion state acquisition unit) that acquires emotion information of persons PS20[1]-20[n] being provided in the conference system SYS, and the single emotion state acquisition unit being composed of emotion information acquisition units 20[1]-20[n] (a total of n second emotion state acquisition units).

[0150] Instead of providing each terminal device 10 with an emotion information acquisition unit 20, the server device 30 may be provided with a single emotion state acquisition unit (first emotion state acquisition unit) that acquires emotion information of the person PS20[1]-20[n]. In this case, the single emotion state acquisition unit (first emotion state acquisition unit) may be a functional block realized by the server control unit 33 executing the server program PG2, and the emotion information of the person PS[i] may be acquired according to the method of each embodiment described above by receiving information required to acquire emotion information of the person PS[i] from the terminal device 10[i] via the network 50. In this case, even if the camera image output condition is set to off in any terminal device 10[i] among the terminal devices 10[1]-10[n], the conference system SYS (e.g., the terminal program PG1 and the server program PG2) may have a function to cause the camera unit 16[i] in the terminal device 10[i] to take an image and transmit an image signal of the camera image by the camera unit 16[i] to the server device 30. However, at this time, the server device 30 blocks the transmission of the image signal of the camera image captured by the camera unit 16[i] of the terminal device 10[i] to the other terminal devices 10.

[0151] Among the terminal devices 10[1] to 10[n], one or more of the terminal devices 10 may be an in-vehicle device mounted on a vehicle such as an automobile.

[0152] A specific program that defines any specific process and specific operation to be realized in the terminal device 10[i] or the server device 30 may be created, and the specific program may be executed by a processor provided in the terminal device 10[i] or the server device 30, thereby realizing the specific process and specific operation. The terminal program PG1 or the server program PG2 is an example of the specific program. As already mentioned, a processor is provided in each of the terminal control unit 13[i] and the server control unit 33.

[0153] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each component are not limited to those described in the above embodiments. The specific numerical values ​​shown in the above description are merely examples, and can be changed to various numerical values ​​as a matter of course. [Explanation of symbols]

[0154] SYS Conference System (Electronic Conference System) PS Person (participant) 10 Terminal Equipment 11, 11[i] Communication processing unit 12, 12[i] Storage section 13, 13[i] Terminal control unit 14, 14[i] Display section 15, 15[i] Operation section 16, 16[i] Camera section 17, 17[i] Microphone 18, 18[i] Speaker 19, 19[i] Sensor section 20, 20[i] Emotional information acquisition unit 22, 22[i] Manual input suggestion notification section 24, 24[i] Image processing section 25, 25[i] Registered image group data 26, 26[i] Processing data 30 Server device 31 Communication processing unit 32 Storage section 33 Server control section 41 Records Management Department 42 Recording media 43 Reproduction Management Department 44 Statistical Processing Unit 45 Statistics Notification Department 50 Network

Claims

1. An electronic conference system for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, an emotion information acquisition unit that acquires emotion information corresponding to an emotion of a first person that is any one of the plurality of people; transmitting emotion information of the first person through the network to a terminal device of a second person, the second person being included in the plurality of people and different from the first person, and notifying the second person of the transmitted emotion information of the first person through the terminal device of the second person; The emotion information acquisition unit is a first process of estimating an emotion of the first person based on emotion estimation information of the first person that is influenced by the emotion of the first person, and generating first candidate information relating to the emotion of the first person based on an estimation result; and a second process of generating second candidate information relating to the emotion of the first person based on the content of an emotion designation operation when an emotion designation operation is received from the first person at a terminal device of the first person, When the emotion designation operation is performed, the second candidate information generated in the second process is set as emotion information of the first person, whereas when the emotion designation operation is not performed, the first candidate information generated in the first process is set as emotion information of the first person. ,Electronic conferencing system.

2. a notification unit that notifies the first person of the input of the emotion specifying operation when the estimation result by the first process satisfies a predetermined condition. The electronic conference system according to claim 1 .

3. an image processing unit that processes a registered image corresponding to the first person in accordance with emotion information of the first person; The processed registered image is displayed on the terminal device of the second person.

3. The electronic conference system according to claim 1 or 2.

4. An electronic conference system for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, an emotion information acquisition unit that acquires emotion information corresponding to an emotion of a first person that is any one of the plurality of people; An image processing unit, transmitting emotion information of the first person through the network to a terminal device of a second person, the second person being included in the plurality of people and different from the first person, and notifying the second person of the transmitted emotion information of the first person through the terminal device of the second person; the image processing unit generates a moving image according to emotion information of the first person and speech information of the first person based on a registered image associated with the first person; The video is displayed on the terminal device of the second person. ,Electronic conferencing system.

5. An electronic conference system for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, an emotion information acquisition unit that acquires emotion information corresponding to an emotion of a first person that is any one of the plurality of people; a record management unit that records conference data relating to the electronic conference on a recording medium, transmitting emotion information of the first person through the network to a terminal device of a second person, the second person being included in the plurality of people and different from the first person, and notifying the second person of the transmitted emotion information of the first person through the terminal device of the second person; The plurality of persons includes a first person to an nth person, where n is an integer of 2 or more; the emotion information acquisition unit acquires emotion information corresponding to emotions for each of the first to n-th persons, thereby acquiring emotion information for the first to n-th persons; The record management unit includes emotion information of each speaker in the conference data for each speech of each person. ,Electronic conferencing system.

6. An electronic conference system for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, an emotion information acquisition unit that acquires emotion information corresponding to an emotion of a first person that is any one of the plurality of people; a record management unit that records conference data relating to the electronic conference on a recording medium, transmitting emotion information of the first person through the network to a terminal device of a second person, the second person being included in the plurality of people and different from the first person, and notifying the second person of the transmitted emotion information of the first person through the terminal device of the second person; The plurality of persons includes a first person to an nth person, where n is an integer of 2 or more; the emotion information acquisition unit acquires emotion information corresponding to emotions for each of the first to n-th persons, thereby acquiring emotion information for the first to n-th persons; The record management unit includes emotion information of the first to nth persons in the conference data. ,Electronic conferencing system.

7. a statistical processing unit that generates emotion statistical data by taking statistics of emotion information of the first to nth persons; The record management unit includes the emotion statistical data in the conference data. The electronic conference system according to claim 6.

8. An electronic conference system for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, an emotion information acquisition unit that acquires emotion information corresponding to an emotion of a first person that is any one of the plurality of people; A statistical processing unit; A statistics notification unit, transmitting emotion information of the first person through the network to a terminal device of a second person, the second person being included in the plurality of people and different from the first person, and notifying the second person of the transmitted emotion information of the first person through the terminal device of the second person; The plurality of persons includes a first person to an nth person, where n is an integer of 2 or more; the emotion information acquisition unit acquires emotion information corresponding to emotions for each of the first to n-th persons, thereby acquiring emotion information for the first to n-th persons; the statistical processing unit generates emotion statistical data by taking statistics of emotion information of the first to nth persons; The statistics notification unit causes one or more of the plurality of terminal devices to perform a notification according to the emotion statistical data. ,Electronic conferencing system.

9. 1. An electronic conference method for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, comprising: an emotion information acquiring step of acquiring emotion information corresponding to an emotion of a first person which is any one of the plurality of people; a transmission step of transmitting emotion information of the first person through the network to a terminal device of a second person included in the plurality of people and different from the first person; a notification step of notifying the second person of the transmitted emotion information of the first person through a terminal device of the second person, In the emotion information acquisition step, a first process of estimating an emotion of the first person based on emotion estimation information of the first person that is influenced by the emotion of the first person, and generating first candidate information relating to the emotion of the first person based on an estimation result; and a second process of generating second candidate information relating to the emotion of the first person based on the content of an emotion designation operation when an emotion designation operation is received from the first person at a terminal device of the first person, When the emotion designation operation is performed, the second candidate information generated in the second process is set as emotion information of the first person, whereas when the emotion designation operation is not performed, the first candidate information generated in the first process is set as emotion information of the first person. ,Electronic conferencing methods.

10. 1. An electronic conference method for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, comprising: an emotion information acquiring step of acquiring emotion information corresponding to an emotion of a first person which is any one of the plurality of people; a transmission step of transmitting emotion information of the first person through the network to a terminal device of a second person included in the plurality of people and different from the first person; a notification step of notifying the second person of the transmitted emotion information of the first person through a terminal device of the second person, generating a moving image according to emotion information of the first person and speech information of the first person based on a registered image associated with the first person; The video is displayed on the terminal device of the second person. ,Electronic conferencing methods.

11. 1. An electronic conference method for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, comprising: an emotion information acquiring step of acquiring emotion information corresponding to an emotion of a first person which is any one of the plurality of people; a transmission step of transmitting emotion information of the first person through the network to a terminal device of a second person included in the plurality of people and different from the first person; a notification step of notifying the second person of the transmitted emotion information of the first person through a terminal device of the second person; a record management step of recording conference data relating to the electronic conference on a recording medium, The plurality of persons includes a first person to an nth person, where n is an integer of 2 or more; In the emotion information acquisition step, emotion information corresponding to an emotion is acquired for each of the first to n-th persons, thereby acquiring emotion information of the first to n-th persons; In the record management step, emotion information of each person is included in the conference data for each speech. ,Electronic conferencing methods.

12. 1. An electronic conference method for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, comprising: an emotion information acquiring step of acquiring emotion information corresponding to an emotion of a first person which is any one of the plurality of people; a transmission step of transmitting emotion information of the first person through the network to a terminal device of a second person included in the plurality of people and different from the first person; a notification step of notifying the second person of the transmitted emotion information of the first person through a terminal device of the second person; a record management step of recording conference data relating to the electronic conference on a recording medium, The plurality of persons includes a first person to an nth person, where n is an integer of 2 or more; In the emotion information acquisition step, emotion information corresponding to an emotion is acquired for each of the first to n-th persons, thereby acquiring emotion information of the first to n-th persons; In the record management step, emotion information of the first to nth persons is included in the conference data. ,Electronic conferencing methods.

13. 1. An electronic conference method for holding an electronic conference among a plurality of people using a plurality of terminal devices connected to each other through a network, comprising: an emotion information acquiring step of acquiring emotion information corresponding to an emotion of a first person which is any one of the plurality of people; a transmission step of transmitting emotion information of the first person through the network to a terminal device of a second person included in the plurality of people and different from the first person; a notification step of notifying the second person of the transmitted emotion information of the first person through a terminal device of the second person, The plurality of persons includes a first person to an nth person, where n is an integer of 2 or more; acquiring emotion information of the first to n-th persons by acquiring emotion information corresponding to the emotion of each of the first to n-th persons in the emotion information acquiring step; A notification corresponding to emotion statistical data, which is generated by another device connected to each terminal device through the network and which is generated by taking statistics of emotion information of the first to nth persons, is made on any one or more of the plurality of terminal devices. ,Electronic conferencing methods.

14. A program for causing a processor to execute the electronic conferencing method described in any one of claims 9 to 13.

15. A terminal device used to hold an electronic conference between a plurality of people via a network, a terminal control unit that acquires emotion information of a first person that is included in the plurality of people and is a user of the terminal device; a communication processing unit that transmits emotion information of the first person to the network so that a terminal device of a second person, the second person being included in the plurality of people and different from the first person, can notify the second person of the emotion information of the first person; The terminal control unit a first process of estimating an emotion of the first person based on emotion estimation information of the first person that is influenced by the emotion of the first person, and generating first candidate information relating to the emotion of the first person based on an estimation result; and a second process of generating second candidate information relating to the emotion of the first person based on the content of an emotion designation operation when an emotion designation operation is received from the first person at a terminal device of the first person, When the emotion designation operation is performed, the second candidate information generated in the second process is set as emotion information of the first person, whereas when the emotion designation operation is not performed, the first candidate information generated in the first process is set as emotion information of the first person. , terminal device.

16. A display unit that displays emotion information of the second person received via the network; and a microphone for transmitting the speech content of the first person to the second person; a speaker for transmitting the speech content of the second person to the first person. The terminal device according to claim 15.

17. A program for causing a processor provided in a terminal device to execute a terminal method, The terminal method includes an emotion information acquisition step of acquiring emotion information of a first person who is included in a plurality of people and is a user of the terminal device, and a transmission step of transmitting emotion information of the first person over a network to a terminal device of a second person who is included in the plurality of people and is different from the first person, so that the second person can be notified of the emotion information of the first person, In the emotion information acquisition step, a first process of estimating an emotion of the first person based on emotion estimation information of the first person that is influenced by the emotion of the first person, and generating first candidate information relating to the emotion of the first person based on an estimation result; and a second process of generating second candidate information relating to the emotion of the first person based on the content of an emotion designation operation when an emotion designation operation is received from the first person at a terminal device of the first person, When the emotion designation operation is performed, the second candidate information generated in the second process is set as emotion information of the first person, whereas when the emotion designation operation is not performed, the first candidate information generated in the first process is set as emotion information of the first person. ,program.

Images