Interactive communication support system and computer program

The two-way communication support system addresses the challenge of cluttered screens and difficult reaction recognition in video conferences by using state data analysis to provide intuitive graphical feedback, improving communication effectiveness.

JP7590866B2Active Publication Date: 2024-11-27SHARP KK
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2020219237
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-12-28
Publication Date
2024-11-27
Estimated Expiration
2040-12-28

AI Technical Summary

Technical Problem

Existing communication control devices struggle with participants in video conferences being unable to follow displayed text due to increased screen clutter as the number of participants grows, and speakers having difficulty recognizing listener reactions as more participants join, making it hard to maintain effective communication.

Method used

A two-way communication support system that transmits and analyzes state data, including image and audio analysis, to provide intuitive visual feedback on listener reactions, allowing speakers to easily recognize listener states through graphical and statistical representations.

Benefits of technology

Enables speakers to effortlessly monitor listener reactions and follow conference content by providing clear, graphical representations of listener states, reducing the need for manual responses and enhancing communication clarity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590866000001
    Figure 0007590866000001
  • Figure 0007590866000002
    Figure 0007590866000002
  • Figure 0007590866000003
    Figure 0007590866000003
Patent Text Reader

Abstract

To provide a bi-directional communication support system which allows a speaker to easily recognize the state of a listener.SOLUTION: A bi-directional communication support system SYS supports bi-directional communication between a first terminal device TM1 and a second terminal device TM2 that outputs an image and sound indicated by first state data D10 by receiving the first state data D10 that changes according to the state of a first user U1 existing on the side where the first terminal device TM1 is arranged from the first terminal device TM1. The bi-directional communication support system SYS comprises: a state specification unit 50 and an output control unit 203. The state specification unit 50 specifies the state of a second user U2 by analyzing the second state data D20 that changes according to the state of the second user U2 existing on the side where the second terminal device TM2 is arranged. The output control unit 203 causes the first terminal device TM1 to output analysis information AN2 indicating a specification result of the state of the second user U2 by the state specification unit 50.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an interactive communication support system and a computer program. [Background technology]

[0002] In the communication control device described in Patent Document 1, the volume level of a speaker's speech in a video conference is acquired. Then, if the volume level is lower than a threshold value, the communication control device performs voice recognition and transmits text data to other communication control devices. In the communication control device that receives the text data, the display text is displayed in an individual area corresponding to the communication control device that sent the text data. In this way, the text data is transmitted only when the speaker's voice is too quiet and the other party has difficulty hearing it. In other words, the text data is transmitted only when necessary, depending on the speaker's speech state. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-232780 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the communication control device described in Patent Document 1, text data showing the same content as the spoken content is generated. Therefore, display text based on the text data is displayed on the video conference screen of the communication control device that received the text data. Then, the participants of the video conference must read the sentences shown by the display text. In particular, the more participants in the video conference, the more display text is displayed on the video conference screen. Therefore, a situation may occur in which the participants are unable to follow the display text and are ultimately unable to read the document shown by the display text.

[0005] In addition, in the communication control device described in Patent Document 1, the contents of the speaker's speech are merely displayed as display text on the listener's video conference screen. Therefore, the speaker must recognize the listener's state (e.g., the listener's reaction) from the video conference screen. In particular, as the number of participants in a video conference increases, the number of participants displayed on the video conference screen also increases. Therefore, it is difficult for the speaker to recognize the state of all listeners from the video conference screen.

[0006] The present invention has been made in view of the above-mentioned problems, and has an object to provide a two-way communication support system and a computer program that enable a speaker to easily recognize the state of a listener. [Means for solving the problem]

[0007] According to a first aspect of the present invention, a two-way communication support system supports two-way communication between a first terminal device and a second terminal device that receives, from a first terminal device, first status data that changes depending on a status of a first user present on a side where a first terminal device is located, and outputs an image and a sound represented by the first status data. The two-way communication support system includes a status identification unit and an output control unit. The status identification unit analyzes the second status data that changes depending on a status of a second user present on a side where the second terminal device is located, to identify the status of the second user. The output control unit causes the first terminal device to output analysis information indicating a result of identification of the status of the second user by the status identification unit.

[0008] According to a second aspect of the present invention, a computer program causes a computer of a two-way communication support system that supports two-way communication between a first terminal device and a second terminal device that receives first status data from a first terminal device, the first status data changing depending on the status of a first user present at the side where the first terminal device is located, and outputs images and sounds indicated by the first status data, to execute the steps of analyzing the second status data that changes depending on the status of a second user present at the side where the second terminal device is located, to identify the status of the second user, and outputting analysis information indicating the identification result of the status of the second user to the first terminal device. Effect of the Invention

[0009] According to the present invention, it is possible to provide a two-way communication support system and a computer program that enable a speaker to easily recognize the state of a listener. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing an interactive communication support system according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram illustrating a second terminal device according to the first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating a first terminal device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing a screen displayed on the first terminal device according to the first embodiment. [Diagram 5] 6 is a diagram showing a screen displayed on the second terminal device in accordance with the first embodiment. FIG. [Figure 6] FIG. 6 is a diagram showing an example of the graphs shown in FIGS. 4 and 5. [Figure 7] FIG. 6 is a diagram showing another example of the graphs shown in FIGS. 4 and 5. [Figure 8] FIG. 6 is a diagram showing still another example of the graphs shown in FIGS. 4 and 5. [Figure 9] 6 is a flowchart showing a process of the second terminal device according to the first embodiment. [Figure 10]10 is a flowchart showing a reaction content identification process of the second terminal device according to the first embodiment. [Figure 11] 5 is a flowchart showing a process of the first terminal device according to the first embodiment. [Figure 12] FIG. 11 is a diagram illustrating a server device according to a second embodiment of the present invention. [Figure 13] 10 is a flowchart showing a process of a server device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are designated by the same reference characters and description thereof will not be repeated.

[0012] (Embodiment 1) The interactive communication support system SYS according to the first embodiment of the present invention will be described with reference to FIGS.

[0013] FIG. 1 is a diagram showing an interactive communication support system SYS. As shown in FIG. 1, the interactive communication support system SYS includes a first terminal device TM1 and at least one second terminal device TM2. In the example of FIG. 1, the interactive communication support system SYS includes a plurality of second terminal devices TM2. The interactive communication support system SYS supports interactive communication between the first terminal device TM1 and the second terminal device TM2. For example, the interactive communication support system SYS supports interactive communication in an online conference, an online seminar, or an online live broadcast between the first terminal device TM1 and the second terminal device TM2. The online conference is, for example, a video conference or a web conference.

[0014] The first terminal device TM1 and the multiple second terminal devices TM2 are connected to a network NW and communicate with each other via the network NW. The network NW includes, for example, the Internet, a LAN (Local Area Network), and a public telephone network.

[0015] Hereinafter, the user of the first terminal device TM1 will be referred to as the "first user U1," and the user of the second terminal device TM2 will be referred to as the "second user U2." The first user U1 is present on the side where the first terminal device TM1 is located. The second user U2 is present on the side where the second terminal device TM2 is located. For ease of understanding, the following will describe an example in which the first user U1 is the "speaker (speaker)" or "caller," and the second user U2 is the "listener" or "viewer."

[0016] The first terminal device TM1 transmits first status data (hereinafter, referred to as "first status data D10") to the second terminal device TM2. The first status data D10 is data that changes depending on the status of the first user U1.

[0017] The second terminal device TM2 receives the first status data D10 from the first terminal device TM1. Then, the second terminal device TM2 outputs the image and sound indicated by the first status data D10.

[0018] The second terminal device TM2 transmits second status data (hereinafter, referred to as "second status data D20") to the first terminal device TM1. The second status data D20 is data that changes depending on the status of the second user U2.

[0019] The first terminal device TM1 receives the second status data D20 from the second terminal device TM2. Then, the first terminal device TM1 outputs the image and sound indicated by the second status data D20.

[0020] Fig. 2 is a diagram showing a second terminal device TM2 according to the first embodiment. As shown in Fig. 2, the second terminal device TM2 includes a second control unit 20, a second operation unit 21, a second display unit 22, a second audio output unit 23, a second communication unit 24, a second state detection unit 25, and a second storage unit 26. The second control unit 20 includes a second display control unit 201, a second audio control unit 202, a second communication control unit 203, a state identification unit 50, and a specific audio detection unit 60. The state identification unit 50 includes an image analysis unit 501, an audio analysis unit 502, and a biometric data analysis unit 503.

[0021] The second control unit 20 controls the second operation unit 21, the second display unit 22, the second audio output unit 23, the second communication unit 24, the second state detection unit 25, and the second storage unit 26. The second control unit 20 includes a processor such as a CPU (Central Processing Unit).

[0022] The second operation unit 21 accepts operations from the second user U2. The second operation unit 21 includes, for example, a keyboard and a mouse. The second operation unit 21 may include a touch screen. In this case, the touch screen is disposed on top of the second display unit 22.

[0023] The second display unit 22 displays various information. The second display unit 22 is, for example, a liquid crystal display or an organic electroluminescence display. The size of the second display unit 22 is not particularly limited, and may be, for example, a size suitable for viewing by one second user U2 or a small number of second users U2, or a size suitable for viewing by a large number of second users U2.

[0024] The second audio output unit 23 outputs audio. The second audio output unit 23 includes, for example, a speaker.

[0025] The second communication unit 24 is a communication device, and includes, for example, a network interface controller (NIC) that communicates according to a predetermined communication protocol. The predetermined communication protocol is, for example, the TCP / IP (Transmission Control Protocol / Internet Protocol) protocol suite (i.e., the Internet Protocol Suite). The second communication unit 24 is connected to the network NW by wire or wirelessly.

[0026] The second state detection unit 25 detects the state of the second user U2. Then, the second state detection unit 25 outputs second state data D20 indicating the state of the second user U2 to the second control unit 20. The second state data D20 changes depending on the state of the second user U2. The second state data D20 includes image data D21, voice data D22, and biometric data D23.

[0027] Specifically, the second state detection unit 25 includes a second imaging unit 261, a second voice input unit 262, and a second biological information detection unit 263.

[0028] The second imaging section 261 captures an image of the second user U2 and outputs image data D21 indicating a captured image including an image of the second user U2 to the second control section 20. The captured image includes a facial image of the second user U2. The image indicated by the image data D21 changes according to the state of the second user U2. For example, the facial image indicates content according to the facial expression of the second user U2. The second imaging section 261 includes, for example, a camera.

[0029] The voice of the second user U2 is input to the second voice input unit 262. Then, the second voice input unit 262 outputs voice data D22 indicating the voice of the second user U2 to the second control unit 20. The voice based on the voice data D22 changes depending on the state of the second user U2. For example, the voice indicates the content according to the speech content and speech volume of the second user U2. The second voice input unit 262 includes, for example, a microphone and an AD converter.

[0030] The second biological information detection unit 263 detects the biological information of the second user U2 and outputs biological data D23 indicating the biological information to the second control unit 20. The biological information is, for example, body temperature, pulse, or blood pressure. Therefore, the second biological information detection unit 263 includes, for example, a thermometer, a pulse meter, or a blood pressure meter. The second biological information detection unit 263 is mounted on, for example, a smart watch. The biological information indicated by the biological data D23 changes depending on the state of the second user U2.

[0031] The second storage unit 26 includes a storage device and stores data and computer programs. Specifically, the second storage unit 26 includes a main storage device such as a semiconductor memory, and an auxiliary storage device such as a semiconductor memory, a solid-state drive, and / or a hard disk drive. The second storage unit 26 may include a removable medium. The second storage unit 26 corresponds to an example of a non-transitory computer-readable storage medium.

[0032] The processor of the second control unit 20 executes computer programs stored in the storage device of the second memory unit 26, and functions as a second display control unit 201, a second audio control unit 202, a second communication control unit 203, a state identification unit 50, and a specific audio detection unit 60.

[0033] The second display control unit 201 controls the second display unit 22. The second audio control unit 202 controls the second audio output unit 23. In addition, in response to a mute instruction from the second user U2 via the operation unit 61, the second audio control unit 202 mutes the audio based on the audio data D22 of the second user U2.

[0034] The second communication control unit 203 controls the second communication unit 24. The second communication control unit 203 corresponds to an example of an "output control unit" of the present invention in the second terminal device TM2.

[0035] The second communication control unit 203 controls the second communication unit 24 so that the second communication unit 24 transmits the second status data D20 to the first terminal device TM1. As a result, the second communication unit 24 transmits the second status data D20 to the first terminal device TM1. Specifically, the second communication unit 24 transmits image data D21 of the second user U2 to the first terminal device TM1. In addition, if the voice of the second user U2 is not muted, the second communication unit 24 transmits voice data D22 of the second user U2 to the first terminal device TM1.

[0036] On the other hand, the second communication unit 24 receives the first status data D10 from the first terminal device TM1. Specifically, the second communication unit 24 receives image data D11 of the first user U1 from the first terminal device TM1. Then, the second display control unit 201 controls the second display unit 22 so that the second display unit 22 displays an image based on the image data D11 of the first user U1. As a result, the second display unit 22 displays an image of the first user U1.

[0037] The second communication unit 24 also receives the voice data D12 of the first user U1 from the first terminal device TM1. The second voice control unit 202 then controls the second voice output unit 23 so that the second voice output unit 23 outputs a voice based on the voice data D12 of the first user U1. As a result, the second voice output unit 23 outputs the voice of the first user U1.

[0038] The state identification unit 50 analyzes the second state data D20 that changes depending on the state of the second user U2 to identify the state of the second user U2. The state of the second user U2 indicates a state that can be identified by analyzing at least one of the face, voice, and biometric information of the second user U2. The state of the second user U2 is, for example, the reaction of the second user U2. The reaction of the second user U2 is, for example, the reaction of the second user U2 to the speech of the first user U1. The "reaction" is, for example, a state in which the second user U2 is laughing, angry, reacting positively, reacting negatively, or unresponsive.

[0039] The second communication control unit 203 transmits analysis information (hereinafter, referred to as "analysis information AN2") indicating the result of identification of the state of the second user U2 by the state identification unit 50 to the first terminal device TM1 via the second communication unit 24, thereby displaying the analysis information AN2 on the first terminal device TM1 (the first display unit 12 in FIG. 3). In other words, the second communication control unit 203 outputs the analysis information AN2 indicating the result of identification of the state of the second user U2 by the state identification unit 50 to the first terminal device TM1 (the first display unit 12 in FIG. 3). Therefore, according to the first embodiment, the first user U1 who is the speaker can easily recognize the state of the second user U2 who is the listener (for example, the reaction of the second user U2) by looking at the analysis information AN2 on the first terminal device TM1. The reason why the state of the second user U2 can be easily recognized is because the analysis information AN2 indicates the result of identifying the state of the second user U2 (i.e., the result of interpreting the state of the second user U2), rather than the image and voice of the second user U2 itself.

[0040] Preferably, when the voice based on the voice data D22 of the second user U2 is muted, the state identification unit 50 identifies the state of the second user U2. When the voice based on the voice data D22 of the second user U2 is muted, the second communication control unit 203 transmits analysis information AN2 indicating the identification result by the state identification unit 50 to the first terminal device TM1 via the second communication unit 24, thereby displaying the analysis information AN2 on the first terminal device TM1 (the first display unit 12 in FIG. 3). In other words, when the voice based on the voice data D22 of the second user U2 is muted, the second communication control unit 203 outputs the analysis information AN2 indicating the identification result by the state identification unit 50 to the first terminal device TM1 (the first display unit 12 in FIG. 3).

[0041] This preferred example provides benefits to both the first user U1 who is the speaker and the second user U2 who is the listener.

[0042] For example, the second user U2, who is a listener, may mute his / her voice so that the first user U1, who is a speaker, does not hear the second user U2's voice. This is to prevent the listener's voice from disturbing the speaker's speech. In this case, if the voice of the second user U2 is muted, it may be difficult for the first user U1, who is a speaker, to recognize the reaction of the second user U2, who is a listener, from the video alone.

[0043] Therefore, in this preferred example, when the voice of the second user U2 is muted, analysis information AN2 indicating the identified state of the second user U2, who is the listener, is output (displayed) on the first terminal device TM1 of the first user U1, who is the speaker, so that the first user U1 can easily recognize the state of the second user U2 (e.g., the reaction of the second user U2) even when the voice of the second user U2 is muted.

[0044] On the other hand, it is bothersome for the second user U2, who is a listener, to repeatedly perform an operation to unmute just to give a brief response (e.g., "yes" or "no"). However, in this preferred example, the state of the second user U2 is transmitted to the first user U1 while the voice of the second user U2 remains muted. Therefore, the second user U2 is not required to perform an operation to unmute, and the bother of the second user U2 can be reduced.

[0045] In detail, the second state data D20 output by the second state detection unit 25 includes image data D21 generated by capturing an image of the face of the second user U2. The image analysis unit 501 of the state identification unit 50 analyzes the image data D21 of the second user U2 to identify the state of the second user U2 represented by the facial expression of the second user U2. Therefore, the state of the second user U2 can be identified with high accuracy by image analysis.

[0046] For example, the image analysis unit 501 recognizes a facial image of the second user U2 from the image data D21. In this case, as a facial recognition algorithm, for example, eigenfaces using principal component analysis, linear discriminant analysis, elastic bunch graph matching, hidden Markov model, or neuron-motivated dynamic link matching can be adopted. Then, the image analysis unit 501 analyzes the facial image to identify the facial expression of the second user U2. Specifically, the image analysis unit 501 extracts features from the facial image and analyzes the features to identify the facial expression of the second user U2. The features are, for example, eyebrows, eyes, and mouth. In this case, the image analysis unit 501 performs pattern matching based on the features to identify the facial expression of the second user U2.

[0047] For example, the image analysis unit 501 analyzes a facial image to determine whether the state of the second user U2 is "smiling", "angry", or "expressionless".

[0048] The image analysis unit 501 outputs analysis information AN2 indicating the state of the second user U2 identified by the facial expression of the second user U2. Hereinafter, the analysis information AN2 by the image analysis unit 501 may be referred to as "analysis information AN21."

[0049] Then, the second communication control unit 203 transmits the analysis information AN21 indicating the result of the identification of the state of the second user U2 by the image analysis unit 501 to the first terminal device TM1 via the second communication unit 24, thereby causing the first terminal device TM1 (the first display unit 12 in FIG. 3) to display the analysis information AN21. Therefore, according to the first embodiment, the first user U1 (speaker) can easily recognize the state of the second user U2 (listener) based on the image analysis by looking at the analysis information AN21 on the first terminal device TM1.

[0050] The second status data D20 output by the status identification unit 50 includes voice data D22 generated by inputting the voice of the second user U2. The voice analysis unit 502 of the status identification unit 50 analyzes the voice data D22 of the second user U2 to identify the status of the second user U2 represented by the voice of the second user U2. Therefore, the status of the second user U2 can be accurately identified by the voice analysis.

[0051] The voice analysis includes, for example, a volume level recognition process and a voice recognition process. In this case, the voice recognition algorithm may be, for example, a voice recognition process using a hidden Markov model, a voice recognition process using a statistical method, or a voice recognition process using a dynamic time warping method.

[0052] For example, the voice analysis unit 502 analyzes the voice of the second user U2 to determine whether the state of the second user U2 is "laughing state", "angry state", or "silent state". In this case, for example, the second storage unit 26 stores various voice patterns indicating various human states. Then, the voice analysis unit 502 performs pattern matching on the voice of the second user U2 to determine the state of the second user U2.

[0053] Also, for example, the voice analysis unit 502 analyzes the voice of the second user U2 to identify the degree of the state of the second user U2. In this case, for example, the voice analysis unit 502 identifies the degree of "laughing state", the degree of "angry state", or the degree of "silent state" based on the volume level.

[0054] The voice analysis unit 502 outputs analysis information AN2 indicating the state of the second user U2 identified by the voice of the second user U2. Hereinafter, the analysis information AN2 by the voice analysis unit 502 may be referred to as "analysis information AN22."

[0055] Then, the second communication control unit 203 transmits the analysis information AN22 indicating the result of the determination of the state of the second user U2 by the voice analysis unit 502 to the first terminal device TM1 via the second communication unit 24, thereby causing the first terminal device TM1 (the first display unit 12 in FIG. 3) to display the analysis information AN22. Therefore, according to the first embodiment, the first user U1 (speaker) can easily recognize the state of the second user U2 (listener) based on the voice analysis by looking at the analysis information AN22 on the first terminal device TM1.

[0056] On the other hand, the first status data D10 that the second communication unit 24 receives from the first terminal device TM1 includes voice data D12 generated by inputting the voice of the first user U1. Then, the specific voice detection unit 60 analyzes the voice data D12 of the first user U1 to detect a voice indicating a question from the first user U1. The voice analysis includes, for example, a voice recognition process. In this case, as the voice recognition algorithm, for example, voice recognition by a hidden Markov model, voice recognition by a statistical method, or voice recognition by a dynamic time warping method can be adopted. Furthermore, in this case, for example, the second storage unit 26 stores a voice pattern indicating a question from a human being. Then, the voice analysis unit 502 performs pattern matching on the voice of the first user U1 to detect a voice indicating a question from the first user U1.

[0057] Then, when a voice indicating a question from the first user U1 is detected, the voice analysis unit 502 analyzes the voice data D22 of the second user U2 to identify the content of the reaction of the second user U2 to the question from the first user U1.

[0058] For example, the voice analysis unit 502 analyzes the voice data D22 of the second user U2 to identify whether the reaction of the second user U2 is a positive reaction (e.g., a voice indicating "yes"), a negative reaction (e.g., a voice indicating "no"), or no reaction (e.g., "silence"). In this case, for example, the second storage unit 26 stores various voice patterns indicating human reactions to a question. Then, the voice analysis unit 502 performs pattern matching on the voice of the second user U2 to identify the content of the reaction of the second user U2.

[0059] Then, the second communication control unit 203 causes the first terminal device TM1 (first display unit 12 in FIG. 3) to display the analyzed information AN22, which includes the identification result of the reaction content of the second user U2, by transmitting the analyzed information AN22, which includes the identification result of the reaction content of the second user U2, to the first terminal device TM1 via the second communication unit 24. In other words, the second communication control unit 203 causes the first terminal device TM1 to output the analyzed information AN22, which includes the identification result of the reaction content of the second user U2.

[0060] Therefore, according to the first embodiment, the first user U1 can easily recognize the content of the reaction of the second user U2 to the question by looking at the analysis information AN22 on the first terminal device TM1.

[0061] Moreover, the second status data D20 output by the status identification unit 50 includes biometric data D23 indicating biometric information of the second user U2. The biometric data analysis unit 503 of the status identification unit 50 analyzes the biometric data D23 of the second user U2 to identify the status of the second user U2 represented by the biometric information of the second user U2. Therefore, the status of the second user U2 can be accurately identified by the biometric information.

[0062] The biometric data analysis unit 503 outputs analysis information AN2 indicating the state of the second user U2 identified by the biometric information of the second user U2. Hereinafter, the analysis information AN2 by the biometric data analysis unit 503 may be referred to as "analysis information AN23."

[0063] Then, the second communication control unit 203 transmits the analysis information AN23 indicating the result of the identification of the state of the second user U2 by the biometric data analysis unit 503 to the first terminal device TM1 via the second communication unit 24, thereby causing the first terminal device TM1 (the first display unit 12 in FIG. 3) to display the analysis information AN23. Therefore, according to the first embodiment, the first user U1 (speaker) can easily recognize the state of the second user U2 (listener) based on the biometric information by looking at the analysis information AN23 on the first terminal device TM1.

[0064] Next, the first terminal device TM1 will be described with reference to Fig. 3. Fig. 3 is a diagram showing the first terminal device TM1. As shown in Fig. 3, the first terminal device TM1 includes a first control unit 10, a first operation unit 11, a first display unit 12, a first audio output unit 13, a first communication unit 14, a first state detection unit 15, and a first storage unit 16. The first control unit 10 includes a first display control unit 101, a first audio control unit 102, a first communication control unit 103, and a statistics calculation unit 70.

[0065] The first control unit 10 controls the first operation unit 11, the first display unit 12, the first audio output unit 13, the first communication unit 14, the first state detection unit 15, and the first storage unit 16. Otherwise, the hardware configuration of the first control unit 10 is similar to the hardware configuration of the second control unit 20 in FIG.

[0066] The first operation unit 11 accepts operations from the first user U1. Otherwise, the hardware configuration of the first operation unit 11 is similar to the hardware configuration of the second operation unit 21 in FIG.

[0067] The first display unit 12 displays various information. Otherwise, the hardware configuration of the first display unit 12 is similar to the hardware configuration of the second display unit 22 in FIG.

[0068] The first audio output unit 13 outputs audio. Otherwise, the hardware configuration of the first audio output unit 13 is similar to the hardware configuration of the second audio output unit 23 in FIG.

[0069] The first communication unit 14 is a communication device, and includes, for example, a network interface controller (NIC) that performs communication according to a predetermined communication protocol. The first communication unit 14 is connected to the network NW by wire or wirelessly. The other hardware configurations of the first communication unit 14 are similar to the hardware configuration of the second communication unit 24 in FIG. 2.

[0070] The first state detection unit 15 detects the state of the first user U1. Then, the first state detection unit 15 outputs first state data D10 indicating the state of the first user U1 to the first control unit 10. The first state data D10 changes depending on the state of the first user U1. The first state data D10 includes image data D11, voice data D12, and biometric data D13.

[0071] Specifically, the first state detection unit 15 includes a first imaging unit 151, a first voice input unit 152, and a first biological information detection unit 153.

[0072] The first imaging unit 151 captures an image of the first user U1 and outputs image data D11 indicating a captured image including an image of the first user U1 to the first control unit 10. The captured image includes a facial image of the first user U1. The image indicated by the image data D11 changes according to the state of the first user U1. For example, the facial image indicates content according to the facial expression of the first user U1. The first imaging unit 151 includes, for example, a camera.

[0073] The voice of the first user U1 is input to the first voice input unit 152. Then, the first voice input unit 152 outputs voice data D12 indicating the voice of the first user U1 to the first control unit 10. The voice based on the voice data D12 changes depending on the state of the first user U1. For example, the voice indicates content according to the speech content and speech volume of the first user U1. The first voice input unit 152 includes, for example, a microphone and an AD converter.

[0074] The first biological information detection unit 153 detects the biological information of the first user U1 and outputs biological data D13 indicating the biological information to the first control unit 10. The biological information is, for example, body temperature, pulse, or blood pressure. Therefore, the first biological information detection unit 153 includes, for example, a thermometer, a pulse meter, or a blood pressure meter. The first biological information detection unit 153 is mounted on, for example, a smart watch. The biological information indicated by the biological data D13 changes depending on the state of the first user U1.

[0075] The first storage unit 16 includes a storage device and stores data and computer programs. The first storage unit 16 corresponds to an example of a non-transitory computer-readable storage medium. Otherwise, the hardware configuration of the first storage unit 16 is similar to the hardware configuration of the second storage unit 26 in FIG. 2.

[0076] The processor of the first control unit 10 executes a computer program stored in the storage device of the first memory unit 16, and functions as a first display control unit 101, a first audio control unit 102, a first communication control unit 103, and a statistical calculation unit 70.

[0077] The first display control unit 101 controls the first display unit 12. The first audio control unit 102 controls the first audio output unit 13. Otherwise, the configuration of the first audio output unit 13 is similar to the configuration of the second audio output unit 23 in FIG.

[0078] The first display control unit 101 in the first terminal device TM1 corresponds to an example of an "output control unit" of the present invention.

[0079] The first communication control unit 103 controls the first communication unit 14. Specifically, the first communication control unit 103 controls the first communication unit 14 so that the first communication unit 14 transmits the first status data D10 to the second terminal device TM2. As a result, the first communication unit 14 transmits the first status data D10 to the second terminal device TM2. Specifically, the first communication unit 14 transmits image data D11 and voice data D12 of the first user U1 to the second terminal device TM2.

[0080] On the other hand, the first communication unit 14 receives the second status data D20 from the second terminal device TM2. Specifically, the first communication unit 14 receives image data D21 of the second user U2 from the second terminal device TM2. Then, the first display control unit 101 controls the first display unit 12 so that the first display unit 12 displays an image based on the image data D21 of the second user U2. As a result, the first display unit 12 displays an image of the second user U2.

[0081] Furthermore, the first communication unit 14 receives the voice data D22 of the second user U2 from the second terminal device TM2 when the voice of the second user U2 is not muted. Then, when the voice of the second user U2 is not muted, the first voice control unit 102 controls the first voice output unit 13 so that the first voice output unit 13 outputs voice based on the voice data D22 of the second user U2. As a result, when the voice of the second user U2 is not muted, the first voice output unit 13 outputs the voice of the second user U2.

[0082] The first communication unit 14 also receives a plurality of pieces of analysis information AN2 from a plurality of second terminal devices TM2. The plurality of pieces of analysis information AN2 correspond to a plurality of second users U2, respectively. Each of the plurality of pieces of analysis information AN2 indicates a result of identifying the state of the corresponding second user U2. The analysis information AN2 by the state identification unit 50 of the second terminal device TM2 includes analysis information AN21 by the image analysis unit 501 of the second terminal device TM2, analysis information AN22 by the voice analysis unit 502 of the second terminal device TM2, and analysis information AN23 by the biometric data analysis unit 503 of the second terminal device TM2.

[0083] Then, the first display control unit 101 controls the first display unit 12 to display a plurality of pieces of analysis information AN2. As a result, the first display unit 12 displays a plurality of pieces of analysis information AN2. In other words, the first display control unit 101 causes the first display unit 12 to output a plurality of pieces of analysis information AN2. Therefore, according to the first embodiment, the first user U1 who is the speaker can easily recognize the state of the second user U2 who is the listener (for example, the reaction of the second user U2) by looking at the analysis information AN2 on the first display unit 12.

[0084] Preferably, the statistics calculation unit 70 calculates statistics of a plurality of pieces of analysis information AN2 each indicating a result of identifying the state of a plurality of second users U2. The statistics indicate a distribution, tendency, or characteristic of the plurality of pieces of analysis information AN2 constituting a group. For example, the statistics indicate a distribution, tendency, or characteristic of the content of reactions of a plurality of second users U2 to an utterance of the first user U1.

[0085] Then, the first display control unit 101 causes the first display unit 12 to display statistical information (hereinafter, referred to as "statistical information ST") indicating the statistical results by the statistical calculation unit 70. In other words, the first display control unit 101 causes the first display unit 12 of the first terminal device TM1 to output the statistical information ST indicating the statistical results by the statistical calculation unit 70. Therefore, according to the first embodiment, the first user U1 who is a speaker can intuitively recognize the state of the multiple second users U2 who are listeners (for example, the reactions of the multiple second users U2) by looking at the statistical information ST on the first display unit 12. For example, the statistical information ST indicates the distribution, tendency, or characteristics of the reaction contents of the multiple second users U2 to the utterance of the first user U1.

[0086] Moreover, the first communication control unit 103 transmits the statistical information ST indicating the statistical result by the statistical calculation unit 70 to the second terminal device TM2 via the first communication unit 14, thereby displaying the statistical information ST on the second terminal device TM2 (the second display unit 22). In other words, the first communication control unit 103 outputs the statistical information ST indicating the statistical result by the statistical calculation unit 70 to the second terminal device TM2 (the second display unit 22). Therefore, according to the first embodiment, the second user U2 who is a listener can intuitively recognize the state of the other second user U2 who is a listener (for example, the reaction of the other second user U2) by looking at the statistical information ST on the second terminal device TM2. In this case, the first communication control unit 103 corresponds to an example of an "output control unit" of the present invention in the first terminal device TM1.

[0087] More preferably, the statistics calculation unit 70 calculates statistics of the plurality of pieces of analysis information AN2 and creates a graph (hereinafter, referred to as "graph GP") as statistical information ST showing the statistical results. Then, the first display control unit 115 displays the graph GP on the first display unit 12 of the first terminal device TM1. Therefore, according to the first embodiment, the first user U1 who is the speaker can more intuitively recognize the state of the plurality of second users U2 who are listeners (for example, the reactions of the plurality of second users U2) by looking at the graph GP on the first display unit 12. For example, the graph GP shows the distribution, tendency, or characteristics of the reaction contents of the plurality of second users U2 to the utterance of the first user U1.

[0088] Further, the first communication control unit 103 transmits the graph GP indicating the statistical results of the plurality of pieces of analysis information AN2 to the second terminal device TM2 via the first communication unit 14, thereby displaying the graph GP on the second terminal device TM2 (the second display unit 22). In other words, the first communication control unit 103 outputs the graph GP indicating the statistical results of the plurality of pieces of analysis information AN2 to the second terminal device TM2 (the second display unit 22). Therefore, according to the first embodiment, the second user U2 who is a listener can further intuitively recognize the state of the other second user U2 who is a listener (for example, the reaction of the other second user U2) by looking at the graph GP on the second terminal device TM2. In this case, the first communication control unit 103 corresponds to an example of the "output control unit" of the present invention in the first terminal device TM1.

[0089] Next, with reference to FIGS. 4 to 6, examples of the graph GP created by the statistics calculation section 70 will be shown.

[0090] Fig. 4 is a diagram showing a screen SC1 displayed on the first terminal device TM1. As shown in Fig. 4, the first display unit 12 of the first terminal device TM1 displays the screen SC1. The screen SC1 includes images SU of a plurality of second users U2 and a graph GP.

[0091] Fig. 5 is a diagram showing a screen SC2 displayed on the second terminal device TM2. As shown in Fig. 5, the second display unit 22 of each of the multiple second terminal devices TM2 displays a screen SC2. The screen SC2 includes an image FU of the first user U1 and a graph GP.

[0092] Fig. 6 is a diagram showing an example of the graph GP shown in Fig. 4 and Fig. 5 (hereinafter, referred to as "graph GP1"). As shown in Fig. 6, graph GP1 shows the relationship between the volume of laughter of second users U2 and the number of people. In graph GP1, the horizontal axis shows the volume of laughter, and the vertical axis shows the number of second users U2 who emitted each volume.

[0093] Fig. 7 is a diagram showing another example of the graph GP shown in Figs. 4 and 5 (hereinafter referred to as "graph GP2"). As shown in Fig. 7, graph GP2 shows the distribution of joy, anger, sadness, and happiness of multiple second users U2. The horizontal axis shows the degree of emotion from the state "sad" to the state "happy". The vertical axis shows the degree of emotion from the state "happy" to the state "anger".

[0094] FIG. 8 is a diagram showing yet another example of the graph GP (hereinafter, referred to as "graph GP3") shown in FIGS. 4 and 5. As shown in FIG. 8, graph GP3 shows the relationship between the content of the reaction of the second user U2 to the utterance of the first user U1 and the number of users. The horizontal axis shows the content of the reaction. On the horizontal axis, "positive reaction" such as "yes", "negative reaction" such as "no", and "other" such as "no reaction" are displayed as the content of the reaction. The vertical axis shows the number of second users U2 who made each reaction.

[0095] As described above with reference to Figures 4 to 8, according to embodiment 1, the first user U1, who is the speaker, can intuitively recognize the reactions of multiple second users U2, who are listeners, by looking at the graph GP on the first display unit 12.

[0096] Moreover, the second user U2 who is a listener can intuitively recognize the reaction of the other second users U2 who are listeners by looking at the graph GP on the second display unit 22.

[0097] Next, the processing of the second terminal device TM2 will be described with reference to Fig. 2 and Fig. 9. Fig. 9 is a flowchart showing the processing of the second terminal device TM2. As shown in Fig. 9, the processing of the second terminal device TM2 includes steps S1 to S14. Specifically, a computer program stored in the second storage unit 26 of the second terminal device TM2 causes the second control unit 20 of the second terminal device TM2 to execute the processing of steps S1 to S14.

[0098] 2 and 9, first, in step S1, the second communication control unit 203 starts receiving image data D11 and voice data D12 of the first user U1 from the first terminal device TM1 via the second communication unit 24. Then, the second display control unit 201 causes the second display unit 22 to display an image based on the image data D11 of the first user U1. In addition, the second voice control unit 202 causes the second voice output unit 23 to output voice based on the voice data D12 of the first user U1.

[0099] Next, in step S2, the second voice control unit 202 mutes the voice based on the voice data D22 of the second user U2 in response to a mute instruction from the second user U2.

[0100] Next, in step S3, the second communication control unit 203 starts transmitting the image data D21 of the second user U2 via the second communication unit 24 to the first terminal device TM1.

[0101] Next, in step S4, the image analysis unit 501 starts image analysis of the image data D21 output by the second imaging unit 261.

[0102] Next, in step S5, the image analysis unit 501 determines whether or not a face image has been recognized from the image represented by the image data D21.

[0103] If it is determined in step S5 that the facial image cannot be recognized, the process proceeds to step S7.

[0104] On the other hand, if it is determined in step S5 that the face image has been recognized, the process proceeds to step S6.

[0105] Next, in step S6, the image analysis unit 501 analyzes the facial expression of the second user U2 based on the face image. Then, the image analysis unit 501 stores analysis information AN21 indicating the analysis result of the facial expression of the second user U2 in the second storage unit 26. The analysis information AN21 indicates the identification result of the state of the second user U2 based on the image.

[0106] Next, in step S7, the voice analysis unit 502 determines whether or not a voice has been input to the second voice input unit 262.

[0107] If it is determined in step S7 that no voice is being input, the process proceeds to step S9.

[0108] On the other hand, if it is determined in step S7 that a voice has been input, the process proceeds to step S8.

[0109] Next, in step S8, the voice analysis unit 502 performs voice analysis on the voice data D22 indicating the voice input to the second voice input unit 262. Then, the voice analysis unit 502 stores analysis information AN22 indicating the analysis result of the voice of the second user U2 in the second storage unit 26. The analysis information AN22 indicates the result of identifying the state of the second user U2 based on the voice.

[0110] Next, in step S9, the biometric data analysis section 503 determines whether or not the biometric data D23 has been input from the second biometric information detection section 263.

[0111] If it is determined in step S9 that the biometric data D23 has not been input, the process proceeds to step S11.

[0112] On the other hand, if it is determined in step S9 that the biometric data D23 has been input, the process proceeds to step S10.

[0113] Next, in step S10, the biometric data analysis unit 503 executes an analysis process on the biometric data D23. Then, the biometric data analysis unit 503 stores analysis information AN23 indicating the analysis result of the biometric data D23 of the second user U2 in the second storage unit 26. The analysis information AN23 indicates the result of identifying the state of the second user U2 based on the biometric information.

[0114] Next, in step S11, the second communication control unit 203 transmits the analyzed information AN2 to the first terminal device TM1 via the second communication unit 24. The analyzed information AN2 includes the analyzed information AN21, the analyzed information AN22, and the analyzed information AN23.

[0115] Next, in step S12, the second communication control unit 203 receives statistical information ST from the first terminal device TM1 via the second communication unit 24. The statistical information ST indicates statistical results of a plurality of pieces of analysis information AN2 corresponding to a plurality of second users U2. Specifically, the second communication control unit 203 receives a graph GP as the statistical information ST from the first terminal device TM1 via the second communication unit 24.

[0116] Next, in step S13, the second display control unit 201 causes the statistical information ST to be displayed on the second display unit 22. Specifically, the second display control unit 201 causes the graph GP to be displayed on the second display unit 22. In this case, the second display control unit 201 corresponds to an example of the "output control unit" of the present invention in the second terminal device TM2.

[0117] Next, in step S14, the second communication control unit 203 determines whether or not the communication between the second terminal device TM2 and the first terminal device TM1 has ended.

[0118] If it is determined in step S14 that the communication has not ended, the process proceeds to step S4.

[0119] On the other hand, if it is determined in step S14 that the communication has ended, the process ends.

[0120] Next, the reaction content identification process of the second terminal device TM2 will be described with reference to Fig. 2 and Fig. 10. Fig. 10 is a flowchart showing the reaction content identification process of the second terminal device TM2. As shown in Fig. 10, the reaction content identification process includes steps S21 to S24. Specifically, a computer program stored in the second storage unit 26 of the second terminal device TM2 causes the second control unit 20 of the second terminal device TM2 to execute the processes of steps S21 to S24.

[0121] 2 and 10, first, in step S21, the specific voice detection unit 60 executes a question detection process based on the voice data D12 of the first user U1. The question detection process indicates a process of analyzing the voice data D12 of the first user U1 and detecting a voice indicating a question from the first user U1 from the voice data D12 of the first user U1.

[0122] Next, in step S22, the voice analysis unit 502 determines whether or not the specific voice detection unit 60 has detected a voice indicating a question.

[0123] If it is determined in step S22 that a voice indicating a question has not been detected, the process ends.

[0124] On the other hand, if it is determined in step S22 that a voice indicating a question has been detected, the process proceeds to step S23.

[0125] Next, in step S23, the voice analysis unit 502 analyzes the voice data D22 of the second user U2 to identify the content of the reaction of the second user U2 to the question. Then, the voice analysis unit 502 stores analysis information AN22 including the identification result of the content of the reaction of the second user U2 in the second storage unit 26.

[0126] Next, in step S24, the second communication control unit 203 transmits the analysis information AN22 to the first terminal device TM1 via the second communication unit 24. Then, the process ends.

[0127] Next, the processing of the first terminal device TM1 will be described with reference to Fig. 3 and Fig. 11. Fig. 11 is a flowchart showing the processing of the first terminal device TM1. As shown in Fig. 11, the processing of the first terminal device TM1 includes steps S31 to S37. Specifically, a computer program stored in the first storage unit 16 of the first terminal device TM1 causes the first control unit 10 of the first terminal device TM1 to execute the processing of steps S31 to S37.

[0128] As shown in Figures 3 and 11, first, in step S31, the first communication control unit 103 starts transmitting image data D11 and voice data D12 of the first user U1 to multiple second terminal devices TM2 via the first communication unit 14.

[0129] Next, in step S32, the first communication control unit 103 starts receiving image data D21 of the second users U2 from the second terminal devices TM2 via the first communication unit 14. Then, the first display control unit 101 causes the first display unit 12 to display an image based on the image data D21 of each of the second users U2.

[0130] Next, in step S33, the first communication control unit 103 receives a plurality of pieces of analyzed information AN2 from a plurality of second terminal devices TM2 via the first communication unit 14. The analyzed information AN2 indicates the result of identifying the state of the second user U2. The analyzed information AN2 includes analyzed information AN21, analyzed information AN22, and analyzed information AN23.

[0131] Next, in step S34, the statistics calculation unit 70 calculates statistics of the plurality of pieces of analysis information AN2 indicating the results of identifying the states of the plurality of second users U2, and generates statistical information ST indicating the statistical results. Specifically, the statistics calculation unit 70 creates a graph GP as the statistical information ST.

[0132] Next, in step S35, the first display control unit 101 causes the statistical information ST to be displayed on the first display unit 12. Specifically, the first display control unit 101 causes the first display unit 12 to display a graph GP.

[0133] Next, in step S36, the first communication control unit 103 transmits the statistical information ST to each second terminal device TM2 via the first communication unit 14. Specifically, the first communication control unit 103 transmits the graph GP to each second terminal device TM2 via the first communication unit 14.

[0134] Next, in step S37, the first communication control unit 103 determines whether or not the communication between the first terminal device TM1 and the second terminal device TM2 has ended.

[0135] If it is determined in step S37 that the communication has not ended, the process proceeds to step S33.

[0136] On the other hand, if it is determined in step S37 that the communication has ended, the process ends.

[0137] (Embodiment 2) Next, an interactive communication support system SYS according to a second embodiment of the present invention will be described with reference to Fig. 1 to Fig. 3, Fig. 12, and Fig. 13. In the interactive communication support system SYS according to the second embodiment, the second embodiment is mainly different from the first embodiment in that the server device SV shown in Fig. 1 includes a state identification unit 50 in Fig. 2, a specific voice detection unit 60 in Fig. 2, and a statistics calculation unit 70 in Fig. 3. Below, the differences between the second embodiment and the first embodiment will be mainly described.

[0138] 1, in the second embodiment, the two-way communication support system SYS includes a first terminal device TM1, at least one second terminal device TM2, and a server device SV. The server device SV mediates communication between the first terminal device TM1 and each of the second terminal devices TM2.

[0139] Moreover, the configuration of the first terminal device TM1 according to the second embodiment is similar to the configuration of the first terminal device TM1 in Fig. 3. However, the first terminal device TM1 according to the second embodiment does not include the statistics calculation unit 70 in Fig. 3.

[0140] Furthermore, the configuration of the second terminal device TM2 according to the second embodiment is similar to the configuration of the second terminal device TM2 in Fig. 2. However, the second terminal device TM2 according to the second embodiment does not include the state identification unit 50 and the specific voice detection unit 60 in Fig. 2.

[0141] Fig. 12 is a diagram showing a server device SV according to embodiment 2 of the present invention. As shown in Fig. 12, the server device SV includes a control unit 30, a communication unit 31, and a storage unit 32. The control unit 30 includes a communication control unit 301, a voice control unit 302, a speech terminal identification unit 40, a state identification unit 50, a specific voice detection unit 60, and a statistics calculation unit 70.

[0142] The control unit 30 controls the communication unit 31 and the storage unit 32. Otherwise, the hardware configuration of the control unit 30 is similar to the hardware configuration of the second control unit 20 in FIG.

[0143] The communication unit 31 is a communication device, and includes, for example, a network interface controller (NIC) that performs communication according to a predetermined communication protocol. The communication unit 31 is connected to the network NW by wire or wirelessly. Otherwise, the hardware configuration of the communication unit 31 is similar to the hardware configuration of the second communication unit 24 in FIG. 2.

[0144] The communication unit 31 receives the first status data D10 from the first terminal device TM1. Specifically, the communication unit 31 receives the image data D11 and the voice data D12 of the first user U1 from the first terminal device TM1.

[0145] The communication unit 31 receives the second status data D20 from the second terminal device TM2. Specifically, the communication unit 31 receives the image data D21 and the biometric data D23 of the second user U2 from the second terminal device TM2. The communication unit 31 also receives the voice data D22 of the second user U2 from the second terminal device TM2 regardless of whether the voice of the second user U2 is muted. That is, the communication unit 31 shown in FIG. 2 transmits the voice data D22 of the second user U2 to the server device SV regardless of whether the voice of the second user U2 is muted.

[0146] The storage unit 32 includes a storage device and stores data and computer programs. The storage unit 32 corresponds to an example of a non-transitory computer-readable storage medium. Otherwise, the hardware configuration of the storage unit 32 is similar to the hardware configuration of the second storage unit 26 in FIG. 2.

[0147] The processor of the control unit 30 executes computer programs stored in the storage device of the memory unit 32, and functions as a communication control unit 301, a voice control unit 302, a speech terminal identification unit 40, a state identification unit 50, a specific voice detection unit 60, and a statistical calculation unit 70.

[0148] The communication control unit 301 controls the communication unit 31. The communication control unit 301 corresponds to an example of the "output control unit" of the present invention in the server device SV.

[0149] Specifically, the communication control unit 301 controls the communication unit 31 so that the communication unit 31 transmits the first status data D10 to the second terminal device TM2. As a result, the communication unit 31 transmits the first status data D10 to the second terminal device TM2. Specifically, the communication unit 31 transmits the image data D11 and the voice data D12 of the first user U1 to the second terminal device TM2.

[0150] Furthermore, the communication control unit 301 controls the communication unit 31 so that the communication unit 31 transmits the second status data D20 to the first terminal device TM1. As a result, the communication unit 31 transmits the second status data D20 to the first terminal device TM1. Specifically, the communication unit 31 transmits image data D21 of the second user U2 to the first terminal device TM1.

[0151] Furthermore, the audio control unit 302 receives an instruction to mute the audio based on the audio data D22 of the second user U2 from the second terminal device TM2 via the communication unit 31. Accordingly, the audio control unit 302 mutes the audio based on the audio data D22 of the second user U2 in response to the mute instruction. Then, when the audio of the second user U2 is muted, the communication control unit 301 prohibits the audio data D22 of the second user U2 from being transmitted to the first terminal device TM1. On the other hand, when the audio of the second user U2 is not muted, the communication unit 31 transmits the audio data D22 of the second user U2 to the first terminal device TM1.

[0152] The utterance terminal identification unit 40 identifies the first terminal device TM1 of the first user U1 who is the speaker (speaker). For example, the utterance terminal identification unit 40 identifies the IP address of the first terminal device TM1.

[0153] The operation of the condition identification unit 50 (image analysis unit 501, voice analysis unit 502, and biometric data analysis unit 503) according to the second embodiment is similar to the operation of the condition identification unit 50 (image analysis unit 501, voice analysis unit 502, and biometric data analysis unit 503) in Fig. 2. Moreover, the operation of the specific voice detection unit 60 according to the second embodiment is similar to the operation of the specific voice detection unit 60 in Fig. 2.

[0154] For example, the state identifying unit 50 analyzes the second state data D20 of the second user U2 to identify the state of the second user U2.

[0155] Then, the communication control unit 301 transmits the analysis information AN2 indicating the result of the identification of the state of the second user U2 by the state identification unit 50 to the first terminal device TM1 via the communication unit 31, thereby causing the first terminal device TM1 to display the analysis information AN2. In other words, the communication control unit 301 causes the first terminal device TM1 to output the analysis information AN2. Therefore, according to the second embodiment, the first user U1 who is the speaker can easily recognize the state of the second user U2 who is the listener (for example, the reaction of the second user U2) by looking at the analysis information AN2 on the first terminal device TM1.

[0156] Preferably, when the voice based on the voice data D22 of the second user U2 is muted, the state identification unit 50 identifies the state of the second user U2. Then, when the voice based on the voice data D22 of the second user U2 is muted, the communication control unit 301 transmits the analysis information AN2 to the first terminal device TM1 via the communication unit 31, thereby causing the first terminal device TM1 to display the analysis information AN2. In other words, when the voice based on the voice data D22 of the second user U2 is muted, the second communication control unit 203 causes the first terminal device TM1 to output the analysis information AN2. The advantages of this preferred example are the same as those of the first embodiment.

[0157] Also, similarly to the first embodiment, for example, the image analysis unit 501 analyzes the image data D21 of the second user U2 to identify the state of the second user U2 represented by the facial expression of the second user U2. Then, the communication control unit 301 transmits analysis information AN21 indicating the result of the identification of the state of the second user U2 by the image analysis unit 501 to the first terminal device TM1, thereby causing the first terminal device TM1 to display the analysis information AN21.

[0158] Furthermore, similarly to the first embodiment, for example, the voice analysis unit 502 analyzes the voice data D22 of the second user U2 to identify the state of the second user U2 represented by the voice of the second user U2. Then, the communication control unit 301 transmits analysis information AN22 indicating the result of the identification of the state of the second user U2 by the voice analysis unit 502 to the first terminal device TM1, thereby causing the first terminal device TM1 to display the analysis information AN22.

[0159] Also, as in the first embodiment, for example, the specific voice detection unit 60 analyzes the voice data D12 of the first user U1 to detect a voice indicating a question from the first user U1. Then, when a voice indicating a question from the first user U1 is detected, the voice analysis unit 502 analyzes the voice data D22 of the second user U2 to identify the reaction content of the second user U2 to the question from the first user U1. Furthermore, the communication control unit 301 transmits the analysis information AN22 including the identification result of the reaction content of the second user U2 to the first terminal device TM1, thereby displaying the analysis information AN22 on the first terminal device TM1. In other words, the communication control unit 301 outputs the analysis information AN22 including the identification result of the reaction content of the second user U2 to the first terminal device TM1.

[0160] Furthermore, similarly to the first embodiment, for example, the biometric data analysis unit 503 analyzes the biometric data D23 of the second user U2 to identify the state of the second user U2 represented by the biometric information of the second user U2. Then, the communication control unit 301 transmits analysis information AN23 indicating the result of the identification of the state of the second user U2 by the biometric data analysis unit 503 to the first terminal device TM1, thereby causing the first terminal device TM1 to display the analysis information AN23.

[0161] Moreover, the operation of the statistics calculation unit 70 according to the second embodiment is similar to the operation of the statistics calculation unit 70 in FIG. 3. For example, similar to the first embodiment, the statistics calculation unit 70 calculates statistics of a plurality of pieces of analysis information AN2 each indicating a result of identifying the state of a plurality of second users U2. Then, the communication control unit 301 transmits the statistical information ST indicating the statistical result by the statistics calculation unit 70 to the first terminal device TM1 and each second terminal device TM2 via the communication unit 31, thereby displaying the statistical information ST on the first terminal device TM1 and each second terminal device TM2. In other words, the communication control unit 301 outputs the statistical information ST to the first terminal device TM1 and each second terminal device TM2. Therefore, according to the second embodiment, the first user U1 who is the speaker can intuitively recognize the state of the plurality of second users U2 who are the listeners (for example, the reaction of the plurality of second users U2) by looking at the statistical information ST on the first terminal device TM1. Furthermore, the second user U2 who is a listener can intuitively recognize the state of the other second users U2 who are listeners (for example, the reaction of the other second users U2) by looking at the statistical information ST on the second terminal device TM2.

[0162] Preferably, as in the first embodiment, for example, the statistics calculation section 70 calculates statistics of a plurality of pieces of analysis information AN2 and creates a graph GP as statistical information ST indicating the statistical results.

[0163] Next, the processing of the server device SV will be described with reference to Fig. 12 and Fig. 13. Fig. 13 is a flowchart showing the processing of the server device SV. As shown in Fig. 13, the processing of the server device SV includes steps S41 to S53. Specifically, a computer program stored in the storage unit 32 of the server device SV causes the control unit 30 of the server device SV to execute the processing of steps S41 to S53.

[0164] As shown in FIGS. 12 and 13, first, in step S41, the communication control unit 301 starts receiving the image data D11 and the voice data D12 of the first user U1 from the first terminal device TM1 via the communication unit 31.

[0165] Next, in step S42, the utterance terminal identification unit 40 identifies the first terminal device TM1 of the first user U1 who is the speaker (speaker).

[0166] Next, in step S43, the communication control unit 301 starts transmitting the image data D11 and the voice data D12 of the first user U1 via the communication unit 31 to the multiple second terminal devices TM2.

[0167] Next, in step S44, the communication control unit 301 starts receiving image data D21 and voice data D22 of the second user U2 from each of the multiple second terminal devices TM2 via the communication unit 31. In addition, the communication control unit 301 starts receiving biometric data D23 of the second user U2 from each of the multiple second terminal devices TM2 via the communication unit 31.

[0168] Next, in step S45, the audio control unit 302 mutes the audio based on the audio data D22 of the second user U2 in response to a mute instruction from the second terminal device TM2.

[0169] Next, in step S46, the communication control unit 301 starts transmitting image data D21 of a plurality of second users U2 to the first terminal device TM1 via the communication unit 31.

[0170] Next, in step S47, the image analysis unit 501 executes image analysis on the image data D21 of the second user U2 for each of the second terminal devices TM2 (that is, for each of the second users U2). Then, the image analysis unit 501 stores analysis information AN21 indicating the analysis result of the facial expression of the second user U2 in the storage unit 32. Specifically, the image analysis unit 501 stores a plurality of pieces of analysis information AN21 corresponding to the plurality of second terminal devices TM2 in the storage unit 32. Otherwise, the process of step S47 is similar to the processes of steps S4 to S6 in FIG. 9.

[0171] Next, in step S48, the voice analysis unit 502 performs voice analysis on the voice data D22 of the second user U2 for each of the second terminal devices TM2 (i.e., for each of the second users U2). Then, the voice analysis unit 502 stores analysis information AN22 indicating the analysis result of the voice of the second user U2 in the storage unit 32. Specifically, the voice analysis unit 502 stores multiple pieces of analysis information AN22 corresponding to the multiple second terminal devices TM2 in the storage unit 32.

[0172] Next, in step S49, the biometric data analysis unit 503 executes an analysis process on the biometric data D23 of the second user U2 for each of the second terminal devices TM2 (i.e., for each of the second users U2). Then, the biometric data analysis unit 503 stores analysis information AN23 indicating the analysis result of the biometric data D23 of the second user U2 in the storage unit 32. Specifically, the biometric data analysis unit 503 stores a plurality of pieces of analysis information AN23 corresponding to the plurality of second terminal devices TM2 in the storage unit 32.

[0173] Next, in step S50, the statistics calculation unit 70 calculates statistics of a plurality of pieces of analysis information AN2 each indicating the result of identifying the state of a plurality of second users U2, and generates statistical information ST indicating the statistical result. Specifically, the statistics calculation unit 70 creates a graph GP as the statistical information ST. The analysis information AN2 includes analysis information AN21 to analysis information AN23.

[0174] Next, in step S51, the communication control unit 301 transmits the statistical information ST (specifically, the graph GP) via the communication unit 31 to the first terminal device TM1.

[0175] Next, in step S52, the communication control unit 301 transmits the statistical information ST (specifically, the graph GP) via the communication unit 31 to each second terminal device TM2.

[0176] Next, in step S53, the communication control section 301 determines whether or not the communication between the first terminal device TM1 and the second terminal device TM2 has ended.

[0177] If it is determined in step S53 that the communication has not ended, the process proceeds to step S47.

[0178] On the other hand, if it is determined in step S53 that the communication has ended, the process ends.

[0179] The embodiments of the present invention have been described above with reference to the drawings. However, the present invention is not limited to the above embodiments, and can be implemented in various aspects without departing from the gist of the present invention. In addition, the components disclosed in the above embodiments can be modified as appropriate. For example, a certain component among all the components shown in one embodiment may be added to a component of another embodiment, or some components among all the components shown in one embodiment may be deleted from the embodiment.

[0180] In addition, the drawings are mainly schematic illustrations of each component to facilitate understanding of the invention, and the thickness, length, number, spacing, etc. of each component shown in the drawings may differ from the actual ones due to the convenience of creating the drawings. In addition, the configurations of each component shown in the above embodiment are merely examples and are not particularly limited, and it goes without saying that various modifications are possible within a range that does not substantially deviate from the effects of the present invention.

[0181] (1) The second state detection unit 25 shown in Fig. 2 may include at least the second imaging unit 261 or the second voice input unit 262. The second state detection unit 25 may not include the second biological information detection unit 263. The first state detection unit 15 shown in Fig. 3 may not include the first biological information detection unit 153.

[0182] 2 and 12 may include at least the image analysis unit 501 or the voice analysis unit 502. The condition identification unit 50 does not have to include the biological data analysis unit 503.

[0183] Furthermore, the condition identification unit 50 may identify the condition of the second user U2 based on a part or all of the image data D21, the voice data D22, and the biometric data D23, and generate analysis information AN2 (hereinafter, referred to as "analysis information AN24") indicating the identification result. Also, the condition identification unit 50 may identify the condition of the second user U2 based on a part or all of the analysis information AN21 by the image analysis unit 501, the analysis information AN22 by the voice analysis unit 502, and the analysis information AN23 by the biometric data analysis unit 503, and generate analysis information AN2 (hereinafter, referred to as "analysis information AN25") indicating the identification result.

[0184] 2 and 12 may calculate statistics of a plurality of pieces of analysis information AN21 to generate statistical information ST indicating the statistical results, may calculate statistics of a plurality of pieces of analysis information AN22 to generate statistical information ST indicating the statistical results, or may calculate statistics of a plurality of pieces of analysis information AN23 to generate statistical information ST indicating the statistical results. Also, the statistics calculation unit 70 may calculate statistics of a plurality of pieces of analysis information AN24 or a plurality of pieces of analysis information AN25 to generate statistical information ST indicating the statistical results.

[0185] (2) The arrangement of the state identification unit 50 and the specific voice detection unit 60 shown in Fig. 2 and Fig. 12, and the statistics calculation unit 70 shown in Fig. 3 and Fig. 12 is not particularly limited. That is, each of the state identification unit 50, the specific voice detection unit 60, and the statistics calculation unit 70 may be realized in any of the first control unit 10 of the first terminal device TM1, the second control unit 20 of the second terminal device TM2, and the control unit 30 of the server device SV.

[0186] For example, the state identification unit 50, the specific voice detection unit 60, and the statistics calculation unit 70 may be realized by the first control unit 10 of the first terminal device TM1, or may be realized by the second control unit 20 of the second terminal device TM2. In addition, in the first embodiment, the statistics calculation unit 70 is realized by the first control unit 10 of the first terminal device TM1, but for example, the two-way communication support system SYS according to the first embodiment includes a server device SV, and the statistics calculation unit 70 may be realized by the control unit 30 of the server device SV. In addition, in the first or second embodiment, for example, the specific voice detection unit 60 may be realized by the first control unit 10 of the first terminal device TM1.

[0187] (3) The second state detection unit 25 shown in Fig. 2 may include other sensors as long as they can detect the state of the second user U2. The sensor detects the state of the second user U2. The sensor then outputs second state data D20 indicating the state of the second user U2 to the state identification unit 50. Furthermore, the state identification unit 50 analyzes the second state data D20 output by the sensor to identify the state (e.g., reaction) of the second user U2.

[0188] Also, for example, the second display unit 22 of the second terminal device TM2 may display a GUI (Graphical User Interface) widget (e.g., a button). In this case, the widget represents the "response content" of the second user U2 when responding to a question from the first user U1. For example, the "response content" is a positive response such as "Like" or a negative response such as "No." For example, the widget is a "Like" button or a "No" button.

[0189] The second communication control unit 203 of the second terminal device TM2 transmits the widget operation content as analysis information AN2 to the first terminal device TM1 or the server device SV. Then, the statistics calculation unit 70 calculates statistics of the multiple pieces of analysis information AN2 transmitted from the multiple second terminal devices TM2.

[0190] (4) The two-way communication support system SYS can also support two-way communication in events such as online seminars and online live shows. In an event, for example, a sender (first user U1) distributes video and audio to a large number of viewers (second user U2). In this case, for example, a large number of viewers may be present in the same place (for example, a seminar venue or concert hall), and one or more second terminal devices TM2 may be prepared. Also, for example, a large number of viewers may be present separately, and a large number of second terminal devices TM2 may be prepared corresponding to each of the large number of viewers. In these cases, for example, the analysis information AN2 by the state identification unit 50 or the statistical information ST by the statistical calculation unit 70 indicates the "degree of excitement" of the viewers. The "degree of excitement" of the viewers is indicated, for example, by a numerical value or a graph. [Industrial Applicability]

[0191] The present invention provides a two-way communication support system and a computer program, and has industrial applicability. [Explanation of symbols]

[0192] 50 Status Identification Unit 60 Specific voice detection unit 70 Statistics calculation section 101 First display control unit (output control unit) 103 First communication control unit (output control unit) 201 Second display control unit (output control unit) 203 Second communication control unit (output control unit) 301 Communication control section (output control section) 501 Image Analysis Unit 502 Voice Analysis Department 503 Biological Data Analysis Department TM1 Terminal 1 TM2 Second terminal device SV server device SYS Two-way communication support system

Claims

1. A two-way communication support system that supports two-way communication between a first terminal device and a second terminal device that receives first status data from the first terminal device, the first status data changing according to a status of a first user present at a side where the first terminal device is located, and outputs an image and a sound represented by the first status data, a state identification unit that analyzes second state data that changes depending on a state of a second user present on a side where the second terminal device is located, and identifies a state of the second user; an output control unit that outputs analysis information indicating a result of the identification of the state of the second user by the state identification unit to the first terminal device; Equipped with the second status data includes voice data generated by inputting a voice of the second user; The state identification unit includes a voice analysis unit, The voice analysis unit analyzes the voice data of the second user to identify a state of the second user represented by the voice of the second user; the first status data includes voice data generated by inputting a voice of the first user; A specific voice detection unit that analyzes the voice data of the first user and detects a voice indicating a question from the first user, When a voice indicating the question from the first user is detected, the voice analysis unit analyzes the voice data of the second user to identify a reaction content of the second user; The output control unit outputs the analysis information, which includes an identification result of the reaction content of the second user, to the first terminal device.

2. the second status data includes image data generated by capturing an image of a face of the second user, The state specifying unit includes an image analyzing unit, The interactive communication support system according to claim 1 , wherein the image analysis unit analyzes the image data of the second user to identify a state of the second user represented by a facial expression of the second user.

3. the second status data includes biometric data indicating biometric information of the second user, The condition specifying unit includes a biological data analyzing unit, The interactive communication support system according to claim 1 , wherein the biometric data analysis unit analyzes the biometric data of the second user to identify a state of the second user represented by the biometric information of the second user.

4. the second status data includes voice data generated by inputting a voice of the second user; 4. The interactive communication support system according to claim 1, wherein when the audio based on the audio data of the second user is muted, the state identification unit identifies the state of the second user, and the output control unit causes the first terminal device to output the analysis information indicating the identification result by the state identification unit.

5. A statistics calculation unit that calculates statistics of the plurality of pieces of analysis information indicating respective results of identifying the states of the plurality of second users, The interactive communication support system according to claim 1 , wherein the output control unit causes the first terminal device to output statistical information indicating a statistical result calculated by the statistical calculation unit.

6. the statistical calculation unit creates a graph as the statistical information; The interactive communication support system according to claim 5 , wherein the output control unit causes the first terminal device to display the graph.

7. The interactive communication support system according to claim 5 , wherein the output control unit causes the second terminal device to output the statistical information.

8. a second terminal device that receives, from a first terminal device, first status data that changes according to a status of a first user present at a side where the first terminal device is located, and outputs an image and a sound represented by the first status data, and a computer of a two-way communication support system that supports two-way communication between the first terminal device and the second terminal device; a first step of analyzing second status data that changes depending on a status of a second user present on a side where the second terminal device is located, to identify a status of the second user; a second step of causing the first terminal device to output analysis information indicating a result of identifying the state of the second user; Run the command, the second status data includes voice data generated by inputting a voice of the second user; The first step includes analyzing the voice data of the second user to identify a state of the second user represented by the voice of the second user; the first status data includes voice data generated by inputting a voice of the first user; A third step of analyzing the voice data of the first user to detect a voice indicating a question from the first user, When a voice indicating the question from the first user is detected, the first step includes analyzing the voice data of the second user to identify a response content of the second user; The second step is a computer program for causing the first terminal device to output the analysis information including an identification result of the reaction content of the second user.

Citation Information

Patent Citations

  • Remote lecturing system

    JP1990184880A

  • Human reaction recognizing device and computer- readable recording medium where program for the same is recorded

    JP2001067482A

  • Remote conference / education system

    JP2006262010A

  • Apparatus, method and program for communication control

    JP2010232780A

  • Understanding state estimation device and program

    JP2016213631A