Information processing system, information processing method, and program

Abstract

To provide a mechanism capable of further improving usability on a voice call system.SOLUTION: An information processing system comprises: a call transmission state determination unit for determining whether or not a voice call system capable of transmitting a user's voice data is in a call transmission state, a state of transmitting the voice data being sonant; an utterance state determination unit for determining whether or not the system is in an utterance state, a state of the user performing utterance, on the basis of image data on the user; and a notification control unit for controlling processing of notifying the user with notification information based on a matching property between a determination result by the call transmission state determination unit and a determination result by the utterance state determination unit.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system, an information processing method, and a program.

Background Art

[0002] In recent years, a voice call system for making voice calls via a network has been widely used. A mute mode can be implemented in the voice call system, in which the voice acquired by a microphone is not transmitted to the other party.

[0003] The following Patent Document 1 discloses a technique for warning the mute mode based on the volume of the voice acquired by a microphone or whether the voice can be text-converted.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technique disclosed in Patent Document 1 above has been developed only recently and has room for improvement from various viewpoints.

[0006] Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to provide a mechanism capable of further improving the usability regarding a voice call system.

Means for Solving the Problems

[0007] In order to solve the above problems, according to one aspect of the present invention, there is provided an information processing system including: a transmission state determination unit that determines whether the voice call system capable of transmitting the user's voice data is in a transmission state in which the voice data is being transmitted; a speaking state determination unit that determines whether the user is in a speaking state based on the user's image data; and a notification control unit that controls a process of notifying the user of notification information based on the consistency between the determination result by the transmission state determination unit and the determination result by the speaking state determination unit.

[0008] The notification information may include information warning that an abnormality has occurred in the voice input unit that acquires the voice data.

[0009] The notification information may include information warning that the voice call system is operating in a mute mode that stops the transmission of the voice data.

[0010] The notification information may include information warning that the voice data includes voice having a sound source other than the user.

[0011] When the volume of the voice included in the voice data acquired by the voice input unit that acquires the voice data is less than a threshold value, the transmission state determination unit may determine that the voice call system is in a non-transmission state in which the voice data with sound is not being transmitted.

[0012] When the voice call system is operating in a mute mode that stops the transmission of the voice data, the control unit may determine that the voice call system is in a non-transmission state in which the voice data with sound is not being transmitted.

[0013] The voice call system further includes an imaging unit that acquires the image data, is capable of transmitting the image data, and the speaking state determination unit may continue to acquire the image data by the imaging unit regardless of whether the voice call system is transmitting the image data.

[0014] The notification control unit may control to display the notification information on a screen provided for the voice call system.

[0015] The speech state determination unit may further determine whether the user is in the speech state based on at least one of the vibration of the user's body or biometric information.

[0016] Also, in order to solve the above problems, according to another aspect of the present invention, there is provided an information processing method executed by a computer, the method including: determining whether a voice call system capable of transmitting user voice data is in a transmission state in which the audible voice data is being transmitted; determining whether the user is in a speech state in which the user is speaking, based on the user's image data; and controlling to notify the user of notification information based on the consistency between the determination result as to whether the voice call system is in the transmission state and the determination result as to whether the user is in the speech state.

[0017] Also, in order to solve the above problems, according to another aspect of the present invention, there is provided a program for causing a computer to function as a transmission state determination unit that determines whether a voice call system capable of transmitting user voice data is in a transmission state in which the audible voice data is being transmitted, a speech state determination unit that determines whether the user is in a speech state in which the user is speaking, based on the user's image data, and a notification control unit that controls to notify the user of notification information based on the consistency between the determination result by the transmission state determination unit and the determination result by the speech state determination unit.

Effect of the Invention

[0018] As described above, according to the present invention, a mechanism capable of further improving the usability regarding a voice call system is provided.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Mode for Carrying Out the Invention

[0020] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant description is omitted.

[0021] In the present specification and drawings, elements having substantially the same functional configuration may be distinguished by attaching an index including different alphabets or numbers after the same reference numeral. For example, a plurality of elements having substantially the same functional configuration are distinguished as the terminal devices 10A and 10B as necessary. However, when it is not necessary to particularly distinguish each of a plurality of elements having substantially the same functional configuration, only the same reference numeral is attached. For example, when it is not necessary to particularly distinguish the terminal device 10A and the terminal device 10B, they are simply referred to as the terminal device 10.

[0022] <1. Embodiment> <1.1. Outline> FIG. 1 is a diagram showing an example of the configuration of a system 1 according to an embodiment of the present invention. As shown in FIG. 1, the system 1 includes two terminal devices 10 (10A and 10B). The two terminal devices 10 are communicably connected via an arbitrary network such as the Internet or a LAN (Local Area Network). The terminal device 10 may be, for example, a PC, a smartphone, or a tablet terminal. Hereinafter, the user of the terminal device 10A is also referred to as user A, and the user of the terminal device 10B is also referred to as user B. User B is an example of a call partner of user A.

[0023] (1) Terminal device 10 As shown in FIG. 1, the terminal device 10A includes an imaging unit 11, a display unit 12, an audio input unit 13, an audio output unit 14, an operation unit 15, and a communication unit 16. Although not shown in FIG. 1, it is assumed that the terminal device 10B also has the same components.

[0024] The imaging unit 11 captures the surroundings and acquires image data. For example, the imaging unit 11 may be a camera.

[0025] The display unit 12 optically reproduces and displays the image included in the input image data. For example, the display unit 12 may be a display or a touch panel.

[0026] The audio input unit 13 converts the surrounding audio into an electrical signal and acquires audio data. For example, the audio input unit 13 may be a microphone. The audio input unit 13 may be configured to be detachable from the terminal device 10A by USB (Universal Serial Bus) or the like.

[0027] The audio output unit 14 outputs the audio included in the input audio data. For example, the audio output unit 14 may be a speaker or a bone conduction speaker.

[0028] The operation unit 15 receives user operations. The operation unit 15 may be, for example, a keyboard, a mouse, or a touch panel.

[0029] The communication unit 16 is an interface capable of communication conforming to any wired or wireless communication standard. The communication unit 16 transmits and receives information to and from, for example, the terminal device 10B.

[0030] (2) Voice call system 20 Programs that function as voice call systems 20 (20A and 20B) are installed in the terminal device 10A and the terminal device 10B. The voice call system 20 is a system that enables voice calls via a network and realizes voice calls between the terminal device 10A and the terminal device 10B.

[0031] The voice call system 20 may be a so-called video conferencing system that transmits and receives image data together with voice data. For example, the voice call system 20A may transmit the image data of user A acquired by the imaging unit 11 and the voice data of user A acquired by the voice input unit 13 to the voice call system 20B by the communication unit 16. Also, the voice call system 20A may display the image included in the image data of user B received by the communication unit 16 on the display unit 12 and output the voice included in the voice data of user B by the voice output unit 14. Further, the voice call system 20A may accept operations such as changing the operation mode or inputting a text message by the operation unit 15. Note that the display of various images and the acceptance of user operations can be performed on a screen provided by the voice call system 20 (hereinafter also referred to as a UI (User Interface) screen).

[0032] The voice call system 20 can transmit voice data, but it is also possible to stop the transmission of voice data. That is, the voice call system 20 can operate in either a normal mode in which the transmission of voice data is continued or a mute mode in which the transmission of voice data is stopped. The user can freely switch the operation mode during a video conference.

[0033] (3) Support system 30 A program that functions as the support system 30 is installed in the terminal device 10A. The support system 30 is a system that supports user A who uses the voice call system 20A, and realizes the support for user A who uses the voice call system 20A.

[0034] For example, the support system 30 may obtain various information from the voice call system 20A by calling an API (Application Programming Interface) provided by the voice call system 20A to an external application, and display a message for supporting user A on the UI screen provided by the voice call system 20A. The detailed configuration of the support system 30 will be described later.

[0035] <1.2. Technical problem> Similar to the technology disclosed in the above Patent Document 1, the voice call system 20 can also warn that the user has spoken during the operation in the mute mode based on the volume of the voice included in the voice data acquired by the voice input unit 13. However, in a situation where it is difficult to appropriately acquire voice data in the first place, such as when the voice input unit 13 is malfunctioning or has dropped off, it becomes difficult to detect the user's speech in the first place. Therefore, there is a concern that the user may continue to talk without being warned at all, and a situation may occur where a conversation cannot be established with the other party on the call.

[0036] For the purpose of solving such problems, the support system 30 according to the present embodiment has been developed.

[0037] <1.3. Details of the support system 30> The support system 30 determines whether user A is speaking based on the image data of user A acquired by the imaging unit 11. Therefore, even in a situation where it is difficult to appropriately acquire voice data because the voice input unit 13 is malfunctioning or the like, the support system 30 can accurately determine whether user A is speaking. As a result, it becomes possible to avoid the above-described concerning situation.

[0038] Hereinafter, while referring to FIG. 1 again, the detailed configuration of the support system 30 will be described. As shown in FIG. 1, the support system 30 includes a transmission state determination unit 31, a speaking state determination unit 32, and a notification control unit 33.

[0039] (1) Transmission state determination unit 31 The transmission state determination unit 31 determines whether it is in a transmission state in which the voice call system 20A capable of transmitting the voice data of user A is transmitting voice data with sound. It is assumed that the voice data having sound means that the voice data includes some kind of voice. The voice here is a voice other than noise, and for example, it refers to the voice of a human.

[0040] On the other hand, the state where the voice call system 20A is not in the transmission state, that is, the state where the voice call system 20A is not transmitting voice data with sound, is also referred to as a non-transmission state. It is assumed that the voice data being silent means that the voice data does not include any voice.

[0041] In the non-transmission state, the voice call system 20A transmits silent voice data or stops transmitting voice data itself. For example, when the voice input unit 13 is malfunctioning or has dropped off, the voice call system 20A can transmit silent voice data. On the other hand, the voice call system 20A can stop transmitting voice data itself while operating in the mute mode.

[0042] As an example, the transmission state determination unit 31 may determine the transmission state / non-transmission state based on at least one of the voice data acquired by the voice input unit 13 or the operation mode of the voice call system 20A.

[0043] Specifically, when the volume of the voice included in the voice data acquired by the voice input unit 13 is less than the threshold value, the transmission state determination unit 31 may determine that the voice call system 20A is in the non-transmission state. The threshold value is preferably set to a value that can exclude the case where only noise is included in the voice data.

[0044] Further, when the voice call system 20A is operating in the mute mode, the transmission state determination unit 31 may determine that the voice call system 20A is in the non-transmission state. Note that the transmission state determination unit 31 can obtain information indicating the operation mode of the voice call system 20A by monitoring whether an operation to switch to the mute mode is performed on the API of the voice call system 20A or on the UI screen displayed on the display unit 12.

[0045] On the other hand, when the voice call system 20A is operating in the normal mode and the volume of the voice included in the voice data acquired by the voice input unit 13 is equal to or greater than the threshold value, the transmission state determination unit 31 may determine that the voice call system 20A is in the transmission state.

[0046] As another example, the transmission state determination unit 31 may determine whether the voice call system 20A is in the transmission state or the non-transmission state by monitoring the volume of the voice included in the voice data being transmitted by the voice call system 20A.

[0047] (2) Speaking state determination unit 32 The speaking state determination unit 32 determines whether the user A is in a speaking state, which is a speaking state, based on the image data of the user A acquired by the imaging unit 11. A state that is not the speaking state, that is, a state where the user A is silent, is also referred to as a non-speaking state.

[0048] As an example, the speaking state determination unit 32 determines that it is in the speaking state when the mouth of the user A is continuously moving, and determines that it is in the non-speaking state otherwise. The speaking state determination unit 32 may determine whether the mouth of the user A is continuously moving after identifying the user A included in the image data by face authentication. According to such a configuration, the speaking state determination unit 32 can accurately determine the speaking state / non-speaking state even in a situation where it is difficult to appropriately acquire voice data due to a failure or the like of the voice input unit 13.

[0049] (3) Notification control unit 33 The notification control unit 33 controls the process of notifying user A of notification information based on the consistency between the determination result by the transmission state determination unit 31 and the determination result by the speech state determination unit 32. The notification information can be displayed by the display unit 12. For example, when the determination result by the transmission state determination unit 31 and the determination result by the speech state determination unit 32 do not match, the notification control unit 33 may control the display unit 12 to display notification information warning that an abnormality has occurred on the UI screen of the voice call system 20A.

[0050] As an example of a situation where the determination result by the transmission state determination unit 31 and the determination result by the speech state determination unit 32 do not match, there is a situation where user A is in a speech state but the voice call system 20A is in a non-transmission state. That is, when user A is in a speech state but the voice call system 20A is in a non-transmission state, the notification control unit 33 may control the display unit 12 to display notification information warning that an abnormality has occurred. According to such a configuration, even in a situation where the voice input unit 13 has malfunctioned or the like and it is difficult to appropriately acquire voice data, it is possible to quickly resolve an abnormal situation where user A is speaking but user A's voice has not reached user B. In this way, it is possible to further improve the usability of the voice call system 20A.

[0051] The notification information may include information warning that an abnormality has occurred in the voice input unit 13 that acquires voice data. For example, when user A is in a speech state and the volume of the voice included in the voice data acquired by the voice input unit 13 is less than a threshold value, the notification control unit 33 can control the display unit 12 to display such notification information. Based on such notification information, user A can reconnect or replace the voice input unit 13 to resume the transmission of voice data including user A's voice.

[0052] The notification information may include information that the voice call system 20A is operating in mute mode, i.e., information warning the user that the mute mode has been forgotten to be cancelled. For example, the notification control unit 33 may control the display unit 12 to display such notification information when the operation mode of the voice call system 20A is in mute mode while the user A is speaking. Based on the notification information, the user A can cancel the mute mode and switch the operation mode of the voice call system 20A to normal mode, thereby resuming transmission of audio data including the voice of the user A.

[0053] (4) Operation processing Next, an example of the flow of processing executed by the assistance system 30 will be described with reference to FIG.

[0054] 2 is a flowchart showing an example of the flow of processing executed by the support system 30 according to this embodiment. The processing shown in FIG. 2 can be started when a voice call is started by the voice call system 20A.

[0055] As shown in FIG. 2, first, the support system 30 acquires image data acquired by the imaging unit 11 and audio data acquired by the audio input unit 13 (step S102).

[0056] Next, the utterance state determination unit 32 determines whether or not the user A is in an utterance state based on the image data acquired in step S102 (step S104).

[0057] If it is determined that user A is in a non-utterance state (step S104: NO), the process proceeds to step S114.

[0058] On the other hand, if it is determined that the user A is in a speaking state (step S104: YES), the talk-transmitting state determining unit 31 determines whether the operation mode of the voice communication system 20A is in the mute mode (step S106).

[0059] If it is determined that the operation mode of the voice communication system 20A is the normal mode (step S106: NO), the process proceeds to step S110.

[0060] On the other hand, if it is determined that the operation mode of the voice call system 20A is the mute mode (step S106: YES), the notification control unit 33 controls the display unit 12 to display information warning that the mute mode has been forgotten to be cancelled (step S108).

[0061] An example of information displayed on the display unit 12 to warn the user that they have forgotten to turn off the mute mode will now be described with reference to FIG.

[0062] FIG. 3 is a diagram showing an example of a UI screen 40A of the voice call system 20A displayed by the terminal device 10A according to the present embodiment. The UI screen 40A includes a party image display field 41, a user image display field 42, and a mute button 43. The voice call system 20A displays an image of user B included in image data received from the voice call system 20B in the party image display field 41. The voice call system 20A displays an image of user A included in image data acquired by the imaging unit 11 in the user image display field 42. The voice call system 20A accepts a user operation to enter or exit mute mode via the mute button 43. The mute button 43 also has a function of displaying whether the operation mode of the voice call system 20A is mute mode or normal mode. In the example shown in FIG. 3, the mute button 43 indicates that the voice call system 20A is operating in mute mode.

[0063] 3, the UI screen 40A includes a message 44 warning the user that the mute mode has been forgotten to be turned off. Based on this message, the user A can press the mute button 43 to turn off the mute mode, thereby resuming transmission of audio data including the voice of the user A.

[0064] Thereafter, the speech state determination unit 31 determines whether the volume of the speech included in the speech data acquired in step S102 is equal to or greater than a threshold value (step S110).

[0065] If it is determined that the volume is equal to or greater than the threshold value (step S110: YES), the process proceeds to step S114.

[0066] On the other hand, if it is determined that the volume is less than the threshold value (step S110: NO), the notification control unit 33 controls the display unit 12 to display information warning that an abnormality has occurred in the voice input unit 13 (step S112).

[0067] Here, an example of the information warning that an abnormality has occurred in the voice input unit 13, which is displayed on the display unit 12, will be described with reference to FIGS. 4 and 5.

[0068] FIG. 4 is a diagram showing an example of a UI screen 40B of the voice call system 20A displayed by the terminal device 10A according to the present embodiment. The UI screen 40B includes a counterpart image display column 41, a self-image display column 42, and a mute button 43.

[0069] Here, the UI screen 40B is displayed when the operation mode of the voice call system 20A is the normal mode (step S106: NO) and step S110 is NO. Therefore, on the UI screen 40B, the mute button 43 indicates that it is operating in the normal mode.

[0070] As shown in FIG. 4, the UI screen 40B includes a message 45 warning that an abnormality has occurred in the voice input unit 13. Based on such notification information, user A can reconnect or replace the voice input unit 13 to resume transmission of voice data including the voice of user A.

[0071] 5 is a diagram showing an example of a UI screen 40C of the voice call system 20A displayed by the terminal device 10A according to the present embodiment. The UI screen 40C includes a partner image display field 41, a self-image display field 42, and a mute button 43.

[0072] Here, the UI screen 40C is displayed when the operation mode of the voice call system 20A is the mute mode (step S106: YES) and step S110 is NO. Therefore, the mute button 43 indicates that the system is operating in the mute mode.

[0073] 5, the UI screen 40C includes a message 46 that warns that the mute mode has been forgotten to be released and that an abnormality has occurred in the audio input unit 13. Based on this message, the user A can switch the operation mode of the voice call system 20A to the normal mode, and further reconnect or replace the audio input unit 13, thereby resuming transmission of audio data including the voice of the user A.

[0074] Thereafter, the notification control unit 33 determines whether the call has ended (step S114).

[0075] If it is determined that the call has ended (step S114: YES), the process ends.

[0076] If it is determined that the call has not ended (step S114: NO), the process returns to step S102.

[0077] An example of the flow of the processing executed by the assistance system 30 has been described above.

[0078] By executing the steps from S102 to S114 shown in FIG. 2, the support system 30 determines the consistency between the state of the voice call system 20 (transmission state / non - transmission state) and the state of the user (speaking state / non - speaking state). If it is determined that the consistency is not achieved, it is possible to notify the user to that effect. For example, by executing the steps related to S104 and S106, the support system 30 can notify the user that they are in the mute state despite being in the speaking state. Also, by executing the steps related to S104 and S110, the support system 30 can notify the user that although they are in the speaking state, no voice is being acquired due to an abnormality in the voice input unit. In FIG. 2, a processing flow for making these two consistency determinations is illustrated. However, of course, the support system 30 may make only one of the consistency determinations.

[0079] <2. Hardware Configuration Example> Subsequently, with reference to FIG. 6, the hardware configuration of the information processing apparatus according to the present embodiment will be described. FIG. 6 is a block diagram showing an example of the hardware configuration of the information processing apparatus according to the present embodiment. Note that the information processing apparatus 900 shown in FIG. 6 can realize, for example, the terminal device 10 shown in FIG. 1. The information processing by the terminal device 10 according to the present embodiment is realized by the cooperation of software and the hardware described below.

[0080] As shown in FIG. 6, the information processing apparatus 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, an interface 907, an input device 908, an output device 909, a storage device 910, and a communication device 911.

[0081] The CPU 901 functions as an arithmetic processing unit and a control unit, and controls the overall operations within the information processing apparatus 900 according to various programs. Also, the CPU 901 may be a microprocessor. The ROM 902 stores programs used by the CPU 901, arithmetic parameters, and the like. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that appropriately change during the execution. These are interconnected by a host bus 904 composed of a CPU bus or the like. The CPU 901 can form, for example, the voice call system 20 and the support system 30 (transmission state determination unit 31, speaking state determination unit 32, and notification control unit 33) shown in FIG. 1.

[0082] The host bus 904 is connected to an external bus 906 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 905. Note that it is not necessarily required to separately configure the host bus 904, the bridge 905, and the external bus 906, and these functions may be implemented on a single bus.

[0083] The input device 908 is composed of input means such as a mouse, keyboard, touch panel, button, microphone, camera, switch, and lever for the user to input information, and an input control circuit that generates an input signal based on the input by the user and outputs it to the CPU 901. The user who operates the information processing apparatus 900 can input various data to the information processing apparatus 900 or instruct a processing operation by operating this input device 908. The input device 908 can form, for example, the imaging unit 11, the voice input unit 13, and the operation unit 15 shown in FIG. 1.

[0084] The output device 909 includes, for example, a display device such as a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a lamp, and a voice output device such as a speaker. The output device 909 can form, for example, the display unit 12 and the voice output unit 14 shown in FIG. 1.

[0085] The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device for recording data on the storage medium, a reading device for reading data from the storage medium, a deleting device for deleting data recorded on the storage medium, and the like. The storage device 910 is composed of, for example, an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

[0086] The communication device 911 is, for example, a communication interface composed of a communication device or the like for connecting to a network. Also, the communication device 911 may support either wireless communication or wired communication. The communication device 911 can form, for example, the communication unit 16 shown in FIG. 1.

[0087] Above, an example of the hardware configuration capable of realizing the functions of the information processing device 900 according to the present embodiment has been shown. Each of the above components may be realized using general-purpose members, or may be realized by hardware specialized for the functions of each component. Therefore, it is possible to appropriately change the hardware configuration to be used according to the technical level at the time of implementing the present embodiment.

[0088] <3. Supplementary Explanation> As described above, the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that those having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modification examples or correction examples within the scope of the technical idea described in the claims, and it is naturally understood that these also belong to the technical scope of the present invention.

[0089] In the above embodiment, as an example of a situation where the determination result by the transmission state determination unit 31 and the determination result by the speech state determination unit 32 do not match, a situation where the user A is in a speech state but the voice call system 20A is in a non-transmission state is given. However, the present invention is not limited to such an example. As another example of a situation where the determination result by the transmission state determination unit 31 and the determination result by the speech state determination unit 32 do not match, a situation where the user A is in a non-speech state but the voice call system 20A is in a transmission state can be given.

[0090] Specifically, when the user A is in a non-speech state but the voice call system 20A is in a transmission state, the notification control unit 33 may control the display unit 12 to display notification information warning that an abnormality has occurred. In this case, the notification information may include information warning that the voice data includes voice having a sound source other than the user A. Note that examples of the voice having a sound source other than the user A include noise and the voices of people around. Whether or not the voice data includes voice having a sound source other than the user A may be simply determined by whether or not text can be extracted from the voice included in the voice data by voice recognition. Alternatively, whether or not the voice data includes voice having a sound source other than the user A may be determined in detail by voice authentication as to whether the voice included in the voice data is the voice of the user A. According to such a configuration, it is possible to prompt the user A to move to a quiet environment or the like and to quickly eliminate an abnormal situation such as the voice of someone other than the user A being erroneously received by the user B.

[0091] In the above embodiment, it has been described that the voice call system 20A can switch the transmission / transmission stop of the voice data acquired by the voice input unit 13. However, the present invention is not limited to such an example. The voice call system 20A may switch the transmission / transmission stop of the image data acquired by the imaging unit 11. However, it is desirable that the speech state determination unit 32 continue to acquire the image data by the imaging unit 11 regardless of whether the voice call system 20A is transmitting the image data. More specifically, even when the voice call system 20A stops transmitting the image data and leaves the self-image display column 42 blank, it is desirable that the speech state determination unit 32 continue to acquire the image data by the imaging unit 11 in the background. According to such a configuration, it becomes possible to continue the determination of the speech state / non-speech state of user A based on the image data.

[0092] In the above embodiment, an example has been described in which the voice call system 20 is a so-called video conferencing system that transmits and receives image data together with voice data. However, the present invention is not limited to such an example. The voice call system 20 may omit the transmission and reception of the image data. For example, the voice call system 20 may provide a voice call service in the metaverse. In that case, the support system 30 may control to display the notification information in an area on the metaverse as a display unit.

[0093] In the above embodiment, an example has been described in which the notification information is visually output by the display unit 12. However, the present invention is not limited to such an example. The notification information may be audibly output by the voice output unit 14, or may be tactilely output by vibration or electrical stimulation or the like.

[0094] In the above-described embodiment, an example has been described in which the speech state determination unit 32 determines whether or not the user A is in a speech state based on the image data acquired by the imaging unit 11. However, the present invention is not limited to such an example. The speech state determination unit 32 may determine whether or not the user A is in a speech state based on at least any one of the body vibration or biometric information of the user A, together with or instead of the image data acquired by the imaging unit 11. An example of the body vibration is the vibration of the head. Examples of the biometric information include heartbeat and respiration. According to such a configuration, it becomes possible to more accurately determine whether or not the user A is in a speech state. Note that the body vibration and biometric information of the user A can be acquired by a wearable device such as a smartwatch, separately from the terminal device 10. Alternatively, when the terminal device 10 is configured as a wearable device such as an HMD (Head Mounted Display), the body vibration and biometric information of the user A may be acquired by the terminal device 10 itself.

[0095] In the above-described embodiment, an example has been described in which the voice call system 20A and the support system 30 are configured as separate information processing systems. However, the present invention is not limited to such an example. The support system 30 may be incorporated in the voice call system 20A.

[0096] In addition, a series of processes performed by each device described in this specification may be realized using any of software, hardware, and a combination of software and hardware. The programs constituting the software are, for example, pre-stored in a recording medium (specifically, a non-transitory computer-readable storage medium) provided inside or outside each device. Then, each program is read into a RAM (Random Access Memory) when executed by a computer that controls each device described in this specification, and is executed by a processing circuit such as a CPU (Central Processing Unit). The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Also, the above computer program may be distributed via a network, for example, without using a recording medium. Also, the above computer may be a specific-purpose integrated circuit such as an ASIC (application specific integrated circuit), a general-purpose processor that executes functions by reading a software program, or a computer on a server used for cloud computing, etc. In addition, a series of processes performed by each device described in this specification may be centrally processed by a single computer or may be distributed and processed by a plurality of computers. Further, in each of the above embodiments, two or more communication means existing in one device may be physically realized by one medium.

[0097] Also, the processes described using flowcharts or sequence diagrams in this specification do not necessarily have to be executed in the order shown. Some processing steps may be executed in parallel. Also, additional processing steps may be adopted, and some processing steps may be omitted.

Explanation of Reference Numerals

[0098] 1 System 10 Terminal Device 11 Imaging Unit 12 Display Unit 13 Voice Input Unit 14 Voice output unit 15 Operation unit 16 Communication unit 20 Voice call system 30 Support system 31 Transmission state determination unit 32 Speaking state determination unit 33 Notification control unit 40 UI screen 41 Other party image display column 42 Self-image display column 43 Mute button

Claims

1. A transmission state determination unit that determines whether or not it is a transmission state in which the voice call system capable of transmitting the user's voice data is transmitting the voice data with sound; A speech state determination unit that determines whether or not it is a speech state in which the user is speaking, based on the user's image data; A notification control unit that controls a process of notifying the user of notification information based on the consistency between the determination result by the transmission state determination unit and the determination result by the speech state determination unit; An information processing system comprising:

2. The notification information includes information warning that an abnormality has occurred in a voice input unit that acquires the voice data. The information processing system according to claim 1.

3. The notification information includes information warning that the voice call system is operating in a mute mode in which transmission of the voice data is stopped. The information processing system according to claim 1.

4. The notification information includes information warning that the voice data includes voice having a sound source other than the user. The information processing system according to claim 1.

5. When the volume of the voice included in the voice data acquired by the voice input unit that acquires the voice data is less than a threshold value, the transmission state determination unit determines that it is a non-transmission state in which the voice call system is not transmitting the voice data with sound. The information processing system according to claim 1.

6. When the voice call system is operating in a mute mode in which transmission of the voice data is stopped, the control unit determines that it is a non-transmission state in which the voice call system is not transmitting the voice data with sound. The information processing system according to claim 1.

7. The voice call system further includes an imaging unit that acquires the image data, and is capable of transmitting the image data. Regardless of whether or not the voice call system is transmitting the image data, the speech state determination unit continues to acquire the image data by the imaging unit. The information processing system according to claim 1.

8. The notification control unit controls to display the notification information on a screen provided to the voice call system. The information processing system according to claim 1.

9. The speech state determination unit further determines whether or not the user is in the speech state based on at least one of the vibration of the user's body or biometric information. The information processing system according to claim 1.

10. An information processing method executed by a computer, determining whether it is a transmission state in which a voice call system capable of transmitting user voice data is transmitting the audible voice data; determining whether it is a speaking state in which the user is speaking, based on the user's image data; controlling a process of notifying the user of notification information based on the consistency between the determination result as to whether the voice call system is in the transmission state and the determination result as to whether the user is in the speaking state; An information processing method comprising:

11. A computer, a transmission state determination unit that determines whether it is a transmission state in which a voice call system capable of transmitting user voice data is transmitting the audible voice data; a speaking state determination unit that determines whether it is a speaking state in which the user is speaking, based on the user's image data; a notification control unit that controls a process of notifying the user of notification information based on the consistency between the determination result by the transmission state determination unit and the determination result by the speaking state determination unit; A program for causing the computer to function as such.

Images