Information processing device, conferencing system, and information processing method

Abstract

The present invention comprises: a first determination unit (103b) that determines the magnitude relation between a determination subject and an audio level in accordance with audio data based on speech content transmitted from at least two information processing devices other than a host information processing device; a selection unit (104b) that selects an information processing terminal on the basis of the result of the determination; a first transmission unit (105b) that transmits an evaluation result to the selected information processing terminal; a first reception unit (106b) that receives an evaluation result from an information processing device other than a first information processing device; and a second determination unit (104b) that, on the basis of the received evaluation result, determines whether the fact that the audio level does not satisfy a criterion is caused by the host information processing device or by the information processing device which transmitted the evaluation result. The present invention outputs a notification in accordance with the determination result.

Description

Information processing device, conference system, and information processing method

[0001] The present invention relates to an information processing device, a conference system, and an information processing method.

[0002] There is an online conference system in which multiple terminal devices are connected to each other so that audio data corresponding to the content of a speaker's speech is transmitted from one terminal device to another. In some cases, two terminal devices are connected to such an online conference system, but in other cases, three or more terminal devices are connected. Patent Document 1 also discloses an electronic conference system in which multiple participants belong to the same conference room and hold an electronic conference.

[0003] JP 2017-063416 A

[0004] When participating in an online conference using such an online conference system, it may be difficult to hear the voice based on the voice data transmitted from a terminal device of a particular speaker. Causes of difficulty in hearing include a low voice level of the voice data or excessive noise. This difficulty may be due to the sender or receiver of the voice data, but it is difficult to determine which is the cause. While Patent Document 1 allows a user to check whether information output from their own terminal device is being properly transmitted to the other party, there are also periods in an online conference when participants do not speak. In such cases, it is not possible to check whether the voice based on the voice data is audible.

[0005] One aspect of the present invention is an information processing device used by participants in a conference system to which the information processing device is communicatively connected, the information processing device having: a first judgment unit that judges the voice level of voice data based on the speech content transmitted from at least two information processing devices other than the information processing device itself; a selection unit that selects, based on the judgment result, an information processing terminal that is the sender of the voice data whose voice level does not reach a standard; a first transmission unit that transmits an evaluation result indicating that the voice level does not reach the standard to the selected information processing terminal; a first receiving unit that receives the evaluation result from an information processing device other than the information processing device itself; a second judgment unit that judges, based on the received evaluation result, whether the cause of the voice level not reaching the standard is the information processing device itself or the information processing device that sent the evaluation result; and an output unit that, if it is determined that the cause lies in the information processing device itself based on the result determined by the second judgment unit, outputs a notification to the information processing device itself indicating that the cause lies in the information processing device itself.

[0006] Another aspect of the present invention is an information processing device in a conference system that is used by a participant and that is communicably connected to the information processing device, the information processing device including: an audio output unit that outputs audio in accordance with audio data based on speech content transmitted from at least two information processing devices other than the information processing device itself; an instruction input unit that receives an instruction indicating that, among the sounds output from the audio output unit, the audio level corresponding to the audio data is lower than that of the other information processing devices and that the sound is difficult to hear; and an instruction input unit that receives an instruction indicating which information processing device is the source of the difficult-to-hear sound, based on the received instruction. the information processing device has a selection unit that selects an information processing device that is the sender of a hard sound; a first transmission unit that transmits an evaluation result indicating that the sound is hard to hear to the selected information processing terminal; a first reception unit that receives the evaluation result from an information processing device other than the information processing device itself; a second determination unit that determines, based on the received evaluation result, whether the cause of the hard sound to hear is the information processing device itself or the information processing device that sent the evaluation result; and an output unit that, when it is determined, based on the result determined by the second determination unit, that the cause is the information processing device itself, outputs a notification to the information processing device indicating that the cause is the information processing device itself.

[0007] Another aspect of the present invention is a conference system in which information processing devices used by participants are connected to each other in a manner that allows communication, the conference system having: a first judgment unit that judges the audio level of audio data based on the content of speech transmitted from at least two information processing devices other than a first information processing device; a selection unit that selects, based on the judgment result, an information processing terminal that is the sender of audio data whose audio level does not reach a standard; a first transmission unit that transmits an evaluation result indicating that the audio level does not reach the standard to the selected information processing terminal; a first receiving unit that receives the evaluation result from an information processing device other than the first information processing device; a second judgment unit that judges, based on the received evaluation result, whether the cause of the audio level not reaching the standard is the first information processing device or the information processing device that sent the evaluation result; and an output unit that, when it is determined that the cause lies in the first information processing device based on the result determined by the second judgment unit, outputs a notification to the information processing device indicating that the cause lies in the first information processing device.

[0008] Another aspect of the present invention is an information processing method in a conference system in which information processing devices used by participants are connected to each other in a manner that allows communication, the information processing method including: a first determination step of determining the audio level of audio data based on the speech content transmitted from at least two information processing devices other than a first information processing device; a selection step of selecting, based on the determination result, an information processing terminal that is the sender of audio data whose audio level does not reach a standard; a first transmission step of transmitting an evaluation result indicating that the audio level does not reach the standard to the selected information processing terminal; a first reception step of receiving the evaluation result from an information processing device other than the first information processing device; a second determination step of determining, based on the received evaluation result, whether the cause of the audio level not reaching the standard is the first information processing device or the information processing device that sent the evaluation result; and an output step of outputting a notification to the information processing device itself, if it is determined that the cause lies in the first information processing device based on the result determined by the second determination step.

[0009] It is possible to select a terminal device that may be the cause of the low audio level from among the information processing devices of the participants participating in the online conference system.

[0010] 1 is a schematic system configuration diagram showing the configuration of an audio adjustment support system S for online conferences. FIG. 1 is a schematic functional block diagram explaining the functions of a terminal device. FIG. 2 is a flowchart explaining the operation of a terminal device participating in an online conference of the online conference system S. FIG. 3 is a diagram showing an example of an audio level according to audio data received from a terminal device 12. FIG. 4 is a diagram showing an example of an audio level according to audio data received from a terminal device 13. FIG. 5 is a diagram showing an example of transmitting feedback. FIG. 6 is a flowchart showing the operation of a terminal device when feedback is received. FIG. 7 is a diagram showing an example of a dialog screen. FIG. 8 is a diagram explaining the operation of a terminal device that has received a response signal to feedback from another terminal device. FIG. 9 is a diagram showing an example of a dialog screen. FIG. 10 is a system configuration diagram showing the configuration of an online conference system Sa in a second embodiment. FIG. 11 is a flowchart explaining processing for selecting a destination of feedback. FIG. 12 is a diagram showing an example of an audio level according to audio data. FIG. 13 is a schematic functional block diagram showing the configuration of a terminal device used in an online conference system in a third embodiment. FIG. 14 is a flowchart explaining the operation of a terminal device. FIG. 15 is a diagram showing an example of a display screen displayed on the output unit 108 of a terminal device 11a. FIG. 16 is a schematic block diagram showing the configuration of an information processing device 11b used in an online conference system in a fourth embodiment.

[0011] FIG. 1 is a schematic system configuration diagram showing the configuration of an audio adjustment support system S for online conferences. FIG. 1 illustrates an example in which three terminal devices, terminal device 11, terminal device 12, and terminal device 13, participate in the same conference room in an online conference in the online conference system S. In the online conference system S, multiple terminal devices are communicably connected to a conference server 20 via a network NW. Here, three terminal devices, terminal device 11, terminal device 12, and terminal device 13, are connected as the multiple terminal devices. Terminal device 11, terminal device 12, and terminal device 13 are each used by a different user. Although the devices participating in the online conference are described here as terminal devices, they do not necessarily have to be in the form of terminal devices as long as they are information processing devices. In this case, the information processing devices have the same functions as the terminal devices.

[0012] The conference server 20 allows participants to participate in the online conference in response to a participation request transmitted from each terminal device in response to an operation input from the participant participating in the online conference, and transmits voice data corresponding to the speech content of the participant to each terminal device among the multiple terminals permitted to participate. As a result, the speech content of the participant is distributed to the terminal devices of the other participants, and the voice data is played back, thereby enabling the online conference to be held.

[0013] The terminal device 11, the terminal device 12, and the terminal device 13 are used by different participants. For example, the terminal device 11 is used by participant A, the terminal device 12 is used by participant B, and the terminal device 13 is used by participant C.

[0014] 2 is a schematic functional block diagram illustrating the functions of the terminal devices. Here, the functions of the terminal device 11 will be described, but the other terminal devices (terminal device 12, terminal device 13) also have similar functions.

[0015] The communication unit 101 is communicably connected to the conference server 20 via the network NW. The communication unit 101 transmits voice data corresponding to voice input from a voice input unit 111 of its own terminal device (e.g., terminal device 11) to the conference server 20 via the network NW, and receives voice data generated in other terminal devices and transmitted from the conference server 20. The communication unit 101 also transmits notifications (e.g., messages) generated in its own terminal device (e.g., terminal device 11) to the conference server 20 via the network NW, and receives messages generated in other terminal devices and transmitted from the conference server 20.

[0016] The storage unit 102 stores various data. For example, the storage unit 202 stores a voice level corresponding to voice data transmitted from a terminal device other than the terminal device itself. Here, the storage unit 102 may store voice data instead of the voice level.

[0017] The first determination unit 103 determines whether a voice level corresponding to voice data based on speech content transmitted from at least two terminal devices other than the terminal device itself is greater than or less than the determination target. The voice level is a value corresponding to the amplitude of the voice represented by the voice data. For example, a higher voice level indicates a louder volume of the voice represented by the voice data, and a lower voice level indicates a quieter volume of the voice represented by the voice data. The voice level may be generated by the first determination unit 103 based on the voice data, or may be generated by the control unit 115 based on the voice data. The determination target may be any target as long as it is possible to grasp the variation in voice levels between at least two terminal devices other than the terminal device itself, with respect to the voice levels corresponding to the voice data transmitted from these at least two terminal devices.

[0018] <Determination Method> For example, either one of the following (1) or (2) may be used as the determination target. (1) The first determination unit 103 calculates the difference between the audio levels of audio data received from at least two terminal devices other than the terminal device itself (e.g., terminal device 11) and audio data transmitted from different terminal devices. In other words, the difference in audio levels is calculated for each combination of terminal devices participating in the same conference room as the conference room of the online conference in which the terminal device itself (e.g., terminal device 11) is participating. The first determination unit 103 determines whether the calculated difference is larger than a reference value. The reference value may be a predetermined value, or a value transmitted as a reference value from the conference server 20. If the difference in audio levels is larger than the reference value, the terminal device with the lower average audio level among the combinations of terminal devices for which the difference was calculated can be determined to have an audio level that deviates from the audio levels of the terminal devices participating in the online conference.

[0019] (2) The first determination unit 103 determines whether the audio levels corresponding to the audio data transmitted from at least two information processing devices other than the terminal device itself (e.g., terminal device 11) are greater or less than a predetermined reference value. In other words, the first determination unit 103 determines whether the audio levels corresponding to the audio data transmitted from each terminal device participating in the same conference room as the conference room of the online conference in which the terminal device (e.g., terminal device 11) is participating are greater or less than the reference value. A predetermined value may be used as the reference value. The reference value may be stored in the storage unit 202, and the first determination unit 103 may read the reference value stored in the storage unit 202 and determine whether the reference value is greater or less than each audio level.

[0020] <Target Period of Determination> The first determination unit 103 also determines whether the voice level is high or low based on the voice data based on the speech content. The determination of the voice level may be performed on voice data received during the evaluation period. The evaluation period is a certain period of time from the target time to be determined. The target time may be the current time. The certain period of time may be a predetermined period of time (e.g., five minutes). For example, the first determination unit 103 uses the voice level corresponding to the voice data obtained within the past five minutes from the current time as the target of determination. If the target period of determination is a period from the current time to a certain time ago, it is possible to determine a situation in which the voice level was maintained at a certain level at the start of the online conference, but then decreased after a certain period of time has passed. Here, the case where the target time of determination is the current time has been described. However, the target time of determination may also be a time before the current time after the start of the online conference, such as 10 minutes ago. For example, if the first judgment unit 103 makes a judgment based on the past 10 to 20 minutes, it can judge participants who had been speaking until recently but have not spoken for a while since then.

[0021] <Regarding the Audio Level Used for Determination> The first determination unit 103 calculates an average audio level for each source of audio data from audio data based on speech content within a certain period of time in the past, and uses the calculated average to determine whether the audio levels are large or small. For example, the first determination unit 103 calculates an average audio level for each source of audio data received during the period to be determined. The first determination unit 103 then determines whether the calculated average audio level is large or small relative to an average audio level calculated based on audio levels transmitted from other terminal devices. Here, a case where an average audio level is used to determine whether the audio levels are large or small is described, but the peak absolute audio level value corresponding to the audio data obtained during the period to be determined may also be used.

[0022] Based on the determination result, the selection unit 104 selects a terminal device that is a sender of audio data whose audio level does not reach the standard. The criterion for determining whether the standard is reached may be any criterion for determining whether the audio level of a terminal device is lower than that of some of the terminal devices other than the terminal device itself. For example, the following criteria (A) and (B) may be used.

[0023] (A) Based on the result of the judgment based on the judgment target shown in (1) above, the selection unit 104 selects as the destination the terminal device from which the audio data with a low audio level is sent in a combination in which the difference in audio level based on the magnitude relationship is equal to or greater than a reference value.

[0024] (B) The selection unit 104 selects as the destination the terminal device from which the voice data was sent that was determined to be smaller than the reference value based on the magnitude relationship as a result of the determination based on the determination target shown in (2) above.

[0025] The first transmission unit 105 transmits the evaluation result to the terminal device selected by the selection unit 104. The evaluation result indicates that the audio level of the audio data is lower than that of other terminal devices among the terminal devices belonging to the same conference room in the online conference system.

[0026] The first receiving unit 106 receives an evaluation result indicating that the audio level does not meet the standard from a terminal device other than the terminal device itself. That is, the process for obtaining the evaluation result is performed not only by the terminal device itself but also by each terminal device participating in the same conference room of the online conference, and the terminal device that requests the evaluation result transmits the evaluation result to each of the other terminal devices. Therefore, for example, the first receiving unit 106 of a terminal device (e.g., terminal device 11) receives the evaluation results from terminal device 12 and terminal device 13, respectively.

[0027] Based on the received evaluation result, the second determination unit 107 determines whether the cause of the audio level not reaching the standard is the terminal device itself or the terminal device that has transmitted the evaluation result.

[0028] If it is determined based on the result of the determination by the second determination unit 107 that the cause lies in the terminal device itself, the output unit 108 outputs a notification (e.g., a message) to the terminal device itself indicating that the cause lies in the terminal device itself. The output unit 108 may be a display panel such as an LCD panel, or an output circuit that outputs a video signal to display a display screen on a display device. By outputting such a message, the user of the terminal device can understand that the reason they are having difficulty hearing other participants in the online conference may be due to the sound output function of the terminal device they are using. This allows them to take various measures, such as changing the audio output settings or, if they are using a headset, checking for any problems with the headset.

[0029] If it is determined based on the result determined by the second determination unit 107 that the cause is not the terminal device itself, the second transmission unit 109 sends a message to the terminal device that sent the evaluation result indicating that the cause is in the terminal device that sent the evaluation result.

[0030] The second receiving unit 110 receives a message transmitted from a terminal device other than the terminal device itself. When the second receiving unit 110 receives the message, the output unit 108 outputs a message indicating that the problem is due to the sound pickup function of the terminal device itself, based on the received message. By outputting such a message, the user of the terminal device can understand that the reason why other participants in the online conference are having difficulty hearing the user's voice may be due to the sound pickup function of the terminal device they are using. This allows the user to take various measures, such as changing the settings related to the sound pickup function or checking the microphone connection status.

[0031] The voice input unit 111 receives voice input from participants who use their own terminal devices to participate in the online conference. The voice input unit 111 may be a microphone. The voice input unit 111 may also acquire an analog or digital signal generated in response to the microphone detecting a spoken voice.

[0032] The audio output unit 112 outputs audio based on the audio data. The audio output unit 112 may be, for example, a speaker. Alternatively, the audio output unit 112 may be a speaker drive circuit that outputs an audio signal based on the audio data to the speaker. For example, the audio output unit 112 outputs audio corresponding to audio data based on the content of speech transmitted from at least two information processing devices other than the terminal device itself.

[0033] The setting adjustment unit 113 performs various settings for the audio input from the audio input unit 111. The setting adjustment unit 113 also performs various settings for the audio output from the audio output unit 112.

[0034] The input unit 114 accepts various operation inputs from a user. The input unit 114 may be at least one of input devices such as a keyboard, a mouse, a touch panel, etc. The input unit 114 may also acquire, from the input device, operation content input by the user to the input device.

[0035] The control unit 115 controls each unit in the terminal device itself.

[0036] <<Operation>> Next, a description will be given of the operation of the above-described online conference system S. Fig. 3 is a flowchart illustrating the operation of a terminal device participating in an online conference of the online conference system S.

[0037] Here, a case will be described in which three participants (Participant A, Participant B, and Participant C) use different terminal devices to participate in an online conference, as shown in Fig. 1. Of the three terminal devices (Terminal Device 11, Terminal Device 12, and Terminal Device 13), the operation of Terminal Device 11 will be described as an example, but Terminal Device 12 and Terminal Device 13 each perform the same processing as Terminal Device 11 between themselves and the other terminal devices.

[0038] When participant A inputs an instruction to join the online conference from input unit 114, control unit 115 of terminal device 11 transmits a request signal to join the online conference to conference server 20 via communication unit 101. Here, terminal devices 12 and 13 similarly transmit request signals to join the online conference to conference server 20. Conference server 20 connects terminal devices 11, 12, and 13 so that they can communicate with each other as if they belong to the same conference room. This allows terminal devices 11, 12, and 13 to send and receive voice data to each other, enabling conversation among the three.

[0039] The control unit 115 of the terminal device 11 determines whether an online conference is being held (whether the user is able to participate in the online conference) (step S200). If an online conference is being held (step S200-YES), the control unit 115 proceeds to step S201, and if no conference is being held (step S200-NO), the control unit 115 ends the process.

[0040] When an online conference is being held (when the terminal device 11 is participating in an online conference), the control unit 115 of the terminal device 11 receives, via the communication unit 101, voice data transmitted from the terminal devices of the participants other than the terminal device 11 (here, terminal device 12 and terminal device 13) participating in the online conference, and stores the received voice data in the storage unit 102. The control unit 115 also counts the number of terminal devices connected to the online conference in which the terminal device 11 is currently participating (S201). The control unit 115 assigns the count result to a variable N.

[0041] The control unit 115 determines whether the number of connected terminal devices is two or more (step S202), and if it is less than two (step S202-NO), the process proceeds to step S200. In this case, the online conference is one-to-one, that is, between the terminal device 11 itself and another terminal device, so there is no need to perform an evaluation process regarding the audio.

[0042] Next, if the number of participating terminal devices other than the terminal device 11 is two or more (step S202—YES), the first determination unit 103 generates a voice level corresponding to the voice data stored in step S201 for each terminal device that transmitted the voice data. Here, a number is assigned to each terminal device. For example, the conference server 20 assigns numbers to the terminal devices in the order in which they received requests to connect to the online conference, generates a list indicating the correspondence between the identification information of the participating terminal devices and the assigned numbers, and transmits the list to each terminal device. By referring to this list, each terminal device can identify the relationship between the number assigned to the terminal device and the terminal device connected to the online conference. The first determination unit 103 stores the voice level in association with the number assigned to the terminal device.

[0043] 4A and 4B are diagrams showing examples of audio levels corresponding to audio data, respectively. Fig. 4A is a diagram showing an example of audio levels corresponding to audio data received from the terminal device 12, and Fig. 4B is a diagram showing an example of audio levels corresponding to audio data received from the terminal device 13.

[0044] 4A and 4B, the horizontal axis represents time, and the vertical axis represents audio levels. The figures show audio levels that belong to the target of determination within a certain period of time (e.g., the past 5 minutes) from the target time (e.g., the present).

[0045] The first determination unit 103 calculates an average value for each terminal device for an evaluation period, which is a period to be determined. Here, the first determination unit 103 calculates an average value of the audio level of audio data obtained during the evaluation period from among the audio data obtained from the terminal device 12. The first determination unit 103 also calculates an average value of the audio level of audio data obtained during the evaluation period from among the audio data obtained from the terminal device 13. The first determination unit 103 then calculates the difference between the average value of the audio level calculated based on the audio data of the terminal device 12 and the average value of the audio level calculated based on the audio data of the terminal device 13, and determines whether the value of this difference is greater than a predetermined reference value.

[0046] Based on the determination result of the first determination unit 103, the selection unit 104 identifies a combination of terminal devices for which it is determined that the difference in audio levels exceeds the reference value, and identifies a terminal device among the identified combinations whose average audio levels is lower than the average audio level of one of the terminal devices. The selection unit 104 then selects the identified terminal device as a feedback destination (step S203). The selection unit 104 stores the number of the selected terminal device and data (e.g., a flag) indicating that the terminal device has been selected as a feedback destination in the storage unit 102. Here, if there are multiple combinations of terminal devices, the terminal device 11 determines the magnitude relationship between the difference in the average audio levels and the reference value for each combination of terminal devices. The terminal device 11 then selects a feedback destination based on the determination result of the magnitude relationship. The terminal device selected in this manner may have a lower audio level according to the audio data than the other terminal devices, making it difficult to hear.

[0047] Next, the control unit 115 sets a variable K to 1 (step S204), and determines whether the value indicated by the variable K is equal to or less than N, which is the total number of terminal devices (step S205). If the value of K is not equal to or less than N (step S205-NO), that is, if the value of K exceeds N, the control unit 115 shifts the process to step S200.

[0048] On the other hand, if the value of K is equal to or less than the value of N (step S205-YES), the control unit 115 determines whether or not the Kth terminal device has been selected as the destination of the feedback (step S206). For example, if the value of K is 1, the control unit 115 determines whether or not the terminal device assigned as the first terminal device has been selected as the destination of the feedback. The feedback is a signal that indicates a request to improve the audio settings.

[0049] If the Kth terminal device is selected as the destination (step S206—YES), the control unit 115 determines whether or not there is a history of feedback being sent to the Kth terminal device in the currently held online conference (step S207). When the control unit 115 sends feedback to any terminal device, the history indicating that feedback has been sent is stored in the storage unit 102 in association with the number of the destination terminal device, information indicating that feedback has been sent (e.g., a flag), and the time of transmission.

[0050] If there is a history of sending feedback to the Kth terminal device (step S207—YES), the control unit 115 determines whether a specified time has elapsed since the last time feedback was sent (step S208). Here, the specified time is a predetermined time, such as 5 minutes.

[0051] If a specified time has elapsed since the time when the previous feedback was transmitted (step S208—YES), the control unit 115 outputs an instruction to cause the first transmission unit 105 to transmit the feedback. In response to this instruction, the first transmission unit 105 transmits the feedback to the terminal device to which the feedback is to be transmitted. For example, as shown in FIG. 5 , during an online conference, the terminal device 11 compares the audio levels of the terminal devices of the other participants (terminal device 12 and terminal device 13). If the audio level of the terminal device 13 is significantly low, the control unit 11 transmits feedback to the terminal device 13 to prompt the terminal device 13 to improve its audio settings.

[0052] After transmitting the feedback, the control unit 115 stores the time of transmission of the feedback and the number assigned to the destination terminal device in the storage unit 102 (step S209). The control unit 115 then adds 1 to the value of K (step S210) and proceeds to step S205. By adding 1 to the value of K, processing can be performed for the terminal device with the next number depending on whether it was selected as a feedback destination. This processing can be performed for each terminal device participating in the online conference.

[0053] On the other hand, if the Kth terminal device is not a feedback destination in step S206 (step S206—NO), the control unit 115 refers to the data stored in the storage unit 102 and determines whether or not there is a feedback transmission history for the Kth terminal device (step S211). If there is no feedback transmission history for the Kth terminal device (step S211—NO), the control unit 115 proceeds to step S210. On the other hand, if there is a feedback transmission history, the control unit 115 deletes the transmission history indicating that feedback was sent to the Kth terminal device (step S212—YES). Here, if it is determined that the Kth terminal device is not a feedback destination and a transmission history remains, the transmission history is deleted. As a result, if the Kth terminal device was previously selected as a feedback destination but the audio level has since improved, it is no longer necessary to select it as a transmission destination, and feedback can be prevented from being sent.

[0054] Also, in step S207, if there is no transmission history for the Kth terminal device (step S207-NO), the control unit 115 sends feedback for the Kth terminal device and stores information indicating that feedback has been sent to the Kth terminal device and the transmission time in the memory unit 102 as transmission history (step S213).

[0055] Furthermore, in step S208, if the specified time has not elapsed since the previous transmission (step S208-NO), the control unit 115 proceeds to step S210. Here, if there is a transmission history and the specified time has not elapsed since the previous transmission time, it is possible to prevent feedback from being sent multiple times in a short period of time.

[0056] 3, if the audio level of the audio data is above a certain level, the destination of the feedback may not be selected for any of the terminal devices. In such a case, the audio based on the audio data from the terminal devices participating in the online conference can be heard at any of the terminal devices.

[0057] <Receiving Feedback> FIG. 6 is a flowchart showing the operation of a terminal device when feedback is received. The process shown in FIG. 6 is executed by a terminal device that receives feedback from a terminal device other than the terminal device itself among the terminal devices participating in the online conference. Here, a case where the terminal device 13 receives feedback from at least one of the terminal device 11 and the terminal device 12 will be described. Upon receiving feedback (step S301), the control unit 115 of the terminal device 13 counts the total number M of terminal devices that have sent feedback (step S302). For example, if the control unit 115 of the terminal device 13 receives feedback only from the terminal device 11 or only from the terminal device 12, the control unit 115 counts the total number M as 1. If the control unit 115 receives feedback from both the terminal device 11 and the terminal device 12, the control unit 115 counts the total number M as 2.

[0058] Then, the control unit 115 of the terminal device 13 determines whether the total number M is 2 or more (step S303), and if the total number M is not 2 or more (step S303-NO), that is, if the total number M is 1, it transmits a response signal to the terminal device that sent the feedback (step S304). In this case, the terminal device that sent the feedback is terminal device 11 or terminal device 12, and no feedback has been sent from the remaining terminal devices. Therefore, the control unit 115 of the terminal device 13 determines that there may be a problem with the terminal device that sent the feedback, and can transmit a response signal to the feedback. In other words, it is possible to isolate the cause as being the source of the feedback.

[0059] On the other hand, if the total number M is equal to or greater than 2 in step S303, the control unit 115 displays a dialog box prompting the user to adjust and set the audio on the display screen of the terminal device 11 (step S305). In this case, since feedback is received not only from the terminal device 11 but also from the terminal device 12, it is possible to isolate the cause as being in the terminal device that sent the audio data (terminal device 13 in this case).

[0060] 7 is a diagram showing an example of a dialog screen displayed by the processing of step S305. As shown in this figure, the display screen displays a message indicating that the other terminal devices are having difficulty hearing the audio based on the audio data transmitted from the terminal device itself, and also displays a message urging the participant at terminal device 13 to review their microphone settings. This allows the participant at terminal device 13 to review their microphone settings, for example, to increase the microphone pickup level.

[0061] 8 is a diagram illustrating the operation of a terminal device that receives a response signal to feedback from another terminal device. Here, a case will be described in which terminal device 12 receives a response signal to feedback from terminal device 13. Receiving a response signal to feedback from another terminal device (terminal device 13) indicates that no event has occurred in terminal device 11 or terminal device 13 that would require feedback to be sent. Therefore, it is necessary to review the settings of terminal device 12 itself.

[0062] When the control unit 115 of the terminal device 12 receives a response signal to the feedback (step S401), it displays a dialogue screen on the display screen of the terminal device 12 itself, prompting the user to adjust and set the audio (step S402). By displaying this dialogue screen, it is possible to indicate that the function of outputting audio of the terminal device itself (terminal device 12) is malfunctioning. This allows the user to determine that the cause of the difficulty in hearing is in the terminal device itself and isolate the problem.

[0063] FIG. 9 illustrates an example of a dialog screen displayed by the processing of step S402. As shown in this figure, the dialog screen displays a message (notification) prompting the user to review the sound settings of the terminal device 12. For example, if the terminal device 12 sends feedback to other terminal devices but the terminal device 12 is the only terminal device that sent such feedback, the audio level of the audio data sent from these other terminal devices may be relatively high, and the cause may be the audio output settings of the terminal device 12. This may be caused, for example, by the speaker volume being set too low. Another cause may be that the terminal device 12 is connected to multiple speakers and an unintended speaker is mistakenly selected as the output destination, making the participants unable to hear the audio. Other causes include a poor communication environment between the terminal device 12 and the conference server 20, or a faulty headset connected to the terminal device 12 and worn by a participant, causing the audio to sound quiet.

[0064] Even in such a case, by displaying a dialogue screen on the terminal device 12, the participant using the terminal device 12 can check the settings and review various settings related to the audio output of the terminal device 12. For example, improvements can be made by having the participant check and review whether the volume of the terminal device 12 is set low, whether the speaker selected as the output destination is the intended speaker, whether there is a malfunction such as a breakdown in the headset, etc.

[0065] Next, Fig. 10 is a system configuration diagram showing the configuration of an online conference system Sa according to a second embodiment. Fig. 10 illustrates an example in which four terminal devices, terminal device 11, terminal device 12, terminal device 13, and terminal device 14, participate in the same conference room in an online conference in the online conference system Sa. These terminal devices 11, 12, 13, and 14 each have the same functions as terminal device 11 described using Fig. 2.

[0066] When an online conference is started with these four terminal devices belonging to the same conference room, each terminal device executes the same processing as steps S200 to S202 in Fig. 3. Then, in step S203, the first determination unit 103 of each terminal device determines whether the audio levels of the audio data transmitted from each terminal device are large or small, and selects a destination for sending feedback based on the determination result. This determination of the large or small may be performed by the first determination unit 103 based on the method (2) described above.

[0067] FIG. 11 is a flowchart illustrating a process for selecting a destination of feedback, and FIG. 12 is a diagram showing an example of a sound level according to sound data.

[0068] Then, the terminal device 11 receives voice data from each of the terminal devices 12, 13, and 14, and reads the maximum possible voice level of the voice data (step S500). The terminal device 11 may obtain data representing the maximum value by receiving it from the conference server 20, or may read it if it has been stored in advance in the storage unit 102. In this example, the maximum value is, for example, 100. The first determination unit 103 of the terminal device 11 determines whether the voice level of the voice data received from the terminal device is larger or smaller than the maximum value.

[0069] 12 is a diagram showing an example of audio levels corresponding to audio data. For example, assume that the peak value of the audio level of the audio data obtained from terminal device 12 is 100, the peak value of the audio level of the audio data obtained from terminal device 13 is 30, and the peak value of the audio level of the audio data obtained from terminal device 14 is 40. The first determination unit 103 determines for each terminal device whether the ratio of the peak value to the maximum value exceeds a reference value (step S502). Here, the ratio of the peak value to the maximum value is 100% for terminal device 12, 30% for terminal device 13, and 40% for terminal device 14. The first determination unit 103 determines whether these values ​​are larger or smaller than a reference value of 50%.

[0070] Based on the determination result of the first determination unit 103, the selection unit 104 does not select terminal device 12 as a recipient of feedback because the percentage of terminal device 12 is 100%, which is greater than the reference value of 50% (step S502-YES). On the other hand, the selection unit 104 selects terminal device 13 and terminal device 14 as recipients of feedback (step S503) because the percentage of terminal device 13 is 30%, which is less than the reference value of 50%, and the percentage of terminal device 14 is 40%, which is also less than the reference value of 50% (step S502-NO).

[0071] After the feedback destination is selected in this manner, the processes from step S204 onward in Fig. 3 are executed, whereby a dialog screen for prompting a terminal device on the audio sending or receiving side, which may be causing the audio not to be heard, to review the audio adjustments and settings can be displayed. Note that the process based on method (2) of the first determination unit 103 described in this embodiment has been described for the case where the number of terminal devices participating in the online conference is four, but it can also be applied when the number of terminal devices is three, or five or more.

[0072] According to the embodiment described above, the magnitude relationship of the voice levels is determined for multiple terminal devices other than the own terminal device. As a result, even when the participant of the own terminal device is not speaking, the terminal device on which the dialog screen is to be displayed can be selected and displayed depending on the cause of the difficulty in hearing. This makes it possible to improve hearing comprehension.

[0073] In the embodiment described above, each terminal device may send a join request to an online conference at approximately the same time, but at least some terminal devices may send a join request late. For example, a terminal device may join the online conference approximately 10 minutes after the conference has started. Even in such a case, the processes shown in Figures 3, 6, 7, 8, and 11 can be executed for all terminal devices, including those that joined midway. This allows the cause of the difficulty in hearing to be identified, including for terminal devices that joined midway.

[0074] Furthermore, in the above-described online conference system S and online conference system Sa, among the terminal devices participating in the online conference, all terminal devices may have transmitted voice data in response to participants' speech during the evaluation period. Furthermore, there may be terminal devices participating in the online conference during the evaluation period but not transmitting voice data because no participants spoke. For example, if there are 10 participants attending the online conference, each using a single terminal device, and at least two of the participants have spoken during a certain evaluation period, the cause can be isolated for these six participants. In this case, if there is a terminal device that did not transmit voice data during the evaluation period, the number remaining after excluding it from the total number of terminal devices connected to the online conference can be entered into variable N. This allows the terminal devices participating in the online conference that have spoken during the evaluation period to be targeted, and the terminal devices that have not spoken can be excluded from the target. Then, the audio level relationship of the audio data for the terminal devices that have spoken can be determined, and the cause can be isolated. In this case, the terminal device that is not transmitting voice data can determine the magnitude relationship of the voice levels of the voice data received from devices other than the terminal device itself and transmit feedback based on the determination result. This makes it possible to resolve a situation where the voice of one of the other terminal devices is difficult to hear even when the terminal device that is not transmitting voice data is speaking from a terminal device other than the terminal device itself.

[0075] According to the embodiment described above, when at least two or more of the terminal devices participating in an online conference are transmitting audio data during the evaluation period, it is possible to recognize whether the audio based on the audio data transmitted from the terminal device to other terminal devices is difficult to hear on the other terminal devices, and to prompt the other terminal devices to change the audio settings.

[0076] Furthermore, according to the first and second embodiments described above, in the conference system, participants save the audio level values ​​of audio data received from terminal devices other than the terminal device they use. Then, based on the differences in audio levels of the other terminal devices, a terminal device with a lower audio level than the other terminal devices is selected and feedback is transmitted. If the feedback is received from multiple terminal devices, the terminal device that receives the feedback determines that there is a problem with its own terminal device and displays a dialog box on its display screen prompting the participant to adjust and configure the audio settings, thereby displaying a message suggesting that the cause may be on its own side. On the other hand, if the feedback is received from only one specific terminal device, it determines that there is a problem on the side of the terminal device that sent the feedback and transmits a response signal to the feedback to the transmitting terminal device. As a result, the terminal device that receives the response signal can display a message on its display screen suggesting that the cause may be on its own side, i.e., the transmitting terminal device.

[0077] Next, Fig. 13 is a schematic functional block diagram showing the configuration of a terminal device used in an online conference system according to a third embodiment. The system configuration of the online conference system according to the third embodiment is the same as that of Fig. 1 or 10, but differs in that the functions of each terminal device are the functions of the terminal device shown in Fig. 13. Also, Fig. 13 describes the configuration of terminal device 11a as an example, but other terminal devices participating in the same conference room as the online conference in which terminal device 11a participates also have similar functions.

[0078] Third Embodiment In Fig. 13, the same functions as those in Fig. 2 are denoted by the same reference numerals, and their description will be omitted. In Fig. 13, the instruction input unit 13a receives an instruction indicating that, among sounds output from the audio output unit 112, a sound is difficult to hear because the audio level corresponding to the audio data is lower than that of other terminal devices, and an instruction indicating which terminal device is the source of the difficult-to-hear sound. The instruction input unit 13a receives the instruction by receiving an operation input from a participant using the terminal device 11a. The instruction input unit 13a may be, for example, at least one of an input device such as a keyboard, a mouse, or a touch panel.

[0079] The selection unit 104a selects the terminal device that is the sender of the difficult-to-hear sound based on an instruction input from the instruction input unit 13a. The first transmission unit 105a transmits an evaluation result indicating that the sound is difficult to hear to the terminal device selected by the selection unit 104a. The second determination unit 107a determines, based on the received evaluation result, whether the cause of the difficult-to-hear sound is the terminal device itself or the terminal device that sent the evaluation result.

[0080] When multiple terminal devices functioning as the terminal device 11a described above are connected to the conference server 20 and an online conference is initiated with at least three terminal devices belonging to the same conference room, each terminal device executes the same processing as steps S200 to S202 in FIG. 3 . FIG. 14 is a flowchart illustrating the operation of the terminal device. This flowchart is executed after the processing up to step S202 shown in FIG. 3 has been executed. After step S202, each terminal device receives, based on an instruction from a participant using the terminal device, an instruction indicating that the sound is difficult to hear and an instruction as to which terminal device is the source of the difficult-to-hear sound (step S601). Here, if the sound is difficult to hear, it is sufficient to specify the terminal device; if the sound from all terminal devices can be heard normally, no input is required.

[0081] Based on the instruction input from the instruction input unit 13a, the selection unit 104a selects the terminal device selected as the sender of the difficult-to-hear sound as the terminal device to which feedback will be sent (step S602). After this step, the process proceeds to step 204 of FIG. 3 . This allows the first transmission unit 105a to transmit an evaluation result indicating that the sound is difficult to hear to the terminal device selected as the sender of the feedback. Then, for a terminal device that has received feedback from multiple different terminal devices, a dialogue screen is displayed prompting the participant to improve the audio input function, such as the microphone, of the terminal device. This allows participants who check this dialogue screen to review microphone settings, etc. On the other hand, if only one terminal device has sent feedback, the terminal device sends a response signal to the sender of the feedback. This allows the terminal device that received the response signal to display a dialogue screen prompting the participant to improve the audio output function of the terminal device. This allows participants who check this dialogue screen to review the audio output settings of the terminal device.

[0082] FIG. 15 is a diagram illustrating an example of a display screen displayed on the output unit 108 of the terminal device 11a. This display screen lists the names of the other terminal devices participating in the online conference. Here, the names of participant B (reference numeral 601), participant C (reference numeral 602), and participant D (reference numeral 603) are displayed. Input fields are also displayed adjacent to the names of these terminal devices (reference numerals 611, 612, and 613). If a participant using the terminal device has difficulty hearing a participant, the participant enters a check mark in the corresponding input field. If there are no participants whose voices are difficult to hear, the participant does not need to enter a check mark. If there are multiple participants whose voices are difficult to hear, the participant can enter a check mark for each of the multiple terminal devices. The participant then presses the send button (reference numeral 620) via the input device. This causes feedback to be sent to the selected terminal device.

[0083] In the third embodiment, an evaluation period may also be set, and the cause may be identified based on feedback transmitted during the evaluation period.

[0084] According to the third embodiment, participants are asked to input whether they find it difficult to hear the voices of other participants, so that feedback can be sent based on the ease of hearing perceived by the participants.

[0085] Furthermore, according to the first to third embodiments described above, when an online conference is in progress, a participant who has few opportunities to speak may feel reluctant to say, "I can't hear you," to all other participants, because saying so may give the other participants the impression that they are suddenly making an unrelated remark. This may result in the participant being forced to participate in the conference without improving their difficulty hearing the audio. In contrast, according to the above-described embodiments, participants do not need to directly inform other participants that they are having difficulty hearing the audio, thereby reducing the burden on the participants.

[0086] Furthermore, if one of multiple participants in an online conference is hearing other participants whose voices are all low, that participant may be able to resolve the difficulty in hearing by increasing the volume on the terminal device they are using.

[0087] However, if one participant among multiple participants has a loud voice and another participant has a quiet voice, trying to raise the volume to match the quieter participant's voice will make the louder participant's voice too loud, making it difficult to raise the volume any further. In this case, the quieter participant's voice may not necessarily be audible. As such, when there is a difference in the volume (audio level) of the voices of multiple participants in an online conference, it can be difficult to hear at least some of the participants.

[0088] In contrast, according to the first to third embodiments described above, regardless of whether a participant is speaking or not, when there is variation in the voice levels of multiple other participants, it is possible to determine whether the cause of the variation is the speaking participant or the listening participant, and to transmit a message to the terminal device of the participant according to the determination result to eliminate the variation. This allows the participant to check the message and take action to eliminate the cause.

[0089] 16 is a schematic block diagram showing the configuration of an information processing device 11b used in an online conference system according to a fourth embodiment. The information processing device 11b is an information processing device used by participants, and functions as part of the conference system by being connected to a conference server to which the information processing device is communicably connected.

[0090] The first determination unit 103b determines whether the voice level corresponding to the voice data based on the speech content transmitted from at least two information processing devices other than the information processing device itself is larger or smaller than the voice level of the target information processing device. Based on the determination result, the selection unit 104b selects the information processing terminal that is the sender of the voice data whose voice level does not reach the standard. The first transmission unit 105b transmits an evaluation result indicating that the voice level does not reach the standard to the selected information processing terminal. The first reception unit 106b receives the evaluation result from the information processing device other than the information processing device itself. Based on the received evaluation result, the second determination unit 107b determines whether the cause of the voice level not reaching the standard is the information processing device itself or the information processing device that sent the evaluation result. If it is determined based on the determination result by the second determination unit 107b that the cause lies in the information processing device itself, the output unit 108b outputs a notification indicating that the cause lies in the information processing device itself to the information processing device itself. If it is determined based on the result determined by the second determination unit 107b that the cause is not the information processing device itself, the second transmission unit 109b sends a notification to the information processing device that sent the evaluation result indicating that the cause is in the information processing device that sent the evaluation result.

[0091] In the above-described first to fourth embodiments, the selection unit 104, the selection unit 104a, the first transmission unit 105, the first transmission unit 105a, the first reception unit 106, the second determination unit 107, the second determination unit 107a, the second transmission unit 109, the second reception unit 110, the setting adjustment unit 113, and the control unit 115 are provided in the terminal device. However, at least one of these functions may be provided in the conference server 20. Furthermore, one of these functions may be realized by cooperation between the terminal device and the conference server 20.

[0092] 1 may be recorded on a computer-readable recording medium, and the program may be read into a computer system and executed to perform construction management. Note that the term "computer system" here includes hardware such as the OS and peripheral devices.

[0093] Furthermore, if a WWW system is used, the term "computer system" also includes the homepage provision environment (or display environment). Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes devices that retain programs for a certain period of time, such as volatile memory within a computer system that serves as a server or client. The program may be one that realizes part of the aforementioned functions, or it may be one that realizes the aforementioned functions in combination with a program already stored in the computer system. The program may also be stored on a designated server, and distributed (e.g., downloaded) via a communication line in response to requests from other devices.

[0094] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention.

[0095] 11...Terminal device, 11a...Terminal device, 11b...Information processing device, 12...Terminal device, 13...Terminal device, 14...Terminal device, 20...Conference server, 101...Communication unit, 102...Storage unit, 103...First determination unit, 103a...Instruction input unit, 103b...First determination unit, 104...Selection unit, 104a...Selection unit, 104b...Selection unit, 105...First transmission unit, 105a...First transmission unit, 105b First transmission unit, 106, first reception unit, 106b, first reception unit, 107, second determination unit, 107a, second determination unit, 107b, second determination unit, 108, output unit, 108b, output unit, 109, second transmission unit, 109b, second transmission unit, 110, second reception unit, 111, audio input unit, 112, audio output unit, 113, setting adjustment unit, 114, input unit, 115, control unit, 202, storage unit

Claims

1. An information processing device in a conference system used by a participant and connected in a communicative manner, the information processing device having: a first judgment unit that judges a voice level of voice data based on speech content transmitted from at least two information processing devices other than the information processing device itself; a selection unit that selects an information processing terminal that is the sender of voice data whose voice level does not reach a standard based on the judgment result; a first transmission unit that transmits an evaluation result indicating that the voice level does not reach the standard to the selected information processing terminal; a first reception unit that receives the evaluation result from an information processing device other than the information processing device itself; a second judgment unit that judges, based on the received evaluation result, whether the cause of the voice level not reaching the standard is the information processing device itself or the information processing device that sent the evaluation result; and an output unit that outputs a notification to the information processing device itself indicating that the cause is the information processing device itself when it is judged that the cause lies in the information processing device itself based on the result judged by the second judgment unit.

2. The information processing device of claim 1, further comprising: a second transmission unit which, when it is determined based on the result determined by the second determination unit that the cause is not the information processing device itself, sends a notification to the information processing device that sent the evaluation result indicating that the cause is in the information processing device that sent the evaluation result.

3. An information processing device as described in claim 1, further comprising a second receiving unit that receives a notification sent from an information processing device other than the information processing device itself, and the output unit outputs a notification indicating that the cause is a problem with the sound collection function of the information processing device itself based on the received notification.

4. The information processing device according to claim 3, wherein the first judgment unit calculates the difference in voice levels corresponding to voice data received from at least two information processing devices other than the information processing device itself, for each of the received voice data transmitted from different information processing devices, and judges whether the calculated difference is larger than a reference value; and the selection unit selects as the destination the information processing device that is the source of the voice data having a smaller voice level in a combination in which the difference in voice levels based on the magnitude relationship is equal to or greater than the reference value.

5. The information processing device according to claim 3, wherein the first judgment unit judges whether the audio level corresponding to the audio data transmitted from at least two information processing devices other than the information processing device itself is greater than a predetermined reference value, and the selection unit selects as the destination the information processing device that is the source of the audio data that is judged to be less than the reference value based on the magnitude relationship.

6. An information processing device as described in claim 4 or claim 5, wherein the first judgment unit judges the magnitude relationship of the voice level according to the voice data based on the speech content within a certain period of time in the past from the time to be judged.

7. The information processing device according to claim 6, wherein the first judgment unit calculates an average voice level for each source of voice data based on speech content within the certain period of time in the past, and judges the magnitude relationship using the calculated average value.

8. The information processing device according to any one of claims 1 to 5, wherein the audio level is a magnitude of the amplitude of the audio.

9. An information processing device in a conference system used by a participant and communicatively connected to the information processing device, comprising: an audio output unit that outputs audio in response to audio data based on speech content transmitted from at least two information processing devices other than the information processing device itself; an instruction input unit that receives an instruction indicating that, among the sounds output from the audio output unit, a sound is difficult to hear because the audio level corresponding to the audio data is lower than that of the other information processing devices, and an instruction as to which information processing device is the source of the difficult-to-hear sound; a selection unit that selects the information processing device that is the source of the difficult-to-hear sound based on the received instruction; a first transmission unit that transmits an evaluation result indicating that the sound is difficult to hear to the selected information processing terminal; a first reception unit that receives the evaluation result from an information processing device other than the information processing device itself; and a second determination unit that determines, based on the received evaluation result, whether the cause of the difficult-to-hear sound is the information processing device itself or the information processing device that transmitted the evaluation result. and an output unit that, when it is determined that the cause is in the information processing device based on a result determined by the second determination unit, outputs a notification to the information processing device indicating that the cause is in the information processing device.

10. A conference system in which information processing devices used by participants are communicatively connected, comprising: a first judgment unit that judges a voice level of voice data based on speech content transmitted from at least two information processing devices other than a first information processing device; a selection unit that selects an information processing terminal that is a sender of voice data whose voice level does not reach a standard based on the judgment result; a first transmission unit that transmits an evaluation result indicating that the voice level does not reach the standard to the selected information processing terminal; a first reception unit that receives the evaluation result from an information processing device other than the first information processing device; a second judgment unit that judges, based on the received evaluation result, whether the cause of the voice level not reaching the standard is the first information processing device or the information processing device that sent the evaluation result; and an output unit that outputs a notification to the information processing device itself, when it is determined that the cause is the first information processing device based on the result determined by the second judgment unit.

11. An information processing method in a conference system in which an information processing device used by a participant is communicatively connected, comprising: a first determination step of determining a voice level of voice data based on speech content transmitted from at least two information processing devices other than a first information processing device; a selection step of selecting an information processing terminal that is the sender of voice data whose voice level does not reach a standard based on the determination result; a first transmission step of transmitting an evaluation result indicating that the voice level does not reach the standard to the selected information processing terminal; a first reception step of receiving the evaluation result from an information processing device other than the first information processing device; a second determination step of determining, based on the received evaluation result, whether the cause of the voice level not reaching the standard is the first information processing device or the information processing device that sent the evaluation result; and an output step of outputting a notification to the information processing device corresponding to the cause being the first information processing device when it is determined that the cause lies in the first information processing device based on the result determined by the second determination step.

Images