Information processing device and information processing method

Abstract

To appropriately operate audio input in a web conference.SOLUTION: An information processing device comprises an application control unit for running a web conference application using video and audio, an audio acquisition unit for acquiring the audio of users participating in the web conference, and a detection unit for detecting a line of sight or a face direction of a user relative to a display unit that displays the video. When the detection unit detects that the line of sight or face direction of the user is directed in another direction from a display screen of the display unit, the audio acquisition unit restricts the audio acquired by the audio acquisition unit.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] It has become common to work in various places, such as working from home, but there are cases where unintended voices are picked up during a web conference. Patent Document 1 discloses a technology that mutes input voice when the voice of a person other than the user is detected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-92765 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology described in Patent Document 1, muting is performed after the speech of a speaker other than the user is detected, so the speech of others may be temporarily transmitted as conference audio. In addition, when operating the audio mute function set in the web conference application, unintended audio input is often sudden, and the operation may not be performed in time.

[0005] An object of the present invention is to provide an information processing device and an information processing method that enable appropriate manipulation of voice input in a web conference. [Means for solving the problem]

[0006] The information processing device of the present invention comprises an application control unit that operates a WEB conference application using video and audio, an audio acquisition unit that acquires the audio of users participating in the WEB conference, and a detection unit that detects the direction of the user's line of sight or face relative to a display unit that displays the video, and when the detection unit detects that the user's line of sight or face direction has been directed in another direction away from the display screen of the display unit, the audio acquisition unit restricts the audio acquired by the audio acquisition unit.

[0007] The information processing method of the present invention includes a step of acquiring the voice of a user participating in a WEB conference using video and audio while a WEB conference application is running, a step of detecting the direction of the user's line of sight or face relative to a display unit displaying the video, and a step of restricting the acquisition of the user's voice when it is detected that the user's line of sight or face direction is directed in another direction away from the display screen of the display unit. Effect of the Invention

[0008] According to the present invention, audio input in a web conference can be appropriately controlled. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of the information terminal device according to the first embodiment. [Diagram 3] FIG. 3 is a block diagram illustrating an example of the configuration of a server device according to the first embodiment. [Figure 4] FIG. 4 is a block diagram illustrating an example of the configuration of the information processing device according to the first embodiment. [Diagram 5] FIG. 5 is a flowchart showing an information processing method executed by the information processing device according to the first embodiment. [Figure 6] FIG. 6 is a diagram for explaining a first method for limiting an input voice according to the first embodiment. [Figure 7] FIG. 7 is a diagram for explaining a second method for limiting an input voice according to the first embodiment. [Figure 8] FIG. 8 is a flowchart showing an information processing method according to the second embodiment. [Figure 9] FIG. 9 is a flowchart showing an information processing method according to the third embodiment. [Figure 10] FIG. 10 is a flowchart showing an information processing method according to the fourth embodiment. [Figure 11] FIG. 11 is a diagram for explaining a first method for limiting an input voice according to the fourth embodiment. [Figure 12] FIG. 12 is a diagram for explaining a second method for restricting an input voice according to the fourth embodiment. [Figure 13] FIG. 13 is a flowchart showing an information processing method according to a modification of the fourth embodiment. [Figure 14] FIG. 14 is a diagram for explaining a first method for limiting an input voice according to a modification of the fourth embodiment. [Figure 15] FIG. 15 is a diagram for explaining a second method for restricting an input voice according to a modification of the fourth embodiment. [Figure 16] FIG. 16 is a flowchart showing an information processing method according to the fifth embodiment. [Figure 17] FIG. 17 is a block diagram showing an example of the configuration of an information terminal device according to the sixth embodiment. [Figure 18] FIG. 18 is a block diagram illustrating an example of the configuration of an information processing device according to the sixth embodiment. [Figure 19] FIG. 19 is a flowchart showing an information processing method according to the sixth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to this embodiment, and in the following embodiments, the same components are designated by the same reference numerals, and duplicated explanations will be omitted.

[0011] [First embodiment] (Information Processing System) An example of the configuration of an information processing system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of an information processing system according to the first embodiment.

[0012] 1, an information processing system 1 includes a plurality of information terminal devices 10 and a server device 12. Each information terminal device 10 can hold a WEB conference with the other information terminal devices 10 via the server device 12 by starting a WEB conference application.

[0013] (Information terminal device) An example of the configuration of an information terminal device according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of an information terminal device according to the first embodiment.

[0014] 2, the information terminal device 10 includes a camera 20, a microphone 22, an operation unit 24, a communication unit 26, a display unit 28, and a terminal control device 30. Examples of the information terminal device 10 include information terminal devices such as a desktop personal computer (PC), a notebook PC, and a tablet terminal.

[0015] The camera 20 captures an image of a user standing in front of the information terminal device 10. The camera 20 captures an image of a user using the information terminal device 10. The camera 20 is, for example, an in-camera provided on the display unit 28 side. The camera 20 may be an external camera connected to the information terminal device 10.

[0016] The microphone 22 picks up sounds around the information terminal device 10, such as sounds spoken by a user of the information terminal device 10. The microphone 22 outputs audio information relating to the picked-up sounds to the audio acquisition unit 44 of the terminal control device 30.

[0017] The operation unit 24 accepts various input operations for the information terminal device 10. The operation unit 24 outputs an operation signal corresponding to the accepted input operation to the terminal control device 30. The operation unit 24 includes, for example, a keyboard, a mouse, a touch panel, buttons, switches, etc. When a touch panel is used as the operation unit 24, the operation unit 24 is disposed on the display unit 28.

[0018] The communication unit 26 communicates with an external device via a wired or wireless network. The communication unit 26 communicates with the server device 12, for example, via a wired or wireless network.

[0019] The display unit 28 displays various types of video including characters and images on a display screen. The display unit 28 is, for example, a display including a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display.

[0020] The terminal control device 30 controls each part of the information terminal device 10. The terminal control device 30 has, for example, an information processing device such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), and a storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The terminal control device 30 executes a program that controls the operation of the information terminal device 10 according to the present invention. The terminal control device 30 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The terminal control device 30 may be realized by a combination of hardware and software.

[0021] The terminal control device 30 has functional blocks realized by circuits or executed programs, including a shooting control unit 40, a communication control unit 42, an audio acquisition unit 44, a display control unit 46, an operation control unit 48, an application control unit 50, a memory unit 52, and a detection unit 54.

[0022] The photographing control unit 40 controls the camera 20 to photograph an image. The photographing control unit 40 acquires image data relating to the image photographed by the camera 20. Specifically, the photographing control unit 40 acquires image data photographed by the camera 20 of a user using the information terminal device 10. The image data acquired by photographing with the camera 20 is image data of a moving image.

[0023] The communication control unit 42 controls the communication unit 26 to execute communication between the information terminal device 10 and an external device. Specifically, the communication control unit 42 controls, for example, the communication unit 26 to control communication between the information terminal device 10 and the server device 12. The communication control unit 42 controls, for example, the communication unit 26 to transmit and receive information about a WEB conference with another information terminal device 10 to and from the server device 12. The communication control unit 42 controls, for example, the communication unit 26 to transmit to the server device 12 voice information about the speech of a user using the information terminal device 10 acquired by the voice acquisition unit 44 and image data of a user using the information terminal device 10 acquired by the image capture control unit 40. The communication control unit 42 controls, for example, the communication unit 26 to receive information necessary for a WEB conference, such as speech and image data of a user of another information terminal device 10, from the server device 12.

[0024] The voice acquisition unit 44 acquires voice information based on the voice picked up by the microphone 22. The voice acquisition unit 44 acquires voice information including the speech of the user who uses the information terminal device 10. The voice acquisition unit 44 acquires an analog waveform of the voice picked up by the microphone 22, and converts it into digital data for control by the terminal control device 30.

[0025] The display control unit 46 controls the display unit 28 to display various videos including characters and images. For example, the display control unit 46 controls the display unit 28 to display information obtained by executing a WEB conference application (hereinafter, a WEB conference screen). The display control unit 46 may generate a WEB conference screen and display it on the display unit 28, or the display control unit 46 may display a WEB conference screen generated by the server device 12 on the display unit 28.

[0026] The operation control unit 48 acquires an operation signal corresponding to an operation accepted by the operation unit 24 from the operation unit 24. The operation control unit 48 outputs a control signal corresponding to the acquired operation signal to each unit.

[0027] The application control unit 50 controls the operation of various applications that run on the information terminal device 10. The application control unit 50 controls the operation of applications such as a WEB conference tool and a chat tool. The application control unit 50 may be configured to run an application such as a WEB conference tool in the server device 12 instead of running an application such as a WEB conference tool in the information terminal device 10. The application control unit 50 may also run an application such as a WEB conference tool in a distributed manner in the information terminal device 10 and the server device 12. In other words, the application control unit 50 may be a functional block realized only in the information terminal device 10, as well as a functional block realized in the server device 12, or a functional block realized in the information terminal device 10 and the server device 12.

[0028] The storage unit 52 stores information such as the contents of calculations and programs of the terminal control device 30. The storage unit 52 is composed of, for example, a RAM, a main storage device such as a ROM, and a storage device such as an SSD (Solid State Drive).

[0029] The detection unit 54 detects the line of sight or the direction of the face of the user of the information terminal device 10. The detection unit 54 detects the line of sight or the direction of the face of the user of the information terminal device 10, for example, from an image captured by the camera 20 provided on the display unit 28 side. The detection unit 54 recognizes the user's face from the image capture data captured by the camera 20 acquired by the image capture control unit 40, for example, and detects the direction of the user's face. The detection unit 54 recognizes the user's face from the image capture data captured by the camera 20 acquired by the image capture control unit 40, for example, and detects the direction of the line of sight from the detected image of the eyes. The detection unit 54 transmits the detection result of the line of sight of the user of the information terminal device 10 to the server device 12.

[0030] (Server device) An example of the configuration of the server device according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the configuration of the server device according to the first embodiment.

[0031] 3, the server device 12 includes a communication unit 60 and a server control device 62. The server device 12 is a general-purpose server device realized by a computer or the like. The server device 12 is a server of a company or the like that provides a web conference tool or a web conference application.

[0032] The communication unit 60 communicates with an external device via a wired or wireless network. The communication unit 60 communicates with the information terminal device 10, for example, via a wired or wireless network.

[0033] The server control device 62 controls each unit of the server device 12. The server control device 62 has, for example, an information processing device such as a CPU or an MPU, and a storage device such as a RAM or a ROM. The server control device 62 executes a program that controls the operation of the server device 12 according to the present invention. The server control device 62 may be realized by an integrated circuit such as an ASIC or an FPGA. The server control device 62 may be realized by a combination of hardware and software.

[0034] The server control device 62 includes, as functional blocks realized by circuits or programs executed, a connection control unit 70, a communication control unit 72, and a storage unit 74. As described above, the server control device 62 may include an application control unit that operates an application such as a web conference tool.

[0035] The connection control unit 70 controls the connection between the server device 12 and the information terminal device 10. The connection control unit 70 establishes a connection between the server device 12 and the information terminal device 10.

[0036] The communication control unit 72 controls the communication unit 60 to execute communication between the server device 12 and an external device. Specifically, the communication control unit 72 controls, for example, the communication unit 60 to control communication between the server device 12 and the information terminal device 10. The communication control unit 72 controls, for example, the communication unit 60 to acquire information indicating a detection result of the line of sight or face direction of a user using the information terminal device 10 from the information terminal device 10. The communication control unit 72 controls, for example, the communication unit 60 to acquire voice information acquired by the voice acquisition unit 44, such as an utterance of a user using the information terminal device 10, from the information terminal device 10. The communication control unit 72 transmits, for example, information indicating a detection result of the line of sight or face direction of a user using the information terminal device 10 to another information terminal device 10. The communication control unit 72 transmits, for example, voice information acquired by the voice acquisition unit 44, such as an utterance of a user using the information terminal device 10, to another information terminal device 10.

[0037] The storage unit 74 stores information such as programs and the contents of calculations performed by the connection control unit 70 and the communication control unit 72. The storage unit 74 is composed of a main storage device such as a RAM and a ROM, and a storage device such as an SSD or a hard disk drive (HDD).

[0038] (Information processing device) An example of the configuration of the information processing device according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the configuration of the information processing device according to the first embodiment.

[0039] 4, the information processing device 80 includes an application control unit 82, a voice acquisition unit 84, and a detection unit 86. The information processing device 80 is a conceptual device realized by the cooperation of either one or both of the terminal control device 30 and the server control device 62. Therefore, either the information terminal device 10 or the server device 12 may be referred to as the information processing device 80, and the information processing system 1 consisting of the information terminal device 10 and the server device 12 may be referred to as the information processing device 80. Therefore, the application control unit 82, the voice acquisition unit 84, and the detection unit 86 included in the information processing device 80 are realized by either one or both of the terminal control device 30 and the server control device 62.

[0040] The application control unit 82 operates a WEB conference application using video and audio. The application control unit 50 controls the operation of various applications operating on the information terminal device 10. The application control unit 50 controls the operation of applications such as a WEB conference tool and a chat tool, for example. Instead of operating an application such as a WEB conference tool on the information terminal device 10, the application control unit 50 may be configured to operate an application such as a WEB conference tool on the server device 12. Furthermore, the application control unit 82 may operate an application such as a WEB conference tool in a distributed manner on the information terminal device 10 and the server device 12.

[0041] The voice acquisition unit 84 acquires the voice of each user participating in the WEB conference. When the detection unit 86 detects that the user's line of sight or face direction is directed in another direction from the display screen of the display unit 28, the voice acquisition unit 84 restricts the voice acquired by the voice acquisition unit 84. Restricting the voice acquired by the voice acquisition unit 84 refers to a process of not outputting unnecessary voice to the information terminal device 10 of the other user participating in the WEB conference. As a specific example, the voice acquisition unit 84 restricts the voice by stopping the collection of sound by the microphone 22. As another example, the voice acquisition unit 84 restricts the voice by performing volume reduction or muting processing on the voice collected by the microphone 22. As another example, the voice acquisition unit 84 restricts the voice by performing noise cancellation processing on the voice collected by the microphone 22 and reducing voice other than the voice near the microphone 22 for the voice collected by the microphone 22. The restriction of the voice acquired by the voice acquisition unit 84 is not limited to the above-mentioned method, and any method can be applied as long as it can restrict the voice transmitted to the other information terminal devices 10 participating in the WEB conference.

[0042] The detection unit 86 detects the direction of the line of sight or face of the user of each information terminal device 10 with respect to the display screen of the display unit 28. The detection unit 86 detects that the user of each information terminal device 10 has turned his / her line of sight or his / her face in another direction from the display screen of the display unit 28. The detection unit 86 recognizes the user's face from the shooting data captured by the camera 20 acquired by the shooting control unit 40 of each information terminal device 10, for example, and detects the direction of the user's face. The detection unit 86 recognizes the user's face from the shooting data captured by the camera 20 acquired by the shooting control unit 40 of each information terminal device 10, for example, and detects the direction of the line of sight from the detected image of the eyes. The detection unit 86 acquires information indicating the direction of the line of sight or face of the user of each information terminal device 10 with respect to the display screen of the display unit 28 and transmits it to the server device 12.

[0043] The detection unit 86 detects that the user of each information terminal device 10 has turned his / her line of sight or his / her face in another direction for a predetermined period of time or longer from the display screen of the display unit 28. The detection unit 86 detects the operation time until the user of each information terminal device 10 turns his / her line of sight or his / her face in another direction from the display screen of the display unit 28.

[0044] (Information processing method) The information processing method according to the first embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the information processing method executed by the information processing device according to the first embodiment.

[0045] The process shown in FIG. 5 is started when a Web conference in which multiple participants participate is started.

[0046] When the WEB conference is started, the application control unit 82 starts outputting information to each participant of the WEB conference, that is, starts displaying information about the WEB conference and outputting audio (step S10), and then proceeds to step S12.

[0047] When the output of the WEB conference information is started in step S10, the output of the WEB conference information continues to each participant of the WEB conference until the WEB conference ends with a Yes in step S24 described later. Specifically, the image acquired by the shooting control unit 40 of each information terminal device 10 is displayed on the display unit 28 of the other information terminal device 10, and the sound acquired by the sound acquisition unit 44 of each information terminal device 10 is output to the sound output unit (not shown) of the other information terminal device 10.

[0048] The voice acquisition unit 84 judges whether or not the voice input of the information terminal device 10 used by each participant of the WEB conference is turned on (step S12). The state in which the voice input of the information terminal device 10 is turned on means that the voice picked up by the microphone 22 of the information terminal device 10 is transmitted to the other information terminal device 10. The state in which the voice input of the information terminal device 10 is turned off means that the microphone 22 of the information terminal device 10 does not pick up any voice, or that even if the microphone 22 of the information terminal device 10 picks up any voice, the voice is not transmitted to the other information terminal device 10. If it is judged that the voice input of the information terminal device 10 is turned on (step S12; Yes), the process proceeds to step S14. If it is not judged that the voice input of the information terminal device 10 is turned on (step S12; No), the process proceeds to step S24.

[0049] If the answer to step S12 is Yes, the detection unit 86 starts detecting the line of sight of the user who is in front of the information terminal device 10 with voice input turned on (step S14). The detection unit 86 may start detecting the direction of the user's face. Then, the process proceeds to step S16. Hereinafter, the description of either the direction of the user's line of sight or the direction of the user's face can be applied to both or either of the direction of the user's line of sight and the direction of the user's face.

[0050] The detection unit 86 determines whether or not the user's line of sight is directed in another direction from the display screen of the display unit 28 (step S16). The other direction may include the up-down direction and the left-right direction with respect to the display screen of the display unit 28. The detection unit 86 may determine that the user has been directed in the other direction, for example, when the user has been directed in the other direction for a predetermined period of time (for example, 1.0 second) or more. The detection unit 86 may determine whether or not the orientation of the user's face is directed in the other direction from the display screen of the display unit 28. If it is determined that the line of sight is directed in the other direction from the display screen of the display unit 28 (step S16; Yes), the process proceeds to step S18. If it is not determined that the line of sight is directed in the other direction from the display screen of the display unit 28 (step S16; No), the process proceeds to step S24.

[0051] If the answer to step S16 is Yes, the voice acquisition unit 84 restricts the input voice (step S18). The restriction of the input voice refers to a restriction of the voice acquired by the voice acquisition unit 84. FIG. 6 is a diagram for explaining a first method of restricting the input voice according to the first embodiment. In FIG. 6, the horizontal axis indicates time, and the vertical axis indicates the volume of the input voice. Usually, the volume of the voice acquired by the voice acquisition unit 84 does not maintain a constant value, but for ease of explanation, other embodiments will be described as an example in which a voice with a constant volume is acquired. As shown in FIG. 6, the voice acquisition unit 84 restricts the input voice by reducing the volume so that the volume is small between timing t1 and timing t2. As a result, the volume of the voice acquired by the voice acquisition unit 84 is small between timing t1 and timing t2. As a result, between timing t1 and timing t2, the voice is transmitted to another information terminal device 10 as a voice with a lower volume than the period other than the period between timing t1 and timing t2. The voice acquisition unit 84 may turn on a noise cancellation function between timing t1 and timing t2. By turning on the noise cancellation function, sounds other than those near the microphone 22 are restricted as shown in Fig. 6. Timing t1 shown in Fig. 6 and Fig. 7 is the timing when the determination in step S16 is Yes, and timing t2 is the timing when the determination in step S20 is Yes.

[0052] FIG. 7 is a diagram for explaining a second method of limiting the input voice according to the first embodiment. In FIG. 7, the horizontal axis indicates time, and the vertical axis indicates the volume of the input voice. As shown in FIG. 7, the voice acquisition unit 84 limits the volume of the input voice to be silent between timing t1 and timing t2. In this case, the voice acquisition unit 84 does not acquire voice between timing t1 and timing t2. Also, the voice acquisition unit 84 may limit the input voice by muting it to be silent between timing t1 and timing t2. As a result, the voice of the WEB conference is not transmitted between timing t1 and timing t2. Then, proceed to step S20.

[0053] The detection unit 86 determines whether or not the user's gaze has returned to the display screen of the display unit 28 (step S20). If it is determined that the user's gaze has returned to the display screen of the display unit 28 (step S20; Yes), the process proceeds to step S22. If it is not determined that the user's gaze has returned to the display screen of the display unit 28 (step S20; No), the detection unit 86 repeats the process of step S20.

[0054] If it is determined as Yes in step S20, the voice acquiring unit 84 returns the volume of the input voice to the original volume. In other words, the restriction on the input voice imposed in step S18 is lifted (step S22). Then, the process proceeds to step S24.

[0055] The information processing device 80 judges whether the WEB conference has ended (step S24). If it is judged that the WEB conference has ended (step S24; Yes), the process in FIG. 5 ends. If it is not judged that the WEB conference has ended (step S24; No), the process proceeds to step S12.

[0056] As described above, the first embodiment can reduce the voice detection level by the voice acquisition unit when a user turns their gaze or face toward a child when the child approaches during a web conference while working from home, etc. This makes it possible to prevent unnecessary speech by others from being transmitted to the web conference as conference voice.

[0057] [Second embodiment] In the second embodiment, the detection unit 86 may detect a predetermined action (gesture) of the user. In the second embodiment, when the user's line of sight is directed in another direction from the display screen of the display unit 28 and the predetermined action of the user is detected, a process of restricting the input voice is performed. The details of the predetermined action will be described later.

[0058] (Information processing method) An information processing method according to the second embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the information processing method according to the second embodiment. In the second embodiment, a detection unit 86 detects a predetermined action of a user from image data captured by the camera 20 of each information terminal device 10.

[0059] The processes from step S30 to step S36 shown in FIG. 8 are the same as the processes from step S10 to step S16 shown in FIG. 5, respectively, and therefore will not be described.

[0060] If the result of the determination in step S36 is Yes, the detection unit 86 determines whether or not a predetermined action of the user directing the gaze in another direction has been detected (step S38). The predetermined action includes, for example, an action to stop an approaching child, such as holding a hand over the screen, during a web conference during telecommuting, and an action to suppress the child's voice, such as putting a finger over the mouth. The predetermined action includes, for example, an action to stop a child by moving the mouth, such as saying "wait a moment" without speaking, during a web conference during telecommuting. The predetermined action may be set in advance, or the detection unit 86 may determine whether or not it is a specific action using a machine learning method such as deep learning. If the predetermined action of the user directing the gaze in another direction has been detected (step S38; Yes), the process proceeds to step S40. If the predetermined action of the user directing the gaze in another direction has not been detected (step S38; No), the process proceeds to step S46.

[0061] The processes from step S40 to step S46 shown in FIG. 8 are the same as the processes from step S18 to step S24 shown in FIG. 5, respectively, and therefore will not be described.

[0062] As described above, in the second embodiment, during a web conference while working from home, when a child approaches, the user turns his / her gaze or face toward the child and a predetermined action is detected, the voice acquired by the voice acquisition unit can be restricted. In this way, the second embodiment prevents the voice from being restricted when the user turns his / her gaze or face away from the display screen of the display unit 28 in a situation where it is not necessary to restrict the voice. This makes it possible to more appropriately prevent unnecessary speech and the like from being transmitted as conference voice.

[0063] [Third embodiment] In a third embodiment, the detection unit 86 may detect a predetermined sound. In the third embodiment, when the user's line of sight is directed in another direction from the display screen of the display unit 28 and a predetermined sound is detected, a process of reducing the volume of the input sound is performed. The details of the predetermined sound will be described later.

[0064] (Information processing method) An information processing method according to the third embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the information processing method according to the third embodiment. In the third embodiment, the detection unit 86 detects a predetermined voice from the voice acquired by the voice acquisition unit 44 of each information terminal device 10.

[0065] The processes from step S50 to step S56 shown in FIG. 9 are the same as the processes from step S10 to step S16 shown in FIG. 5, respectively, and therefore will not be described.

[0066] If the result of the determination in step S56 is Yes, the detection unit 86 determines whether or not a predetermined voice has been detected (step S58). The predetermined voice includes, for example, footsteps of a person other than the user, an intercom ringtone, a truck engine sound, and the like. That is, the predetermined voice is a voice that may possibly be detected as a voice other than that of a user participating in a web conference, such as a voice of a family member or a voice of a delivery company delivering a package. The predetermined voice may be set in advance, or the detection unit 86 may determine whether or not it is a specific voice by using a machine learning method such as deep learning.

[0067] The determination in step S58 may include a determination for a period before the point in time when step S56 is determined as Yes. In this case, step S58 determines whether or not a predetermined sound has been detected during a period from when the line of sight was directed in another direction until a predetermined time ago. In this case, the predetermined time ago is, for example, 5 to 10 seconds ago. Then, the process proceeds to step S60.

[0068] The processes from step S60 to step S66 shown in FIG. 9 are the same as the processes from step S18 to step S24 shown in FIG. 5, respectively, and therefore will not be described.

[0069] As described above, in the third embodiment, when a user turns his / her gaze or face away from the display screen of the display unit during a web conference while working from home, and a predetermined voice is detected, the voice acquired by the voice acquisition unit can be restricted. This prevents the voice from being restricted when the user turns his / her gaze or face away from the display screen of the display unit 28 in a situation where it is not necessary to restrict the voice. This makes it possible to more appropriately prevent unnecessary speech and the like from being transmitted as conference voice.

[0070] [Fourth embodiment] In the fourth embodiment, the detection unit 86 may determine whether or not the duration for which the user's line of sight is directed in another direction from the display screen of the display unit 28 is equal to or longer than a predetermined time. In the fourth embodiment, when the duration for which the user's line of sight is directed in another direction from the display screen of the display unit 28 is equal to or longer than a predetermined time, the input voice is restricted.

[0071] (Information processing method) The information processing method according to the fourth embodiment will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the information processing method according to the fourth embodiment.

[0072] The processes from step S70 to step S76 shown in FIG. 10 are the same as the processes from step S10 to step S16 shown in FIG. 5, respectively, and therefore will not be described.

[0073] If the answer is Yes in step S76, the voice acquisition unit 84 limits the input voice (step S78). The input voice is limited to a level that does not cause the input voice to become silent. For example, the volume of the input voice is reduced to about 30% to 70%. Then, the process proceeds to step S80.

[0074] The detection unit 86 determines whether a predetermined time has passed since the user turned to the other direction (step S80). The predetermined time in this case is, for example, about 1.0 to 2.0 seconds, but is not limited to this. If it is determined that the predetermined time has passed since the user turned to the other direction (step S80; Yes), the process proceeds to step S82. If it is not determined that the predetermined time has passed since the user turned to the other direction (step S80; No), the process proceeds to step S84.

[0075] If the answer is Yes in step S80, the voice acquisition unit 84 further restricts the input voice (step S82). FIG. 11 is a diagram for explaining a first method of restricting the input voice according to the fourth embodiment. In FIG. 11, the horizontal axis indicates time, and the vertical axis indicates the volume of the input voice. As shown in FIG. 11, for example, it is assumed that the detection unit 86 determines that the user faces the other direction at timing t11. In this case, the voice acquisition unit 84 restricts the input voice by performing a process of reducing the input voice from timing t11. It is also assumed that the detection unit 86 determines that the user faces the other direction for a predetermined time period from timing t11 to timing t12. In this case, the voice acquisition unit 84 further restricts the input voice by muting the input voice to silence it from timing t12 to timing t13, at which it is determined that the answer is Yes in step S84.

[0076] FIG. 12 is a diagram for explaining a second method of limiting the input voice according to the fourth embodiment. In FIG. 12, the horizontal axis indicates time, and the vertical axis indicates the volume of the input voice. As shown in FIG. 12, the voice acquisition unit 84 may limit the input voice so that the volume decreases linearly between timing t11 and timing t12. In this case, the voice acquisition unit 84 further limits the input voice by muting the input voice so that it is silent between timing t12 and timing t13. If the user faces another direction for a predetermined period of time or more, there is a high possibility that a voice other than the user's voice will be detected. Therefore, by limiting the input voice stepwise or by changing the level, it is possible to appropriately prevent unnecessary voice from being detected. Then, the process proceeds to step S84.

[0077] The processes from step S84 to step S88 shown in FIG. 10 are the same as the processes from step S20 to step S24 shown in FIG. 5, respectively, and therefore will not be described.

[0078] As described above, the fourth embodiment can restrict input voice when the user turns his / her gaze or face away from the display screen of the display unit for a predetermined period of time or more during a web conference while working from home, etc. This makes it possible to more appropriately prevent unnecessary voices, such as voices of people other than the user, from being transmitted as conference voice in a situation where there is a high possibility that voices of people other than the user will be emitted.

[0079] [Modification of the fourth embodiment] In the fourth embodiment, the input voice is restricted in a situation where there is a high possibility that a voice other than that of the user is detected. In such a situation, when the user returns his / her gaze to the display screen of the display unit 28 and the restriction is released, it may be assumed that the voice other than that of the user continues to be spoken. Therefore, in a modified example of the fourth embodiment, when the user returns his / her gaze to the display screen of the display unit 28, a process of gradually returning the volume of the input voice to the original volume is executed.

[0080] (Information processing method) An information processing method according to a modification of the fourth embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an information processing method according to a modification of the fourth embodiment.

[0081] The processes from step S90 to step S104 shown in FIG. 13 are the same as the processes from step S70 to step S84 shown in FIG. 10, respectively, and therefore will not be described.

[0082] If the answer to step S104 is Yes, the voice acquisition unit 84 increases the volume of the input voice halfway (step S106). FIG. 14 is a diagram for explaining a first method for limiting the input voice according to a modified example of the fourth embodiment. In FIG. 14, the horizontal axis indicates time and the vertical axis indicates the volume of the input voice. As shown in FIG. 14, the voice acquisition unit 84 increases the volume of the input voice halfway at timing t13, which indicates when the answer to step S104 is Yes. Then, the process proceeds to step S108.

[0083] After increasing the volume of the input voice halfway, the voice acquisition unit 84 determines whether a predetermined time has elapsed (step S108). If it is determined that the predetermined time has elapsed (step S108; Yes), the process proceeds to step S110. If it is determined that the predetermined time has elapsed (step S108; No), the process proceeds to step S106. That is, the voice acquisition unit 84 maintains the state in which the volume of the input voice has been increased halfway, for example.

[0084] If the result of the determination in step S108 is Yes, the voice acquisition unit 84 returns the volume of the input voice to the original volume (step S110), that is, releases the restriction on the input voice. In the example shown in FIG. 14, the voice acquisition unit 84 maintains the state in which the volume of the input voice is increased halfway between timing t13 and timing t14, and returns the volume of the input voice to the original volume at timing t14, which indicates the time when the result of the determination in step S108 is Yes. That is, in the example shown in FIG. 14, the voice acquisition unit 84 returns the volume of the input voice to the original volume in stages after the user returns his / her gaze to the display screen of the display unit 28. FIG. 15 is a diagram for explaining a second method of restricting the input voice according to the modified example of the fourth embodiment. As shown in FIG. 15, the voice acquisition unit 84 may gradually increase the volume of the input voice at timing t13, and may return the volume of the input voice to the original volume at timing t14. That is, the voice acquisition unit 84 may linearly return the volume of the input voice to the original volume after the user returns his / her gaze to the display screen of the display unit 28. Then, the process proceeds to step S112.

[0085] The process of step S112 shown in FIG. 13 is the same as the process of step S88 shown in FIG. 10, and therefore a description thereof will be omitted.

[0086] As described above, in the modification of the fourth embodiment, when the volume of the input voice is muted, the volume of the input voice is restored to the original volume after a predetermined time has elapsed since the user's gaze returns to the display screen of the display unit. This makes it possible to more appropriately prevent unnecessary speech voices, such as speech voices of people other than the user, from being transmitted as conference voice in a situation where speech voices of people other than the user are likely to be uttered.

[0087] [Fifth embodiment] A fifth embodiment will be described. Even when participating in a web conference, a user may unconsciously move his / her face and turn the face in various directions. In the fifth embodiment, when a user participating in a web conference unconsciously changes the direction of his / her face, a process is executed so that the input voice is not restricted.

[0088] (Information processing method) An information processing method according to the fifth embodiment will be described with reference to Fig. 16. Fig. 16 is a flowchart showing the information processing method according to the fifth embodiment. In the fifth embodiment, the detection unit 86 detects, from the photographed data photographed by the camera 20 of each information terminal device 10, an operation time when the line of sight or the direction of the user's face is directed in another direction from the display screen of the display unit 28.

[0089] The processes from step S120 to step S126 shown in FIG. 16 are the same as the processes from step S10 to step S16 shown in FIG. 5, respectively, and therefore will not be described.

[0090] If it is determined that the answer is Yes in step S126, the detection unit 86 determines whether or not the operation time when the user's line of sight or the direction of the face is turned in another direction from the display screen of the display unit 28 is less than a predetermined time (step S128). The predetermined time is, for example, the time it takes for the user to turn in another direction from the display screen of the display unit 28, for example, 0.5 seconds. The operation of turning in another direction in an instant is, for example, an operation of turning in the direction of another person who may make an unintended utterance in order to be careful when there is a possibility of an unintended utterance being made in the surroundings during a WEB conference while working from home. In other words, the operation of turning in another direction in an instant does not include an operation of the user turning in another direction from the display screen of the display unit 28 unconsciously. If it is determined that the operation time when turning in another direction is less than the predetermined time (step S128; Yes), the process proceeds to step S130. When it is not determined that the motion time when facing the other direction is less than the predetermined time (step S128; No), the process proceeds to step S136.

[0091] As described above, the fifth embodiment limits the voice acquired by the voice acquisition unit when the user moves his / her gaze or face in another direction on the display screen of the display unit for less than a predetermined time. This makes it possible to prevent the input voice from being limited when the user unconsciously changes the direction of his / her gaze or face.

[0092] [Sixth embodiment] An example of the configuration of an information terminal device according to the sixth embodiment will be described with reference to Fig. 17. Fig. 17 is a block diagram showing an example of the configuration of an information terminal device according to the sixth embodiment.

[0093] As shown in FIG. 17, the information terminal device 10A differs from the information terminal device 10 shown in FIG.

[0094] The speech detection unit 56 detects speech by the user of the information terminal device 10A. The speech detection unit 56 detects whether the user of the information terminal device 10A is continuing to speak. Specifically, the speech detection unit 56 detects input voice with a volume equal to or greater than a predetermined volume, or speech detected by speech recognition. The speech detection unit 56 may be realized by the terminal control device 30A and the server control device 62 of the server device 12.

[0095] (Information processing device) An example of the configuration of an information processing device according to the sixth embodiment will be described with reference to Fig. 18. Fig. 18 is a block diagram showing an example of the configuration of an information processing device according to the sixth embodiment.

[0096] As shown in Fig. 18, information processing device 80A differs from information processing device 80 shown in Fig. 4 in that it includes an utterance detection unit 88. Like information terminal device 10 shown in Fig. 4, information processing device 80A is a conceptual device realized by cooperation of either or both of terminal control device 32 and server control device 62. Therefore, utterance detection unit 88 included in information processing device 80A is also realized by either or both of terminal control device 32 and server control device 62.

[0097] The speech detection unit 88 detects speech of the user of the information terminal device 10A. Specifically, the speech detection unit 88 detects an input voice having a volume equal to or greater than a predetermined volume, or a speech detected by speech recognition. The speech detection unit 88 may be realized by the terminal control device 30A and the server control device 62 of the server device 12.

[0098] (Information processing method) An information processing method according to the sixth embodiment will be described with reference to Fig. 19. Fig. 19 is a flowchart showing the information processing method according to the sixth embodiment.

[0099] The processes of steps S140 and S142 shown in FIG. 19 are the same as the processes of steps S10 and S12 shown in FIG. 5, respectively, and therefore will not be described.

[0100] If it is determined as Yes in step S142, the speech detection unit 88 determines whether the user's speech is continuing (step S144). Specifically, the speech detection unit 88 determines that the user's speech is continuing when the input voice or speech is continuously detected. For example, if the input voice or speech is interrupted for 3 seconds or more, the speech detection unit 88 does not determine that the speech is not continuing and that the user's speech is not continuing. If it is not determined that the user's speech is continuing (step S144; No), the process proceeds to step S146. If it is determined that the user's speech is continuing (step S144; Yes), the process proceeds to step S156.

[0101] The processes from step S146 to step S156 shown in FIG. 19 are the same as the processes from step S14 to step S24 shown in FIG. 5, respectively, and therefore will not be described.

[0102] As described above, in the sixth embodiment, when it is determined that the user is continuing to speak, the input voice is not restricted. This makes it possible to prevent the input voice from being restricted while the user is speaking, resulting in the user's speech being interrupted.

[0103] Each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various additions and usage conditions. This distribution and integration configuration may be performed dynamically.

[0104] Although the embodiments of the present invention have been described above, the present invention is not limited to the contents of these embodiments. The above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]

[0105] 1. Information Processing Systems 10, 10A Information terminal device 12 Server equipment 20 Camera 22 Microphone 24 Control section 26,60 Communications Department 28 Display section 30, 30A Terminal control device 40 Shooting control section 44,84 Voice acquisition section 48 Operation control section 50,82 Application control unit 52,74 Storage part 54,86 Detection section 56,88 Speech detection unit 62 Server control device 70 Connection control section 72 Communication control section 80, 80A Information processing equipment

Claims

1. an application control unit that operates a web conference application using video and audio; A voice acquisition unit that acquires voices of users participating in the WEB conference; a detection unit that detects a direction of a user's line of sight or a face with respect to a display unit that displays the image; Equipped with The voice acquisition unit limits the voice to be acquired by the voice acquisition unit when the detection unit detects that the line of sight or the face direction of the user is directed in another direction away from the display screen of the display unit. Information processing device.

2. The detection unit further detects a predetermined action of the user, the voice acquisition unit, when the detection unit detects that the line of sight or the face direction of the user is directed in another direction from the display screen of the display unit and detects the predetermined action, limits the voice acquired by the voice acquisition unit. The information processing device according to claim 1 .

3. The detection unit further detects a predetermined voice of the user, the voice acquisition unit, when the detection unit detects that the line of sight or the face direction of the user is directed in another direction from the display screen of the display unit and detects the predetermined voice, limits the voice to be acquired by the voice acquisition unit. The information processing device according to claim 1 .

4. the voice capture unit, based on the detection result of the detection unit, when an operation time when the line of sight or the face direction of the user is directed in another direction from the display screen of the display unit is less than a predetermined time, limits the voice to be captured by the voice capture unit. The information processing device according to claim 1 .

5. The system further includes an utterance detection unit that detects an utterance of the user, When the speech detection unit determines that the user is not continuing to speak, the speech acquisition unit limits the speech acquired by the speech acquisition unit when the detection unit detects that the user's line of sight or face direction is directed in another direction away from the display screen of the display unit. The information processing device according to claim 1 .

6. acquiring voices of users participating in a WEB conference during operation of a WEB conference application using video and audio; detecting a direction of a user's line of sight or a face with respect to a display unit that displays the image; restricting acquisition of the user's voice when it is detected that the user's line of sight or face direction is directed away from the display screen of the display unit; The information processing method is executed by an information processing device.

Images