Information processing device and program

Abstract

To prevent sounds not to be transmitted to a communication partner from being transmitted thereto in sound transmission communication.SOLUTION: A processor controls sound collection according to the state of an upper body area of a user whose image is being captured in sound transmission communication.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an information processing device and a program. [Background technology]

[0002] Patent Document 1 describes a communication conference system in which receiving terminals communicate with each other via a communication network.

[0003] Patent Document 2 describes a video conference terminal device that is connected to a network and communicates with a remote terminal via images and audio.

[0004] Patent Document 3 describes a device that automatically sets microphone pickup in accordance with the placement of conference attendees. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2000-20683 A [Patent Document 2] JP 2011-66467 A [Patent Document 3] JP 2011-193392 A Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to prevent transmission of sounds that should not be transmitted to the other party during communication that transmits sounds. [Means for solving the problem]

[0007] The invention according to claim 1 has a processor, and the processor changes the control of the sound collection depending on the state of the upper body region of the user being photographed during communication that transmits sound. In this case, sounds other than the registered sounds will not be picked up. , an information processing device.

[0008] The invention according to claim 2 further comprises: The registered sound is the user's voice. 2. The information processing device according to claim 1,

[0018] Claim 3 The invention relates to A processor is included. The processor, In the case of communication that transmits sound, in the case where the control of sound collection is changed according to the state of the upper body region of the user being photographed, When multiple users are being photographed, if the mouths of all the users are not moving, no sound is picked up. Feelings It is an information processing device.

[0019] Claim 4 In the present invention, the processor When the mouth of a pre-designated user among the plurality of users is moving, the sound is collected. The present invention is characterized in that 3 to The information processing device described above is an

[0026] Claim 5 In the present invention, the computer changes the control of the sound collection depending on the state of the upper body area of ​​the user being photographed during communication that transmits sound. In this case, sounds other than the registered sounds will not be picked up. This is a program to make it work like this. The invention of claim 6 is a program for causing a computer to change sound collection control depending on the state of the upper body area of ​​a user being photographed during communication that transmits sound, and when multiple users are photographed, not to collect sound if the mouths of all of the multiple users are not moving. Effect of the Invention

[0027] Claims 1 and 2, 5 According to the present invention, during communication involving the transmission of sound, it is possible to prevent sound that should not be transmitted to the other party of the communication from being transmitted. It is also possible to prevent sounds other than the registered sounds from being transmitted to the other party of communication.

[0037] Claim 3,4,6 According to the invention, In communication, it is possible to prevent sounds that should not be transmitted to the other party from being transmitted. When the mouths of all the users are not moving, it is possible to prevent sounds that should not be conveyed to the other party of communication from being conveyed.

[0041] According to the invention of claim 18, by taking into consideration the movement of the mask worn by the user, it is possible to prevent sounds that should not be transmitted to the other party of communication from being transmitted.

[0042] According to the invention of claim 19, by taking into consideration the change in the image formed on the mask worn by the user, it is possible to prevent sounds that should not be transmitted to the other party of communication from being transmitted. [Brief description of the drawings]

[0043] [Figure 1] 1 is a block diagram showing a configuration of an information processing system according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a block diagram showing a configuration of a server according to the embodiment. [Diagram 3] FIG. 2 is a block diagram showing a configuration of a terminal device according to the present embodiment. [Figure 4] FIG. [Diagram 5] FIG. 2 is a diagram showing the positional relationship between a display and a user. [Figure 6] FIG. [Figure 7] FIG. [Figure 8] FIG. 2 is a diagram showing the positional relationship between a display and a user. [Figure 9] FIG. 1 is a diagram showing a user viewed from the front. [Figure 10] FIG. 2 is a diagram of a user viewed from above the head. [Figure 11] FIG. 2 is a diagram showing the positional relationship between a display and a user. [Figure 12] FIG. 13 is a diagram showing a user's line of sight. [Figure 13] FIG. 2 is a diagram showing a user's face as viewed from the front. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0044] An information processing system according to this embodiment will be described with reference to Fig. 1. Fig. 1 shows an example of the configuration of the information processing system according to this embodiment.

[0045] The information processing system according to the present embodiment includes, as an example, a server 10 and N terminal devices (N is an integer equal to or greater than 1). In the example shown in FIG. 1, the information processing system according to the present embodiment includes terminal devices 12A, 12B, 12C, . . . , 12N. The number of terminal devices shown in FIG. 1 is merely an example, and one or more terminal devices may be included in the information processing system according to the present embodiment. Hereinafter, when it is not necessary to distinguish between the terminal devices 12A, 12B, 12C, . . . , 12N, these will be referred to as "terminal devices 12". The information processing system according to the present embodiment may include devices other than the server 10 and the terminal devices 12.

[0046] The server 10 and each terminal device 12 have a function of communicating with other devices. The communication may be wired communication using a cable, or wireless communication. That is, each device may be physically connected to other devices via a cable to transmit and receive information, or may transmit and receive information wirelessly. Wireless communication is, for example, short-range wireless communication or Wi-Fi (registered trademark). Short-range wireless communication is, for example, Bluetooth (registered trademark), RFID (Radio Frequency Identifier), NFC, etc. For example, each device may communicate with other devices via a communication path N such as a LAN (Local Area Network) or the Internet. Also, each device may communicate with other devices via a telephone line.

[0047] In this embodiment, at least sound is transmitted to the other party of communication by communication. Also, images, videos, character strings, control data, etc. may be transmitted to the other party of communication together with sound. For example, sound may be transmitted to the other party via a communication path N, or sound may be transmitted to the other party via a telephone line. As one scene in which this embodiment is applied, at least sound is transmitted to the other party in an online conference held via the communication path N. As another scene, sound is transmitted to the other party via a telephone line. Of course, this embodiment may be applied to scenes other than these.

[0048] The server 10 transmits at least sound to the other party of communication. For example, the server 10 provides an online service via a communication path N. A user can use the online service by using a terminal device 12. For example, the user can transmit at least sound to the other party by using the online service. Note that a base station forming a telephone line is included in the information processing system, and the user may transmit sound to the other party via the telephone line by using the terminal device 12.

[0049] The online service is, for example, an online conference, a service that provides content online, an online game, an online shopping, a social network service (SNS), or a combination of these. An online conference may be called a Web conference, a remote conference, a video conference, or the like. By using an online conference, a user can transmit at least sound to other users. The content is, for example, entertainment (for example, concerts, plays, movies, videos, music, etc.), sports, e-sports, etc. For example, a video distribution service and a music distribution service are examples of services that provide content online. A user can watch entertainment online, or watch sports or e-sports.

[0050] The online service may be a service that uses a virtual space, or may be a service that does not use a virtual space. A virtual space is a concept that contrasts with a real space, and is, for example, a virtual space realized by a computer, a virtual space formed on a network such as the Internet, a virtual space realized by virtual reality (VR) technology, or cyberspace. For example, a virtual three-dimensional space or a two-dimensional space corresponds to an example of a virtual space.

[0051] The server 10 also stores and manages account information of users who use online services. The account information is information for logging in to an online service to use the online service, and includes, for example, a user ID and a password. For example, by transmitting the account information to the server 10 and logging in to the online service, a user associated with the account information is permitted to participate in the online service and use the online service. Of course, a user may use an online service without registering his / her account information with the online service.

[0052] The terminal device 12 is, for example, a personal computer (hereinafter referred to as a "PC"), a tablet PC, a smartphone, a wearable device (for example, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, hearable devices, etc.), a mobile phone, etc. A user can transmit sound to a communication partner using the terminal device 12.

[0053] In addition, an automatic response partner such as a chatbot may participate in the online service. For example, the automatic response partner functions as a response assistant that responds to a user's inquiry, receives a user's statement, analyzes the content of the statement, creates a response to the statement, and notifies the user. The automatic response partner is realized, for example, by executing a program. The program is stored, for example, in the server 10 or another device (for example, another server or the terminal device 12). The automatic response partner may be realized by artificial intelligence (AI). Any algorithm may be used for the artificial intelligence.

[0054] In the following, as an example, it is assumed that an online conference is used by a user and sound is transmitted to a communication partner through the online conference. The online conference is merely one example of transmitting sound to a communication partner, and sound collection, which will be described later, is controlled in examples other than the online conference in the same way as the online conference.

[0055] The hardware configuration of the server 10 will be described below with reference to Fig. 2. Fig. 2 shows an example of the hardware configuration of the server 10.

[0056] The server 10 includes, for example, a communication device 14, a UI 16, a memory 18, and a processor 20.

[0057] The communication device 14 is a communication interface having a communication chip, a communication circuit, etc., and has a function of transmitting information to other devices and a function of receiving information from other devices. The communication device 14 may have a wireless communication function or a wired communication function. The communication device 14 may communicate with other devices by using, for example, short-range wireless communication, or may communicate with other devices via a communication path N.

[0058] The UI 16 is a user interface and includes at least one of a display and an operation device. The display is a liquid crystal display or an EL display. The operation device is a keyboard, a mouse, an input key, an operation panel, or the like. The UI 16 may be a UI such as a touch panel that combines a display and an operation device.

[0059] The memory 18 is a device constituting one or more storage areas for storing various types of information. The memory 18 is, for example, a hard disk drive, various types of memory (e.g., RAM, DRAM, ROM, etc.), other storage devices (e.g., optical disks, etc.), or a combination thereof. One or more memories 18 are included in the server 10.

[0060] The processor 20 is configured to control the operation of each part of the server 10. The processor 20 may include a memory. For example, the processor 20 provides an online service to a user.

[0061] Hereinafter, the hardware configuration of the terminal device 12 will be described with reference to Fig. 3. Fig. 3 shows an example of the hardware configuration of the terminal device 12.

[0062] The terminal device 12 includes, for example, a communication device 22, a UI 24, a memory 26, and a processor 28.

[0063] The communication device 22 is a communication interface having a communication chip, a communication circuit, etc., and has a function of transmitting information to other devices and a function of receiving information transmitted from other devices. The communication device 22 may have a wireless communication function or a wired communication function. The communication device 22 may communicate with other devices by using, for example, short-range wireless communication, or may communicate with other devices via a communication path N.

[0064] The UI 24 is a user interface and includes at least one of a display and an operation device. The display is a liquid crystal display or an EL display. The operation device is a keyboard, a mouse, an input key, an operation panel, etc. The UI 24 may be a UI such as a touch panel that combines a display and an operation device.

[0065] The terminal device 12 may also include an imaging device such as a camera, a microphone, and a speaker, and all or some of these may be connected to the terminal device 12. Also, earphones or headphones may be connected to the terminal device 12.

[0066] The memory 26 is a device constituting one or more storage areas for storing various types of information. The memory 26 is, for example, a hard disk drive, various types of memory (e.g., RAM, DRAM, ROM, etc.), other storage devices (e.g., optical disks, etc.), or a combination thereof. One or more memories 26 are included in the terminal device 12.

[0067] The processor 28 is configured to control the operation of each part of the terminal device 12. The processor 28 may include a memory.

[0068] For example, the processor 28 displays images, videos, text, etc. transmitted via the online conference on the display of the terminal device 12, produces sounds transmitted via the online conference from a speaker, transmits images, videos, etc. captured by a camera to the other party via the online conference, and transmits sounds picked up by a microphone to the other party via the online conference.

[0069] The terminal device 12 may include at least one of various sensors such as a sensor (for example, a GPS (Global Positioning System)) that acquires location information of the terminal device 12, a gyro sensor that detects orientation and attitude, and an acceleration sensor.

[0070] Each example of the present embodiment will be described below. The processor 20 of the server 10 or the processor 28 of the terminal device 12 may execute the process according to each example, or the processor 20 and the processor 28 may cooperate to execute the process according to each example. A part of a process may be executed by the processor 20, and another part of the process may be executed by the processor 28. The server 10, the terminal device 12, or a combination of these corresponds to an example of an information processing device according to the present embodiment.

[0071] In the following, as an example, users A, B, C, and D participate in the same online conference. User A logs in to the online conference using terminal device 12A and participates in the online conference. User B logs in to the online conference using terminal device 12B and participates in the online conference. User C logs in to the online conference using terminal device 12C and participates in the online conference. User D logs in to the online conference using terminal device 12D and participates in the online conference. Note that multiple users may each log in to the online conference using the same terminal device 12 and participate in the online conference. It may be possible to participate in the online conference without logging in to the online conference.

[0072] An image or video of each user participating in the online conference is displayed on the display of each user's terminal device 12. A character string for identifying the user may be displayed on the display together with the image or video of the user, or without the image or video of the user. An authorized user or an administrator may permit or not permit each user to participate in the online conference. In this case, a user who is permitted to participate can participate in the online conference. When a user invites another user to the online conference and the other user indicates an intention to participate, the other user may be able to participate in the online conference.

[0073] 4 shows a screen 32 displayed on a display 30A. The display 30A is the display of the terminal device 12A of the user A. A screen similar to the screen 32 is also displayed on the displays of the terminal devices 12B, 12C, and 12D.

[0074] In the example shown in FIG. 4, the camera 34, the microphone 36, and the speaker 38 are provided on the display 30A. The camera 34 may be a camera called an in-camera or a front camera. The location where the camera 34, the microphone 36, and the speaker 38 are provided is not limited to the display 30A, and may be a location other than the display 30A in the terminal device 12A. All or a part of the camera 34, the microphone 36, and the speaker 38 may be provided on the terminal device 12A (for example, all or a part may be built into the terminal device 12A), or all or a part may not be provided on the terminal device 12A (for example, all or a part may not be built into the terminal device 12A), but may be provided around the terminal device 12A and connected to the terminal device 12A. A plurality of cameras 34, a plurality of microphones 36, and a plurality of speakers 38 may be connected to the terminal device 12A. The microphone may be worn by the user A, or earphones or headphones may be worn by the user A. When the terminal device 12A is a wearable device such as AR glasses or VR glasses, a microphone and a speaker may be provided in the wearable device.

[0075] For ease of explanation, the camera 34 provided in the terminal device 12A (i.e., a built-in camera) and the camera not provided in the terminal device 12A but connected to the terminal device 12A (i.e., an external camera) will be referred to as the camera of the terminal device 12A. In other words, the category of the concept of the camera of the terminal device 12A includes not only the camera 34 provided in the terminal device 12A, but also a camera not provided in the terminal device 12A but connected to the terminal device 12A. The same is true for the terminal devices 12B, 12C, and 12D, and the category of the concept of the camera of each of the terminal devices 12B, 12C, and 12D includes a built-in camera and an external camera.

[0076] Similarly, the microphone 36 provided in the terminal device 12A (i.e., a built-in microphone) and a microphone not provided in the terminal device 12A but connected to the terminal device 12A (i.e., an external microphone) are referred to as the microphone of the terminal device 12A. In other words, the category of the concept of the microphone of the terminal device 12A includes not only the microphone 36 provided in the terminal device 12A but also a microphone not provided in the terminal device 12A but connected to the terminal device 12A. The same is true for the terminal devices 12B, 12C, and 12D, and the category of the concept of the microphone of each of the terminal devices 12B, 12C, and 12D includes a built-in microphone and an external microphone.

[0077] Similarly, the speaker 38 provided in the terminal device 12A (i.e., a built-in speaker) and a speaker not provided in the terminal device 12A but connected to the terminal device 12A (i.e., an externally connected speaker) are referred to as the speaker of the terminal device 12A. In other words, the category of the concept of the speaker of the terminal device 12A includes not only the speaker 38 provided in the terminal device 12A but also a speaker not provided in the terminal device 12A but connected to the terminal device 12A. The same is true for the terminal devices 12B, 12C, and 12D, and the category of the concept of the speaker of each of the terminal devices 12B, 12C, and 12D includes a built-in speaker and an external speaker.

[0078] On the screen 32, images and videos representing each user participating in the online conference are displayed. Hereinafter, images and videos representing users are referred to as "user images". Note that user images of all users participating in the online conference may be displayed, or user images of some users may be displayed. The user image to be displayed may be specified by user A, or the user may specify whether to display or not display his / her own user image. The user image may be an image or video generated by photographing the user with a camera, or may be a schematic image (e.g., an icon or an avatar) representing the user. The user image may be a still image or a moving image. Data of the user image generated by photographing with a camera is transmitted from the terminal device 12 of the user to the terminal device 12 of the other user via the server 10, and the user image is displayed on the display of the terminal device 12 of each user. In addition, sounds such as the user's voice are collected by a microphone, and the data of the sound is transmitted from the terminal device 12 of the user to the terminal device 12 of the other user via the server 10, and the sound is emitted from the speaker of the terminal device 12 of each user. In addition, the user's remarks may be displayed as a character string. Each user can exchange information such as sounds, images, moving images, and character strings with other users through an online conference.

[0079] For example, display areas according to the number of users participating in an online conference are formed on the screen 32, and user images are displayed in each display area. In the example shown in Fig. 4, users A, B, C, and D participate in the same online conference using their own terminal devices 12, and display areas 32A, 32B, 32C, and 32D are formed on the screen 32. A user image of user A is displayed in the display area 32A, a user image of user B is displayed in the display area 32B, a user image of user C is displayed in the display area 32C, and a user image of user D is displayed in the display area 32D.

[0080] For example, an image or video generated by shooting with the camera of terminal device 12A is displayed in display area 32A. When user A is photographed with the camera of terminal device 12A, a user image representing user A is displayed in display area 32A. Similarly, an image or video generated by shooting with the camera of terminal device 12B is displayed in display area 32B, an image or video generated by shooting with the camera of terminal device 12C is displayed in display area 32C, and an image or video generated by shooting with the camera of terminal device 12D is displayed in display area 32D.

[0081] Even if user A is photographed by the camera of terminal device 12A, an icon, avatar, or the like that typically represents user A may be displayed in display area 32A instead of a user image generated by the photograph. The same applies to users B, C, and D.

[0082] Similarly to the screen 32, display areas 32A, 32B, 32C, and 32D are formed on the screens of the terminal devices 12B, 12C, and 12D, and images and videos captured by the cameras of the terminal devices 12 are displayed thereon.

[0083] The size of each display area may be changed by the user, or may be changed depending on whether each user is speaking. For example, the display area of ​​a user who is speaking may be larger, and the display area of ​​a user who is not speaking may be smaller. In addition, the screen 32 may display information for identifying each user participating in the online conference (for example, account information, name, ID, etc.).

[0084] When a sound is picked up by the microphone of the terminal device 12A, the picked up sound is emitted from the speakers 38 of each of the terminal devices 12B, 12C, and 12D. For example, when user A speaks, the voice of user A is picked up by the microphone of the terminal device 12A and emitted from the speakers of each of the terminal devices 12B, 12C, and 12D. Similarly, when a sound is picked up by the microphone of each of the terminal devices 12B, 12C, and 12D, the picked up sound is emitted from the speakers of the other terminal devices 12.

[0085] The processor 28 of the terminal device 12A changes the control of the sound collection by the microphone of the terminal device 12A according to the state of the upper body region of the photographed user A. Similar processing is performed in the terminal devices 12 of the other users. That is, the processor 28 of the terminal device 12B changes the control of the sound collection by the microphone of the terminal device 12B according to the state of the upper body region of the photographed user B, the processor 28 of the terminal device 12C changes the control of the sound collection by the microphone of the terminal device 12C according to the state of the upper body region of the photographed user C, and the processor 28 of the terminal device 12D changes the control of the sound collection by the microphone of the terminal device 12D according to the state of the upper body region of the photographed user D.

[0086] All of the terminal devices 12A, 12B, 12C, and 12D may change the control of sound collection depending on the state of the upper body region of the user being photographed, or some of the terminal devices 12A, 12B, 12C, and 12D may change the control of sound collection depending on the state of the upper body region of the user being photographed.

[0087] The user may set whether or not to execute a function that changes the control of sound collection according to the state of the upper body region of the user. For example, when the user A turns on the function, the processor 28 of the terminal device 12A changes the control of sound collection according to the state of the upper body region of the user A being photographed. When the user A turns off the function, the processor 28 of the terminal device 12A does not change the control of sound collection according to the state of the upper body region of the user A being photographed. The same applies to the other terminal devices 12.

[0088] The processor 20 of the server 10 may control the collection of sound by the microphone of each terminal device 12. In the following, it is assumed that the processor of the terminal device 12 controls the collection of sound by the microphone of that terminal device 12.

[0089] The upper body region of the user is, for example, the region above the user's shoulders, the region above the user's neck, or the region of the user's face (hereinafter referred to as the "face region"). The state of the user's upper body region is, for example, the orientation, inclination, shape, movement, line of sight, or a combination thereof of the upper body region of the user. The face region of the user is, for example, the user's face itself without a mask or other object, or the user's face with a mask or other object attached.

[0090] The processor 28 of the terminal device 12A detects the state of the upper body region of the user A based on, for example, an image showing the upper body region of the user A (for example, a user image generated by photographing the user A with the camera of the terminal device 12A), detection results of various sensors (for example, a gyro sensor, an acceleration sensor, etc.), or a combination thereof. The same applies to the other terminal devices 12.

[0091] Specifically, the processor 28 of the terminal device 12A may detect the direction, inclination, and shape of the face region of the user A based on an image representing the face region of the user A, or may detect the direction, inclination, shape, movement, and gaze of the region above the neck of the user A based on an image representing the region above the neck of the user A, or may detect the direction, inclination, movement, and gaze of the face region and the region above the neck of the user by various sensors (e.g., gyro sensor, acceleration sensor, etc.) worn by the user A. For example, a hearable device having various sensors is worn on the user's ear, or AR glasses or VR glasses having various sensors are worn by the user. The direction, inclination, and movement of the user's face are detected by various sensors provided in the hearable device, AR glasses, or VR glasses. The same applies to other terminal devices 12.

[0092] In the following, the process according to the embodiment will be described with a focus on the terminal device 12A of the user A, but similar processes are also performed in the terminal devices 12B, 12C, and 12D.

[0093] The processor 28 of the terminal device 12A collects or does not collect sound depending on the state of the upper body region of the user A.

[0094] For example, when the orientation of the upper body region of user A corresponds to a specific orientation, when the inclination of the upper body region of user A corresponds to a specific inclination, when the shape of the upper body region of user A corresponds to a specific shape, or when the movement of the upper body region of user A corresponds to a specific movement, the processor 28 of the terminal device 12A collects sound with the microphone of the terminal device 12A. Specifically, the processor 28 of the terminal device 12A turns on the microphone of the terminal device 12A to collect sound. The sound collected by the microphone of the terminal device 12A is emitted from the speakers of each of the terminal devices 12B, 12C, and 12D. For example, when the user A speaks, the voice of the user A is emitted from the speakers of each of the terminal devices 12B, 12C, and 12D.

[0095] When the orientation of the upper body region of user A does not correspond to a specific orientation, when the inclination of the upper body region of user A does not correspond to a specific inclination, when the shape of the upper body region of user A does not correspond to a specific shape, or when the movement of the upper body region of user A does not correspond to a specific movement, the processor 28 of the terminal device 12A does not pick up sound with the microphone of the terminal device 12A. In this case, even if user A speaks, the voice of user A is not emitted from the speakers of each of the terminal devices 12B, 12C, and 12D.

[0096] The concept of not picking up sound includes turning off the microphone of the terminal device 12A, muting the sound picked up by the microphone of the terminal device 12A, and cutting (e.g. deleting) the sound picked up by the microphone.

[0097] Similarly, the processors 28 of the terminal devices 12B, 12C, and 12D collect or do not collect sounds depending on the state of the upper body regions of the users B, C, and D, respectively.

[0098] Instead of not collecting sound, the processor 28 of the terminal device 12 may collect sound but not output the sound data. For example, the processor 28 of the terminal device 12 may not transmit the sound data to the other terminal device 12. As another example, the processor 28 of the terminal device 12 may transmit the sound data to the server 10, but the server 10 may not transmit the sound data to the other terminal device 12. As yet another example, even if the sound data is sent to the other terminal device 12, the processor 28 of the other terminal device 12 may not generate the sound from a speaker. These aspects may also be aspects of not collecting sound.

[0099] A specific example will be described below with reference to Fig. 5. Fig. 5 is a diagram showing the positional relationship between the display 30A of the terminal device 12A and the user A. Fig. 5 shows the display 30A and the user A when viewed from above the head 40A of the user A. In this specific example, as an example, the state of the upper body region of the user A is the orientation of the face region of the user A, more specifically, the orientation of the face of the user A.

[0100] For example, when the front of the face of the user A is photographed by the camera of the terminal device 12A, the processor 28 of the terminal device 12A collects sound by the microphone of the terminal device 12A, and does not collect sound when the front of the face of the user is not photographed by the camera of the terminal device 12A. In the example shown in FIG. 5, when the front of the face of the user A is photographed by the camera 34 provided on the display 30A, the processor 28 of the terminal device 12A collects sound by the microphone of the terminal device 12A, and does not collect sound when the front of the face of the user A is not photographed by the camera 34. For example, when the user A is looking at the screen 32 of the display 30A, the front of the face of the user A is photographed by the camera 34. In this case, the processor 28 of the terminal device 12A collects sound by the microphone of the terminal device 12A.

[0101] For example, the face of user A is photographed by the camera of terminal device 12A, and processor 28 of terminal device 12A analyzes the user image generated by the photographing to determine whether the front of user A's face is photographed by the camera of terminal device 12. That is, processor 28 of terminal device 12A determines whether the front of user A's face is shown in the user image. For example, a known technique is used as a technique for analyzing an image. When the front of user A's face is shown in the user image, processor 28 of terminal device 12A collects sound with the microphone of terminal device 12A, and when the front of user A's face is not shown in the user image, processor 28 does not collect sound.

[0102] A detailed description will be given with reference to Fig. 5. In Fig. 5, reference symbol 42F indicates the direction in front of user A, reference symbol 42R indicates the direction to the right of user A, and reference symbol 42F indicates the direction to the left of user A. The direction indicated by reference symbol 42F is a direction perpendicular to screen 32, and the directions indicated by each of reference symbols 42R and 42F are directions parallel to screen 32.

[0103] When user A is looking at the screen 32, the orientation of user A coincides with the orientation indicated by reference symbol 42F, and the front of the face of user A is photographed by camera 34. In this case, processor 28 of terminal device 12A collects sound with the microphone of terminal device 12A.

[0104] The direction indicated by the reference symbol 42F may be defined as a reference direction, a direction inclined at an angle θ1 to the right or left from the reference direction may be defined, and a range of 2×angle θ1 may be defined as a range of the front. When the orientation of the face of the user A is included in the range of the front, the processor 28 of the terminal device 12A may collect sound with the microphone of the terminal device 12A. In other words, even if the face of the user A is not photographed directly in front of the camera of the terminal device 12A and the face of the user A is photographed by the camera of the terminal device 12A at an angle θ1 from the front, the processor 28 of the terminal device 12A may collect sound with the microphone of the terminal device 12A.

[0105] 6 and 7 show a user image 44 of user A. In Fig. 6 and 7, a display area 32A, which is a part of the screen 32, is shown, and the illustration of the part other than the display area 32A is omitted. The user image 44 may be a still image or a moving image.

[0106] The user image 44 shown in Fig. 6 is a user image generated when the front of the face of user A is photographed by the camera of the terminal device 12A. When the front of the face of user A is photographed by the camera of the terminal device 12A, the front of the face of user A is displayed in the display area 32A as shown in Fig. 6. In this case, the processor 28 of the terminal device 12A picks up sound by the microphone of the terminal device 12A. For example, when user A speaks, the sound is picked up by the microphone of the terminal device 12A, and the voice of user A is emitted from the speakers of each of the terminal devices 12B, 12C, and 12D.

[0107] The user image 44 shown in Fig. 7 is a user image generated when the left side of the user's face is photographed by the camera of the terminal device 12A. When the user A looks in the direction indicated by the reference symbol 42R in Fig. 5 (i.e., when the user A looks to the right), the left side of the user A's face is photographed by the camera of the terminal device 12A, and the left side of the user A's face is displayed in the display area 32A as shown in Fig. 7. In this case, the processor 28 of the terminal device 12A does not pick up any sound. Similarly, when the user A looks to the left, the processor 28 of the terminal device 12A does not pick up any sound.

[0108] The control of sound collection may be changed by focusing on the area of ​​the user's mouth. For example, when the front of the mouth of the user A is photographed by the camera of the terminal device 12A, the processor 28 of the terminal device 12A collects sound with the microphone of the terminal device 12A, and does not collect sound when the front of the mouth of the user A is not photographed by the camera of the terminal device 12A.

[0109] The processor 28 of the terminal device 12A detects the orientation of the face of the user A by various sensors (for example, a gyro sensor, an acceleration sensor, etc.), and when the face of the user A is facing the screen 32, the microphone of the terminal device 12A collects sound, but does not need to collect sound when the face of the user A is not facing the screen 32. For example, the terminal device 12A is a wearable device (for example, a hearable device, AR glasses, VR glasses, etc.) having various sensors, the terminal device 12A is worn by the user A, and the orientation of the face of the user A is detected by the various sensors.

[0110] If the front of the face of user A is photographed (for example, when user A is looking at the screen 32), it is assumed that user A is in conversation with other users participating in the online conference or is about to converse with other users. Therefore, the processor 28 of the terminal device 12A picks up sound with the microphone of the terminal device 12A so that if user A speaks, the remark can be picked up.

[0111] When the front of the face of user A is not photographed (for example, when user A is not looking at the screen 32), if user A makes a statement, the statement may not be addressed to each user participating in the online conference. For example, an event may occur in the vicinity of where user A is, and user A may make a statement in response to the event. Specifically, user A may be called, the intercom may ring, or the phone may ring, and user A may make a statement in response to these. For example, when user A is participating in an online conference at his / her home, it is conceivable that a family member of user A at the home may call out to user A, saying, "User A, it's XXX." In this case, it is conceivable that user A may turn in the direction where user A's family is (for example, turn behind user A) and say, "Wait a moment." User A's statement is not addressed to other users participating in the online conference, and is a statement that should not be conveyed to other users. In this case, since the processor 28 of the terminal device 12A does not collect sound, the speech of user A is not collected, and it is possible to prevent the speech of user A from being transmitted to other users participating in the online conference. In this way, when the front of the face of user A is not photographed, it is possible to prevent the speech of user A that should not be transmitted to other users participating in the online conference from being transmitted to the other users. This makes it possible to prevent, for example, information that should not be transmitted from being transmitted to other users, and to prevent the information from being leaked. In other words, it is possible to improve the security of information.

[0112] The processor 28 of the terminal device 12A may collect sound by the microphone of the terminal device 12A when the front of the face of the user A is photographed and the mouth of the user A is moving. For example, the processor 28 of the terminal device 12A determines whether the mouth of the user A is moving by analyzing a user image representing the user A. When the front of the face of the user A is photographed and the mouth of the user A is moving, it is presumed that the user A is speaking to other users participating in the online conference. In this case, the processor 28 of the terminal device 12A collects sound by the microphone of the terminal device 12A, and can convey the speech of the user A to other users participating in the online conference.

[0113] The processor 28 of the terminal device 12A may not pick up the sound if the user A's mouth is not moving, even if the front of the user A's face is photographed. If the user A's mouth is not moving, it is assumed that the user A is not speaking. In this case, by not picking up the sound, it is possible to prevent sounds other than the user A's speech from being transmitted to other users participating in the online conference.

[0114] When sound is picked up by the microphone of the terminal device 12A, if output of the picked up sound is permitted by user A, the processor 28 of the terminal device 12A may send data of the picked up sound to the terminal devices 12B, 12C, and 12D of other users participating in the online conference. In this case, the sound picked up by the microphone of the terminal device 12A is emitted from the speakers of the terminal devices 12B, 12C, and 12D.

[0115] For example, the processor 28 of the terminal device 12A holds data of sounds collected for each predetermined period (for example, every minute), and when the output of the sound is permitted by the user A, transmits data of the sounds collected during the period when the output is permitted to the terminal devices 12B, 12C, and 12D of the other users participating in the online conference. As a result, the sounds collected during the period when the output is permitted are emitted from the speakers of the terminal devices 12B, 12C, and 12D with a delay from the time when the sounds were collected. For example, the processor 28 of the terminal device 12A requests permission to output from the user for each period (for example, every minute), and transmits data of the sounds collected during the period when the permission was granted to the terminal devices 12B, 12C, and 12D of the other users.

[0116] Even if sound is picked up by the microphone of the terminal device 12A, if output of the picked up sound is not permitted, the processor 28 of the terminal device 12A does not send data of the picked up sound to the terminal devices 12B, 12C, and 12D of other users participating in the online conference.

[0117] For example, a button for instructing permission to output sound may be displayed on the display 30A of the terminal device 12A, and sound output may be permitted when the user A presses the button. Alternatively, the user A may permit sound output by voice.

[0118] For example, user A can confirm the picked-up sound and then permit or disallow the output of the sound. If the picked-up sound is a sound that should be transmitted to other users participating in the online conference, the sound is transmitted to the other users by permitting the output of the sound. If the picked-up sound is a sound that should not be transmitted to other users participating in the online conference, the sound can be prevented from being transmitted to other users by not permitting the output of the sound.

[0119] The processor 28 of the terminal device 12A may not pick up sounds other than the registered sounds. For example, the voice of the user A is registered in the terminal device 12A, and the processor 28 of the terminal device 12A picks up the voice of the user A by the microphone of the terminal device 12A, and does not pick up sounds other than the voice of the user A. For example, the processor 28 of the terminal device 12A recognizes the voice of the registered user by using a voice recognition technology, picks up the voice of the registered user, and does not pick up the voice of an unregistered user. The timing of registering the voice may be before the online conference starts, may be during the online conference, or may be specified by the user A. The processor 28 of the terminal device 12A may remove sounds other than the registered sounds by a voice filter function.

[0120] The processor 28 of the terminal device 12A may collect the voices of users up to the number of registered users. For example, the number of users is registered in the terminal device 12A, and the processor 28 of the terminal device 12A collects the voices of users up to the number of registered users by the microphone of the terminal device 12A, and does not collect the voices of users exceeding the number of registered users. The timing of registering the number of users may be before the online conference starts, may be during the online conference, or may be specified by the user.

[0121] For example, if three people are registered in the terminal device 12A, the processor 28 of the terminal device 12A collects the voices of the three users and does not collect the voices of the fourth user and subsequent users. In more detail, the processor 28 of the terminal device 12A detects the voice of the user A1 collected first by the microphone of the terminal device 12 as the voice of the first user, detects the voice of the user A2 collected next as the voice of the second user, and detects the voice of the user A3 collected next as the voice of the third user. The processor 28 of the terminal device 12A identifies the voices of the users A1, A2, and A3, respectively, and manages the voices of the users A1, A2, and A3 separately. After that, the processor 28 of the terminal device 12A collects the voices of the users A1, A2, and A3, and does not collect the voices of users other than the users A1, A2, and A3. For example, even if user A4 speaks, user A4 is identified as the fourth user, and the processor 28 of the terminal device 12A does not pick up the voice of user A4. Note that the voices of users A1, A2, and A3 may be registered in the terminal device 12A, and the processor 28 of the terminal device 12A may pick up the voices of the registered users A1, A2, and A3, but may not pick up the voices of users other than users A1, A2, and A3.

[0122] In the case where multiple users are photographed, if the mouths of all the multiple users are not moving, the processor 28 of the terminal device 12A does not need to collect sound. This process will be described with reference to Fig. 8. Fig. 8 is a diagram showing the positional relationship between the display 30A and the users A1, A2, and A3. Fig. 8 shows the display 30A and the users A1, A2, and A3 when viewed from above the heads 40A1, 40A2, and 40A3 of the users A1, A2, and A3, respectively.

[0123] 8, users A1, A2, and A3 are present within a shooting range 46 of a camera 34 provided on the display 30A, and the faces of the users A1, A2, and A3 are photographed by the camera 34. The user images of the users A1, A2, and A3 are displayed in a display area 32A of the screen 32.

[0124] 8, when users A1, A2, and A3 are photographed, if the mouths of all users A1, A2, and A3 are not moving, the processor 28 of the terminal device 12A does not collect any sound. If the mouths of all users A1, A2, and A3 are not moving, it is assumed that none of users A1, A2, and A3 is speaking, and therefore the processor 28 of the terminal device 12A does not collect any sound.

[0125] When the mouth of at least one of the users A1, A2, and A3 is moving, the processor 28 of the terminal device 12A picks up sound with the microphone of the terminal device 12A.

[0126] When the mouth of a specific user among the users A1, A2, and A3 is moving, the processor 28 of the terminal device 12A may pick up the sound by the microphone of the terminal device 12A. The specific user is, for example, a user who is logged in to the online conference on behalf of the users A1, A2, and A3, or a user designated in advance.

[0127] The processor 28 of the terminal device 12A may control the collection of sound by the microphone of the terminal device 12A based on the habit (i.e., habits) of the movement of the upper body region of the user A. The habit of the movement of the upper body region is, for example, a habit of moving the neck or face, or a habit due to an illness. The movement habit of the upper body region of the user A is registered in the terminal device 12A. The timing of the registration may be before the start of the online conference, may be during the online conference, or may be specified by the user A.

[0128] For example, in the case where the front of the face of the user A is photographed and the sound is collected by the microphone of the terminal device 12A, even if the front of the face of the user A is no longer photographed due to the habit of moving the neck or face of the user A, the processor 28 of the terminal device 12A collects the sound by the microphone of the terminal device 12A. The processor 28 of the terminal device 12A may detect the habit of moving the neck or face of the user A based on a user image generated by photographing the user A by the camera of the terminal device 12A, or may detect the habit by various sensors (e.g., a gyro sensor, an acceleration sensor, etc.). For example, a wearable device (e.g., a hearable device, an AR glass, a VR glass, etc.) having various sensors is worn by the user A, and the habit of the user A is detected by the various sensors. When the habit is detected, the processor 28 of the terminal device 12A collects the sound by the microphone of the terminal device 12A even if the front of the face of the user A is no longer photographed. That is, even if the front of the face of the user A is no longer captured, the processor 28 of the terminal device 12A continues to collect sound without turning off the microphone of the terminal device 12A.

[0129] Habit-based control will be described in more detail with reference to Figures 9 and 10. Figure 9 is a diagram of user A as viewed from the front. Figure 10 is a diagram of user A as viewed from above the head.

[0130] 9 and 10 show heads 48, 50 of user A. It is assumed that user A has a habit of moving his / her neck (for example, a habit of twisting his / her neck). When user A twists his / her neck, the user's head tilts and the front direction of user A's face changes. Head 48 is the head of user A in a state where the user A is not twisting his / her neck due to habit. Head 50 is the head of user A in a state where the user A has twisted his / her neck due to habit.

[0131] 9, when user A twists his / her neck, the head of user A tilts laterally by angle θ2. That is, head 50 tilts laterally from the position of head 48 by angle θ2.

[0132] FIG. 10 shows the direction of the face of user A. Direction 48A is the direction of the face of user A before twisting his / her neck. Direction 50A is the direction of the face of user A after twisting his / her neck. As shown in FIG. 10, the direction of the face of user A changes after twisting his / her neck due to habit, and as a result, the front of the face of user A may not be captured. Even in this case, if the action of twisting the neck of user A is registered as a habit of user A, processor 28 of terminal device 12A collects the sound with the microphone of terminal device 12A. For example, when user A speaks while twisting his / her neck, even if the front of the face of user A is no longer captured, processor 28 of terminal device 12A collects the sound with the microphone of terminal device 12A. As a result, the voice of user A is collected by the microphone of terminal device 12A and transmitted to users B, C, and D participating in the online conference.

[0133] Also, an allowable range of a habitual motion may be determined. When the motion of user A is within the allowable range, the processor 28 of the terminal device 12A may collect sound with the microphone of the terminal device 12A, and when the motion of user A is outside the allowable range, the processor 28 of the terminal device 12A may not collect sound. For example, when a motion of twisting the neck is registered as a habit of user A, an allowable range of the angle of rotation of the user A's neck is determined, and when the angle of rotation of the user A's neck is within the allowable range, the processor 28 of the terminal device 12A collects sound with the microphone of the terminal device 12A.

[0134] The state of the user's face area may be the user's line of sight. For example, the processor 28 of the terminal device 12A changes the control of the sound collection according to the line of sight of the user A toward a display (for example, the display of the terminal device 12A) used in the online conference. Specifically, the processor 28 of the terminal device 12A collects the sound with the microphone of the terminal device 12A when the user A is looking at the display (i.e., when the display is in the line of sight of the user A), and does not collect the sound when the user A is not looking at the display (i.e., when the display is not in the line of sight of the user A). The processor 28 of the terminal device 12A collects the sound with the microphone of the terminal device 12A when the user A is speaking toward the display (i.e., when the user A is speaking while looking at the display), and does not need to collect the sound when the user A is not speaking toward the display (i.e., when the user A is speaking without looking at the display).

[0135] An embodiment in which sound collection is controlled in accordance with the line of sight of a user will be described below with reference to Fig. 11. Fig. 11 is a diagram showing the positional relationship between a display and a user.

[0136] 11 shows user A and multiple displays (for example, six displays). As an example, it is assumed here that user A is participating in an online conference using six displays. The number of displays is merely an example, and one display may be used, or multiple displays other than six may be used.

[0137] For example, displays 30CU and 30CL are provided in the center, displays 30LU and 30LL are provided on the left side, and displays 30RU and 30RL are provided on the right side. Displays 30CU, 30LU, and 30RU are provided on the upper side, and displays 30CL, 30LL, and 30RL are provided on the lower side. The arrangement of the displays is merely an example, and the position of each display may be changed by the user. User images of other users participating in the online conference are displayed on each display.

[0138] Information indicating the position of each display is stored in the memory 26 of the terminal device 12A, and the position of each display is registered in the terminal device 12A. For example, a screen for setting the position of each display is displayed on the display of the terminal device 12A, and the user A specifies the display to be used on the screen and sets the position of each display. In the example shown in FIG. 11, the displays 30CU, 30CL, 30LU, 30LL, 30RU, and 30RL are specified as the displays to be used in the online conference, and the positions of each display are set. That is, the position of the display 30CU is specified to be the upper center position, and the position of the display 30CL is specified to be the lower center position, which is registered in the terminal device 12A. The same applies to the other displays.

[0139] For example, the processor 28 of the terminal device 12A may detect the line of sight 52 of the user A based on a user image generated by photographing the user A with the camera of the terminal device 12A, or may detect the line of sight 52 of the user A using various sensors (e.g., a gyro sensor, an acceleration sensor, etc.).

[0140] The processor 28 of the terminal device 12A may detect the position of the user A by photographing the user A with the camera of the terminal device 12A, may detect the position of the user based on a user image generated by the photographing, may detect the position of the user A by using a GPS, or may determine a predetermined position (for example, a position facing the central displays 30CU, 30CL) as the position of the user A. For example, the processor 28 of the terminal device 12A photographs the user A with one or more cameras installed at predetermined positions, and identifies the position of the user A based on the results of the photographing (for example, the results of which camera photographed the user A, how the user A appears, etc.).

[0141] The processor 28 of the terminal device 12A determines whether or not the user A is looking at the display based on the position of the user A, the line of sight 52 of the user A, and the position of each display. When a display is present in the line of sight 52 of the user A based on the position of the user A, the processor 28 of the terminal device 12A determines that the user A is looking at the display. When a display is not present in the line of sight 52 of the user A based on the position of the user A, the processor 28 of the terminal device 12A determines that the user A is not looking at the display.

[0142] Based on the above-mentioned determination result, the processor 28 of the terminal device 12A picks up the sound with the microphone of the terminal device 12A when user A is looking at the display and speaking, and does not pick up the sound when user A is not looking at the display. In this way, when user A is looking at the display and speaking, the voice of user A is transmitted to users B, C, and D participating in the online conference. On the other hand, when user A is not looking at the display, even if user A speaks, the voice of user A is not transmitted to users B, C, and D participating in the online conference.

[0143] When user A is speaking while looking at the display, it is presumed that user A is speaking to other users participating in the online conference. In this case, by picking up the sound with the microphone of the terminal device 12A, the voice of user A can be transmitted to other users. On the other hand, when user A is speaking without looking at the display, it is presumed that the speech is not addressed to other users participating in the online conference. In this case, by not picking up the sound with the microphone of the terminal device 12A, it is possible to prevent sounds that should not be transmitted to other users from being transmitted to other users.

[0144] It should be noted that if the display provided in front of the line of sight 52 of the user A is not set as the display to be used for the online conference, the processor 28 of the terminal device 12A does not collect any sound.

[0145] The processor 28 of the terminal device 12A may change the control of the sound collection according to the angle of the line of sight of the user A. For example, when the line of sight of the user A is lowered, the processor 28 of the terminal device 12A may or may not collect the sound with the microphone of the terminal device 12A. Whether or not to collect the sound may be set by the user.

[0146] For example, the processor 28 of the terminal device 12A picks up sound using the microphone of the terminal device 12A when the line of sight of the user A is within a predetermined range, and does not pick up sound when the line of sight of the user A is not within the predetermined range.

[0147] More specifically, processor 28 of terminal device 12A defines the horizontal (i.e., the direction perpendicular to the direction of gravity) as the reference direction, and if the angle of user A's line of sight is within a predetermined angle range from the reference direction, sound is picked up by the microphone of terminal device 12A, and if the angle of user A's line of sight is not within that angle range, sound is not picked up.

[0148] Hereinafter, an embodiment in which sound collection is controlled according to the line of sight of a user will be described with reference to Fig. 12. Fig. 12 is a diagram showing the line of sight of a user A. Fig. 12 shows a schematic diagram of a user A, a display 30A, and a keyboard 54A. The display 30A and the keyboard 54A are included in a terminal device 12A.

[0149] For example, when user A is operating the keyboard 54A (for example, pressing a key) or looking at an image displayed on the display 30A, the line of sight of user A drops. In the example shown in FIG. 12, the line of sight 56 ​​of user A drops below the horizontal line 58. In this case, the processor 28 of the terminal device 12A picks up sound using the microphone of the terminal device 12A. For example, when user A is speaking while typing characters on the keyboard 54A, the voice of user A is picked up and transmitted to other users participating in the online conference.

[0150] The processor 28 of the terminal device 12A picks up sound using the microphone of the terminal device 12A when the line of sight 56 ​​of the user A is within a range of angle θ3 based on the horizontal 58 (a range below the horizontal 58), but does not have to pick up sound when the line of sight 56 ​​of the user A is not within that range.

[0151] As another example, the processor 28 of the terminal device 12A may change the control of sound collection depending on whether or not the keyboard 54A is operated. That is, the processor 28 of the terminal device 12A may detect the operation of the keyboard 54A and change the control of sound collection depending on the detection result. For example, the processor 28 of the terminal device 12A may collect sound with the microphone of the terminal device 12A when the keyboard 54A is operated (for example, when the user A is typing using the keyboard 54A), and may not collect sound when the keyboard 54A is not operated.

[0152] In the example shown in FIG. 12, an example in which the keyboard 54A is operated has been described, but a mouse, a touch panel, or the like may be used as an operating device other than the keyboard 54A.

[0153] In addition, when multiple users are photographed by the camera of the terminal device 12A, even if at least one of the multiple users lowers his / her gaze, the processor 28 of the terminal device 12A may not collect the sound. In this case, when at least one user speaks, the processor 28 of the terminal device 12A may collect the sound with the microphone of the terminal device 12A. When the user who spoke is registered in the terminal device 12A, the processor 28 of the terminal device 12A may collect the sound with the microphone of the terminal device 12A, and when the user who spoke is not registered in the terminal device 12A, the processor 28 of the terminal device 12A may not collect the sound.

[0154] The state of the user's face area may be the movement of a mask worn by the user. For example, the processor 28 of the terminal device 12A changes the control of the sound collection according to the movement of the mask worn by the user A. Specifically, the processor 28 of the terminal device 12A collects sound with the microphone of the terminal device 12A when the mask is moving (for example, when there is a change in the shape of the mask), and does not collect sound when the mask is not moving (for example, when there is no change in the shape of the mask).

[0155] If the mask is moving, it is assumed that the mouth of user A is moving. In other words, it is assumed that the mask is moving because user A is speaking. In this case, by collecting audio, when user A speaks, the voice of user A is transmitted to other users participating in the online conference.

[0156] If the mask is not moving, it is assumed that User A's mouth is not moving. In other words, it is assumed that User A is not speaking. In this case, no sound is picked up. This makes it possible to prevent sounds that should not be transmitted to other users participating in the online conference (for example, sounds other than User A's voice) from being transmitted to other users.

[0157] Hereinafter, an embodiment in which sound collection is controlled according to the movement of the mask will be described with reference to Fig. 13. Fig. 13 is a diagram of the face of user A as viewed from the front. Fig. 13 shows the head 40A and mask 60 of user A. User A is wearing the mask 60, and the mouth and surrounding area of ​​user A are covered by the mask 60.

[0158] For example, the processor 28 of the terminal device 12A detects a change in the shape of the surface of the mask 60. When the processor 28 of the terminal device 12A detects a change in the shape of the surface of the mask 60, the processor 28 of the terminal device 12A collects sound with the microphone of the terminal device 12A, and when the processor 28 of the terminal device 12A does not detect a change in the shape of the surface of the mask 60, the processor 28 of the terminal device 12A does not collect sound. For example, when unevenness 62 is formed on the surface of the mask 60, if the shape of the unevenness 62 changes, the processor 28 of the terminal device 12A collects sound with the microphone of the terminal device 12A. When the shape of the unevenness 62 does not change, the processor 28 of the terminal device 12A does not collect sound.

[0159] The processor 28 of the terminal device 12A may collect sound with the microphone of the terminal device 12A when the amount (e.g., the magnitude of change) of the shape of the surface of the mask 60 (e.g., the change in the shape of the unevenness 62) is equal to or greater than a predetermined threshold, and may not collect sound when the amount of change is less than the threshold. The threshold may be changed by the user.

[0160] The processor 28 of the terminal device 12A may change the control of the sound collection in response to a change in the shadow formed on the surface of the mask 60. In other words, the processor 28 of the terminal device 12A may collect sound with the microphone of the terminal device 12A when the shadow formed on the surface of the mask 60 has changed or when the amount of the change is equal to or greater than a threshold, and may not collect sound when the shadow has not changed or when the amount of the change is less than the threshold.

[0161] An image 64 may be formed on the surface of the mask 60. In this case, the state of the face area of ​​the user A may be a change in the image 64. For example, the image 64 may be printed on the surface of the mask 60, or the image 64 may be expressed by the fabric of the mask 60.

[0162] For example, the processor 28 of the terminal device 12A picks up sound with the microphone of the terminal device 12A when the image 64 changes, and does not pick up sound when the image 64 does not change. When the image 64 changes, it is presumed that the mouth of user A is moving. In other words, it is presumed that user A is speaking. In this case, by picking up sound, when user A speaks, the voice of user A can be transmitted to other users participating in the online conference. When the image 64 does not change, it is presumed that the mouth of user A is not moving. In other words, it is presumed that user A is not speaking. In this case, by not picking up sound, it is possible to prevent sounds that should not be transmitted to other users (for example, sounds other than the voice of user A) from being transmitted to other users.

[0163] The processor 28 of the terminal device 12A may pick up sound with the microphone of the terminal device 12A when the amount of change in the image 64 is equal to or greater than a predetermined threshold, and may not pick up sound when the amount of change is less than the threshold.

[0164] For example, the processor 28 of the terminal device 12A detects changes in the shape of the surface of the mask 60, changes in the shadow formed on the surface of the mask 60, and changes in the image 64 by analyzing a user image generated by photographing the face of the user A with a camera.

[0165] If sound is picked up by the microphone of the terminal device 12A even though the shape of the surface of the mask 60 and the image 64 have not changed, the processor 28 of the terminal device 12A may cut (for example, delete) the picked up sound. In this case, the picked up sound is not transmitted to other users participating in the online conference.

[0166] In the above-mentioned embodiment, various processes are described by taking an online conference as an example, but the processes according to the present embodiment may be applied to situations other than online conferences. For example, when a user distributes a video using the terminal device 12, if the face of the user is photographed, the video may be distributed, and if the face of the user is not photographed, the video may not be distributed, for example, by stopping the video distribution. For example, the video distribution may be stopped when the user turns away, and may be distributed when the user faces forward. After the video distribution is stopped, if the user presses a button (for example, a button displayed on the display) for instructing the start of distribution, the video distribution may be resumed.

[0167] The functions of each part of the server 10 and the terminal device 12 are realized, for example, by cooperation between hardware and software. For example, the processor of each device reads and executes a program stored in the memory of each device, thereby realizing the function of each device. The program is stored in the memory via a recording medium such as a CD or DVD, or via a communication path such as a network.

[0168] In each of the above embodiments, the term "processor" refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.). Furthermore, the operation of the processor in each of the above embodiments may not only be performed by one processor, but may also be performed by multiple processors located at physically separate locations working together. Furthermore, the order of each operation of the processor is not limited to the order described in each of the above embodiments, and may be changed as appropriate. [Explanation of symbols]

[0169] 10 servers, 12 terminals, 20,28 processors.

Claims

1. A processor is included. The processor, When communication is performed to transmit sound, when the control of sound collection is changed according to the state of the upper body region of the user being photographed, sounds other than the registered sounds are not collected. Information processing device.

2. The processor, The registered sound is the user's voice.

2. The information processing apparatus according to claim 1,

3. A processor, The processor, In a case where sound collection control is changed depending on the state of the upper body region of a user being photographed during communication that transmits sound, when multiple users are photographed, sound is not collected if the mouths of all of the multiple users are not moving. Information processing device.

4. The processor, When the mouth of a pre-designated user among the plurality of users is moving, the sound is collected.

4. The information processing apparatus according to claim 3.

5. The computer When communication is performed to transmit sound, when the control of sound collection is changed according to the state of the upper body region of the user being photographed, sounds other than the registered sounds are not collected. A program to make it work like this.

6. The computer In a case where sound collection control is changed depending on the state of the upper body region of a user being photographed during communication that transmits sound, when multiple users are photographed, sound is not collected if the mouths of all of the multiple users are not moving. A program to make it work like this.

Images