Information processing device, information processing method, and program

The information processing device addresses the issue of users being unaware of individuals outside the camera's field of view by detecting and notifying users through audio or visual cues, enhancing communication and privacy in video conferencing.

JP7593327B2Active Publication Date: 2024-12-03SONY GROUP CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2021554884
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-11-07
Filing Date
2020-10-23
Publication Date
2024-12-03
Estimated Expiration
2040-10-23

AI Technical Summary

Technical Problem

In video communication systems, users at remote locations may not be aware of the presence of individuals outside the camera's field of view, which can hinder effective communication, especially when some users are concerned about what is outside the viewing angle.

Method used

An information processing device and method that utilizes imaging and display devices in two separate spaces to detect users outside the camera's view and notify users in real-time through audio or visual cues, adjusting the output based on the user's attention level or the confidentiality of their actions.

Benefits of technology

Enhances user awareness of individuals outside the camera's field of view, particularly for those concerned, by providing appropriate notifications, thus improving communication quality and privacy in video conferencing scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007593327000001
    Figure 0007593327000001
  • Figure 0007593327000002
    Figure 0007593327000002
  • Figure 0007593327000003
    Figure 0007593327000003
Patent Text Reader

Abstract

The present technology pertains to an information processing device, an information processing method, and a program that enable a user, who sees a video of a remote location, to be aware of the presence of a person out of the field of view of the video in a more appropriate form. The information processing device according to the present technology is provided with a processing unit for performing a process for providing notification to a first user in a first space where a first imaging device and a first display device are disposed, about presence of a second user out of a field of view of a second imaging device when the second user has been detected at a position out of the field of view of the second imaging device in a case where, between the first space and a second space where the second imaging device and a second display device are disposed, a captured image captured by the imaging device in one space is displayed in real time by the display device in the other space. The present technology can be applied to, for example, a video communication system.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable a user watching video in a remote location to be aware of the presence of a person outside the field of view of the video in a more appropriate manner. [Background technology]

[0002] 2. Description of the Related Art Conventionally, video communication systems have been used to enable users in remote locations to talk to each other while looking at each other's faces, thereby enabling more intimate communication.

[0003] For example, Patent Document 1 discloses a video display system that enables users in remote locations to communicate with each other with a sense of realism as if they were actually face-to-face. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-192686 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, in the above-mentioned video display system, a user viewing a remote location via the display may not notice a person outside the angle of view of the camera used to capture the video of the remote location.

[0006] In particular, when there are multiple users viewing the remote location, it is preferable to make only some of the users who are concerned about the area outside the viewing angle aware that there is a person outside the viewing angle.

[0007] The present technology has been developed in consideration of such circumstances, and is intended to enable a user viewing video in a remote location to be made aware of the presence of people outside the field of view of the video in a more appropriate manner. [Means for solving the problem]

[0008] An information processing device according to one aspect of the present technology is an information processing device that, between a first space in which a first photographing device and a first display device are installed and a second space in which a second photographing device and a second display device are installed, displays an image photographed by a photographing device in one space in real time on a display device in the other space, and includes a processing unit that, when a second user is detected in a position outside the angle of view of the second photographing device, performs processing to notify a first user in the first space that the second user is outside the angle of view of the second photographing device.

[0009] An information processing method according to one aspect of the present technology is an information processing method in which, when an information processing device detects a second user located outside the angle of view of the second imaging device, the information processing device performs processing to notify a first user located in the first space that the second user is located outside the angle of view of the second imaging device when the information processing device displays an image captured by an imaging device in one space in real time on a display device in the other space between a first space in which a first imaging device and a first display device are installed, and the information processing device detects a second user located outside the angle of view of the second imaging device.

[0010] A program of one aspect of the present technology is a program for causing a computer to execute a process for notifying a first user in a first space that a first photographing device and a first display device are installed and a second space that a second photographing device and a second display device are installed, when an image photographed by a photographing device in one space is displayed in real time by a display device in the other space, when a second user is detected in a position outside the angle of view of the second photographing device, that the second user is outside the angle of view of the second photographing device.

[0011] In one aspect of the present technology, when an image captured by a first photographing device in one space is displayed in real time by a display device in a second space between a first space in which a first photographing device and a first display device are installed and a second space in which a second photographing device and a second display device are installed, when a second user is detected in a position outside the angle of view of the second photographing device, a process is performed to notify a first user in the first space that the second user is outside the angle of view of the second photographing device. [Brief description of the drawings]

[0012] [Figure 1] 1 is a diagram illustrating an example of the configuration of a video communication system according to an embodiment of the present technology. [Diagram 2] FIG. 1 is a front view showing an example of the external configuration of a telecommunication device. [Diagram 3] 1 illustrates an example of a situation in which a telecommunications device may be used; [Figure 4] FIG. 2 is a diagram showing an overview of a first output control of a telecommunications device. [Diagram 5] FIG. 13 is a diagram showing an example of a combination of a user's attention level and an output method at point A. [Figure 6] FIG. 1 is a block diagram showing a first configuration example of a telecommunication device. [Figure 7] 10 is a flowchart illustrating the operation of the telecommunication device. [Figure 8] FIG. 13 is a diagram showing an overview of a second output control of a telecommunications device. [Figure 9] FIG. 13 is a diagram showing an overview of a second output control of a telecommunications device. [Figure 10] FIG. 13 is a diagram showing an example of a combination of the situation at point B and an output method at point A. [Figure 11] FIG. 11 is a block diagram showing a second configuration example of a telecommunication device. [Figure 12] 10 is a flowchart illustrating the operation of the telecommunication device. [Figure 13] FIG. 13 is a diagram showing an overview of a third output control of a telecommunications device. [Figure 14] FIG. 13 is a diagram showing an overview of a third output control of a telecommunications device. [Figure 15] FIG. 13 is a diagram showing an overview of a fourth output control of a telecommunications device. [Figure 16] FIG. 13 is a block diagram showing a third exemplary configuration of a telecommunications device. [Figure 17] 10 is a flowchart illustrating the operation of the telecommunication device. [Figure 18] FIG. 13 is a diagram showing an overview of another output control performed by a telecommunication device. [Figure 19] FIG. 1 is a front view showing an example of a telecommunications device provided with a light emitter. [Figure 20] FIG. 13 is a diagram illustrating an example of the relationship between light intensity and attention level. [Figure 21] FIG. 1 is a diagram showing schematic examples of the configuration of a camera and a sensor. [Figure 22] FIG. 13 is a diagram illustrating another example of the configuration of a video communication system. [Figure 23] FIG. 13 is a diagram illustrating yet another example configuration of a video communication system. [Figure 24] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present technology will be described in the following order. 1. About video communication systems 2. First output control 3. Second output control 4. Third output control 5. Regarding the fourth output control 6. Variations

[0014] <1. About video communication systems> FIG. 1 is a diagram showing an example of the configuration of a video communication system according to an embodiment of the present technology.

[0015] As shown in FIG. 1, a video communication system 1 is configured by connecting a telecommunication device 11a and a telecommunication device 11b, which serve as two information processing devices, via a network 12 such as the Internet.

[0016] The telecommunication device 11a and the telecommunication device 11b are provided in different spaces, such as different buildings, different rooms, etc. A user near the telecommunication device 11a and a user near the telecommunication device 11b shown in Fig. 1 are mutually remote users.

[0017] The telecommunication device 11a and the telecommunication device 11b basically have the same configuration. As will be described in detail later, the telecommunication device 11a and the telecommunication device 11b each include a large-sized display, a camera for capturing images of the surroundings, a microphone for collecting ambient sounds, and a speaker for outputting sounds.

[0018] Between the telecommunication device 11a and the telecommunication device 11b, transmission and reception of images captured by their respective cameras, sounds collected by their respective microphones, and the like is performed constantly in real time, for example, while a connection between the two is established.

[0019] The telecommunication device 11a displays the video captured by the telecommunication device 11b, and outputs the sound collected by the telecommunication device 11b.

[0020] The video captured by the telecommunication device 11b shows the appearance of the space in which the telecommunication device 11b is installed, including the appearance of the user of the telecommunication device 11b. The sounds collected by the telecommunication device 11b include environmental sounds in the space in which the telecommunication device 11b is installed, including the voice of the user of the telecommunication device 11b.

[0021] This allows, for example, a user of telecommunications device 11a to feel as if a user of telecommunications device 11b is present opposite to the telecommunications device 11a located nearby.

[0022] Similarly, the telecommunication device 11b displays the image captured by the telecommunication device 11a, and outputs the sound collected by the telecommunication device 11a.

[0023] The video captured by the telecommunication device 11a shows the appearance of the space in which the telecommunication device 11a is installed, including the appearance of the user of the telecommunication device 11a. The sounds collected by the telecommunication device 11a include environmental sounds in the space in which the telecommunication device 11a is installed, including the voice of the user of the telecommunication device 11a.

[0024] This allows, for example, a user of telecommunications device 11b to feel as if the user of telecommunications device 11a is standing opposite to the telecommunications device 11b located nearby.

[0025] The user of the telecommunication device 11a can communicate with the user of the telecommunication device 11b in a natural manner as if the user of the telecommunication device 11b were in an adjacent space.

[0026] Similarly, the user of the telecommunication device 11b can communicate naturally with the user of the telecommunication device 11a as if they were in an adjacent space.

[0027] That is, the users of the telecommunication device 11a and the telecommunication device 11b can communicate more smoothly while feeling each other's close presence without being actively conscious of communicating with each other.

[0028] Hereinafter, when there is no need to distinguish between the telecommunications devices 11a and 11b, they will be collectively referred to as the telecommunications device 11. Other configurations provided in pairs will also be collectively described in the same manner.

[0029] FIG. 2 is a front view showing an example of the external configuration of the telecommunication device 11a.

[0030] As shown in FIG. 2, a vertically long rectangular display 22 made of an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display, or the like is provided on the front of the housing of the telecommunication device 11a, leaving a narrow frame 21 therebetween.

[0031] A sensor unit 23 is provided in front of the display 22 via, for example, a support member (not shown) fixed to the frame 21. The sensor unit 23 is provided with a camera 24 and two sensors, namely, sensors 25-1 and 25-2.

[0032] Furthermore, of the upper, lower, left and right edge portions constituting the frame 21, a microphone 26 is provided on the upper edge portion, and speakers 27-1 and 27-2 are provided on the left and right edge portions, respectively.

[0033] The display 22 displays an image corresponding to an image captured by the telecommunication device 11b, based on the image information transmitted from the telecommunication device 11b.

[0034] The camera 24 captures an image of the space in front of the telecommunication device 11a. Video information representing an image corresponding to the captured image captured by the camera 24 is transmitted to the telecommunication device 11b.

[0035] The sensors 25-1 and 25-2 are configured with various sensors such as a camera, an infrared sensor, etc. The sensors 25-1 and 25-2 generate environmental information based on the detection results of the sensors.

[0036] The environmental information is information including a video corresponding to an image captured by a camera constituting sensor 25, information acquired by an infrared sensor, etc. The video corresponding to an image captured by a camera constituting sensor 25 also shows a state outside the camera's angle of view, which is a range outside the shooting angle of view of camera 24.

[0037] The environmental information including the information outside the camera's angle of view is transmitted to the telecommunication device 11b. In the telecommunication device 11b, for example, a sound heard from outside the camera's angle of view (a sound whose source position is outside the camera's angle of view) is analyzed based on the environmental information.

[0038] The microphone 26 collects sounds in the space in which the telecommunication device 11a is installed. Sound information representing the sounds collected by the microphone 26 is transmitted to the telecommunication device 11b.

[0039] The speakers 27-1 and 27-2 output sounds of the space in which the telecommunication device 11b is installed, based on sound information transmitted from the telecommunication device 11b.

[0040] The telecommunication device 11b has the same configuration as the telecommunication device 11a shown in Fig. 2. In the following description, the configuration of the telecommunication device 11a shown in Fig. 2 will be appropriately used as the configuration of the telecommunication device 11b.

[0041] In addition, the installation positions of the camera 24, sensors 25-1, 25-2, microphone 26, and speakers 27-1, 27-2 in FIG. 2 are just an example, and they may be installed in other positions as long as their functions can be realized, and the number of installations is also arbitrary.

[0042] FIG. 3 shows an example of a situation in which the telecommunications device 11 may be used.

[0043] As shown in Fig. 3, for example, telecommunication devices 11a and 11b are installed at points A and B, which are separate spaces. The wavy line shown in the center of Fig. 3 indicates that the space where telecommunication device 11a is installed is separate from the space where telecommunication device 11b is installed.

[0044] In the example of Fig. 3, it is assumed that there are two users, users A1 and A2, in front of the telecommunication device 11a at point A. For example, user A1 is a user who is watching a video displayed on the telecommunication device 11a and talking with user B1 at point B. User A2 is a user who is looking toward the telecommunication device 11a from a position behind user A1.

[0045] On the other hand, it is assumed that there are two users, users B1 and B2, in front of the telecommunication device 11b at point B. For example, user B1 is a user who is watching the video displayed on the telecommunication device 11b and talking with user A1 at point A. User B2 is a user who is working outside the camera's angle of view.

[0046] Range #1, indicated by a dashed line as the range of point B, is the range of the camera angle of view of (the camera 24 installed in) the telecommunications device 11b. Ranges #2 and #3, indicated by dashed lines, are ranges outside the camera angle of view of (the camera 24 installed in) the telecommunications device 11b.

[0047] In this way, the telecommunication device 11a and the telecommunication device 11b may be used in a situation where there are multiple users around. The outputs of the telecommunication device 11a and the telecommunication device 11b are controlled according to the actions of the users around them.

[0048] <2. First output control> Output control overview FIG. 4 is a diagram showing an overview of the first output control of the telecommunication device 11. As shown in FIG.

[0049] In the example of Fig. 4, it is assumed that only user A2 is present in front of the telecommunication device 11a at point A. On the other hand, it is assumed that only user B2 is present at point B. User B2 is a user who is present in range #3, which is outside the camera's angle of view.

[0050] At this time, since there is no one within range #1, which is the range of the camera angle of view of the telecommunication device 11b, no one appears in the image displayed on the telecommunication device 11a. It is also assumed that user A2 in front of the telecommunication device 11a looks at the outside of the camera angle of view as shown in the balloon in Fig. 4, and peers in to see if there is anyone there.

[0051] In such a situation, when the telecommunication device 11a detects that the user A2 has performed an attention-seeking action, the telecommunication device 11a outputs an emphasized presence sound toward the user A2. The presence sound is a sound heard from outside the camera's field of view so as to make the user A2 sense the presence of a person outside the camera's field of view of the telecommunication device 11b.

[0052] The attention behavior is a behavior in which the user is concerned about the outside of the camera's angle of view. The behavior in which the user is concerned about the outside of the camera's angle of view includes an action of paying attention to either the left or right edge of the display 22 of the telecommunication device 11a, and an action of peering into the telecommunication device 11a as if concerned about the outside of the camera's angle of view.

[0053] The output of the sound presence is controlled according to the level of the user A2's attention to areas outside the camera's angle of view. The telecommunication device 11a analyzes the level of the attention level based on the attention behavior of the user A2, and controls the output method of the sound presence, etc., according to the level of the attention level.

[0054] FIG. 5 is a diagram showing an example of a combination of the attention level of the user A2 and the output method at the point A side.

[0055] In the example of Figure 5, the attention actions of user A2 are shown, in order of decreasing level of attention, as an action of staring at the edge of display 22 showing the image of point B, an action of peeking at the situation at point B, an action of checking whether there is anyone outside the field of view of the camera at point B by saying something like "Is anyone there?", and an action of turning one's face toward point B.

[0056] 5, when user A2 performs an attention action of gazing at the edge of display 22 displaying the image of point B, telecommunication device 11a adjusts the directivity of speakers 27-1 and 27-2 and outputs emphasized presence sounds toward user A2. At this time, the telecommunication device 11a adjusts the directivity of speakers 27-1 and 27-2 so as to face user A2, and also increases the volume of the presence sounds.

[0057] 5, when user A2 performs the attention behavior of peeking at the situation at point B, the telecommunication device 11a adjusts the directivity of the speakers 27-1 and 27-2 and outputs the presence sound toward user A2. At this time, the telecommunication device 11a performs a process of adjusting the directivity of the speakers 27-1 and 27-2 so as to face user A2.

[0058] As shown in the fourth row of the table in FIG. 5, when user A2 performs an attention action of checking whether there is a person outside the angle of view of the camera at point B by uttering something like “Is anyone there?”, the telecommunication device 11a outputs a sound presence in an output method according to whether there is a response to the utterance of user A2.

[0059] For example, when user B2 responds, the telecommunication device 11a performs output in a normal video communication system. For example, as output in a normal video communication system, the telecommunication device 11a performs processing to output environmental sounds collected at point B, instead of presence sounds heard from outside the camera's angle of view.

[0060] On the other hand, if there is no response from user B2, the telecommunications device 11a outputs an emphasized presence sound toward the entirety of point A. At this time, a process of increasing the volume of the presence sound is performed by the telecommunications device 11a.

[0061] As shown in the fifth row of the table in Figure 5, when user A2 performs the attention action of turning his / her face toward point B by looking at the display of telecommunication device 11a, telecommunication device 11a performs output in a normal video communication system.

[0062] In this way, the higher the attention level of the attention action of the user A2, the more emphasized the sound presence heard from outside the camera's angle of view is output to the user A2.

[0063] This enables the user A2 to notice, by the presence sound output from the telecommunication device 11a, that there is a person outside the camera angle of view of the connected telecommunication device 11b (the camera 24 installed in the telecommunication device 11b).

[0064] In addition, since a sound presence is output toward user A2 who has performed an attention action, the telecommunication device 11a can make only user A2, who is concerned about areas outside the angle of view of the connected camera, aware of the presence of a person outside the angle of view of the camera.

[0065] For example, the presence sound is not output to user A1 who is having a conversation with user B1 without worrying about people outside the camera's viewing angle of the connected camera, as in the case of user A1 in Fig. 3. User A1 can continue the conversation with user B1 without worrying about people outside the camera's viewing angle.

[0066] In this way, the presence sound is presented only to the user who is concerned about the area outside the camera's angle of view, making it possible to make the user more appropriately aware of the presence of a person outside the camera's angle of view.

[0067] · Telecommunications equipment 11 configuration FIG. 6 is a block diagram showing a first configuration example of the telecommunication device 11. As shown in FIG.

[0068] As shown in FIG. 6, the telecommunications device 11 comprises an input unit 31, a processing unit 32, a transmitting unit 33, a receiving unit , and an output unit .

[0069] 2, sensing devices such as sensors 25-1 and 25-2, and sound collection devices such as microphone 26. For example, input unit 31 supplies to processing unit 32 video information of a space corresponding to an image captured by camera 24, environmental information generated by sensors 25-1 and 25-2, and sound information of environmental sounds collected by microphone 26.

[0070] A microphone that collects sounds outside the camera's angle of view, a depth sensor that measures the distance to a subject photographed by the camera 24, and the like may be included as sensors 25-1 and 25-2 in the input unit 31. In this case, the environmental information also includes the detection results by the microphone and the depth sensor.

[0071] The processing unit 32 is composed of a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The processing unit 32 controls the overall operation of the telecommunication device 11 by executing a predetermined program.

[0072] As shown in FIG. 6, the processing unit 32 is made up of an attention level analysis unit 41, an environmental information analysis unit 42, a sound presence output method selection unit 43, and a sound presence control unit 44.

[0073] The attention level analysis unit 41 supplies the video information, the environmental information, and the sound information of the environmental sound supplied from the input unit 31 to the transmission unit 33 .

[0074] In addition, the attention analysis unit 41 analyzes the attention of a user in front of the telecommunication device 11 to areas outside the angle of view of the connected camera based on the video information, environmental information, and sound information of environmental sounds supplied from the input unit 31.

[0075] The attention level analysis unit 41 analyzes the level of attention of the user based on, for example, the position where the user in front of the telecommunication device 11 is gazing, the length of time the user directs his / her gaze at the display 22, the emitted voice, and the direction of the user's head. Various methods can be used to analyze the level of attention to areas outside the angle of view of the connected camera.

[0076] Attention level information, which is information indicating the user's attention level analyzed by the attention level analysis unit 41, is supplied to a presence sound output method selection unit 43.

[0077] The environmental information analysis unit 42 supplies the video information of the connection destination and the sound information of the environmental sound supplied from the receiving unit 34 to the output unit 35.

[0078] Furthermore, the environmental information analysis unit 42 analyzes sounds heard from outside the camera's angle of view and sounds heard from within the camera's angle of view, which are included in the environmental sounds of the destination, based on the environmental information of the destination supplied from the receiving unit 34. The sounds heard from outside the camera's angle of view are used as presence sounds to be appropriately presented to the user.

[0079] It should be noted that the environmental information analysis unit 42, which analyzes sounds heard from outside the camera's angle of view and sounds heard from within the camera's angle of view, also detects users who are present outside the angle of view of the connected camera.

[0080] Specifically, the environmental information analysis unit 42 performs bone estimation using a video showing a scene outside the viewing angle of the connected camera, thereby detecting a user who exists outside the viewing angle of the camera. Bone estimation is a technique for estimating a person's skeleton using a captured image. In addition, the user who exists outside the viewing angle of the camera may be detected by detecting background differences and recognizing faces using a video showing a scene outside the viewing angle of the connected camera.

[0081] Note that the video showing the state outside the viewing angle of the camera of the connection destination, which is used to detect a user who is outside the viewing angle of the camera, is information included in the environmental information of the connection destination.

[0082] Information representing the results of the analysis by the environmental information analysis unit 42 is supplied to the presence sound output method selection unit 43 together with sound information of the environmental sound of the connection destination.

[0083] The presence sound output method selection unit 43 selects a method for outputting a presence sound based on the information supplied from the environmental information analysis unit 42 and the attention level information supplied from the attention level analysis unit 41. For example, the presence sound output method selection unit 43 selects the directivity of the speakers 27-1, 27-2 that output the presence sound and the volume of the presence sound.

[0084] The sound information of the environmental sound of the connection destination, information indicating the result of the analysis by the environmental information analysis unit 42, and information indicating the method of outputting the presence sound are supplied to the presence sound control unit 44.

[0085] The presence sound control unit 44 generates sound information of a presence sound by processing sound information of the environmental sound of the connection destination based on information supplied from the presence sound output method selection unit 43. For example, the presence sound control unit 44 generates sound information of a presence sound by extracting sound information of a presence sound heard from outside the angle of view of the camera of the connection destination from sound information of the environmental sound of the connection destination.

[0086] The presence sound control unit 44 supplies sound information of the presence sound to the output unit 35, and causes the presence sound to be output by the output method selected by the presence sound output method selection unit 43.

[0087] The transmitting unit 33 transmits the video information, the environmental information, and the sound information of the environmental sound supplied from the processing unit 32 to the connected telecommunication device 11 via the network 12.

[0088] The receiving unit 34 receives video information, environmental information, and audio information of environmental sounds of the destination transmitted from the destination telecommunication device 11 via the network 12, and supplies these to an environmental information analysis unit .

[0089] The transmitter 33 and the receiver 34 are configured from a communication module compatible with wireless communication or wired communication such as a wireless LAN (Local Area Network) or cellular communication (e.g., LTE-Advanced or 5G).

[0090] 2, and sound output devices such as speakers 27-1 and 27-2. Video information of the connection destination and sound information of the environmental sound are supplied to the output unit 35 from the environmental information analysis unit 42. In addition, sound information of the presence sound is supplied to the output unit 35 from the presence sound control unit 44.

[0091] The output unit 35 displays an image of the space in which the destination telecommunication device 11 is installed on the display 22 based on the video information of the destination. Also, the output unit 35 outputs the environmental sound of the space in which the destination telecommunication device 11 is installed from the speakers 27-1 and 27-2 based on the sound information of the environmental sound of the destination. Furthermore, the output unit 35 outputs the presence sound from the speakers 27-1 and 27-2 according to the control of the presence sound control unit 44.

[0092] The configuration of the telecommunication device 11 shown in Fig. 6 is merely an example, and other configurations may be used. For example, a storage unit as an auxiliary storage device composed of a semiconductor memory, a hard disk drive (HDD), etc. may be provided. In the output unit 35, a display device such as a projector may be used instead of the display 22, and a sound output device such as earphones connected by wire or wirelessly may be used instead of the speakers 27-1, 27-2.

[0093] The sound presence control unit 44 may notify the user using other methods, not limited to sound presence. That is, in addition to sound presence (sound), other methods may be used, such as illuminating a light-emitting body such as an LED (Light Emitting Diode), displaying a GUI (Graphical User Interface) on the display 22, or vibrating a device carried by the user through wireless communication. A combination of multiple methods may also be used. In this way, the sound presence control unit 44, as a so-called output control unit, controls the output unit 35 having various output functions so that a notification is given to the user.

[0094] ·Operation of telecommunications equipment 11 Now, the operation of the telecommunication device 11 having the above-mentioned configuration will be described with reference to the flowchart of FIG.

[0095] The process described with reference to FIG. 7 is started, for example, when video information and audio information of environmental sounds are transmitted from the connected telecommunication device 11 via the network 12.

[0096] The attention analysis unit 41 performs processing for analyzing the user's attention to areas outside the angle of view of the connected camera as appropriate in parallel with the processing in Fig. 7. The video information, environmental information, and sound information of environmental sounds acquired by the input unit 31 are transmitted by the transmission unit 33 to the connected telecommunication device 11 via the network 12.

[0097] In step S1, the receiving unit 34 receives the video information, environmental information, and sound information of the environmental sound of the destination transmitted from the destination telecommunication device 11.

[0098] In step S2, the environmental information analysis unit 42 analyzes sounds heard from within the camera's angle of view and sounds heard from outside the camera's angle of view, which are included in the environmental sounds of the connection destination, based on the environmental information of the connection destination.

[0099] In step S3, the presence sound output method selection unit 43 selects a presence sound output method according to the attention level information analyzed by the attention level analysis unit 41.

[0100] In step S4, the presence sound control unit 44 generates sound information of the presence sound by processing sound information of the environmental sound of the connection destination. As described above, the generation of sound information of the presence sound is a process that is performed when there is a user who is performing an attention behavior of paying attention to the outside of the camera's angle of view.

[0101] In step S5, the presence sound control unit 44 causes the output unit 35 to output the presence sound by the output method selected by the presence sound output method selection unit 43 together with the environmental sound and video of the connected destination.

[0102] Through the above process, the telecommunication device 11 can make only a user who is concerned about areas outside the angle of view of the camera of the connected party aware of the presence of a person outside the angle of view of the camera of the connected party.

[0103] By focusing on an area outside the angle of view of the camera of the connected device, the user of the telecommunication device 11 can receive a notification of information indicating that there is a person outside the angle of view of the camera of the connected device.

[0104] <3. Second output control> Output control overview 8 and 9 are diagrams showing an overview of the second output control of the telecommunication device 11. In FIG.

[0105] In this example, the output of the presence sound is controlled according to the attention level of users outside the angle of view of the camera of the connected location, rather than the attention level of users watching the video of the connected location.

[0106] In the example of Fig. 8, as in the case described with reference to Fig. 4, it is assumed that user A2 is in front of telecommunication device 11a at point A. Meanwhile, it is assumed that user B2 is in range #3 at point B. As shown in the balloon in Fig. 8, user B2 is not paying attention to the video of point A.

[0107] In this way, when user B2, who is outside the camera's viewing angle, is not paying attention to the video at point A, the telecommunication device 11a performs output in a normal video communication system.

[0108] On the other hand, as shown in the balloon in FIG. 9, when the user B2 is paying attention to the video of the point A, the telecommunication device 11a outputs an emphasized sound presence toward the user A2 whom the user B2 is paying attention to.

[0109] FIG. 10 is a diagram showing an example of a combination of the situation at point B and the output method at point A.

[0110] In the example of Figure 10, the situations at point B are shown, in order of the attention level of user B2, as follows: a situation where user B2 is present outside the camera's angle of view and user B2 has a high attention level; a situation where user B2 is present outside the camera's angle of view and is attracting attention but the attention level of user B2 is below a predetermined level; a situation where user B2 is present outside the camera's angle of view but is not being paid attention to; and a situation where user B2 is not present outside the camera's angle of view.

[0111] As shown in the second row below the first row of the table in FIG. 10, when the situation at point B is such that user B2 is outside the camera's angle of view and is attracting a lot of attention, the telecommunication device 11a adjusts the directivity of the speakers 27-1 and 27-2 and outputs emphasized presence sounds toward user A2.

[0112] The telecommunication device 11a determines that the attention of user B2 is high when user B2 stares at user A2 near the telecommunication device 11b, when user B2 directs his / her gaze at user A2 for a predetermined period of time or more, when user B2 directs his / her gaze at user A2 a predetermined number of times or more, etc.

[0113] As shown in the third row of the table in FIG. 10, when the situation at point B is such that user B2 is outside the camera's angle of view and is paying attention to user A2 but the level of attention is less than a threshold, the telecommunication device 11a adjusts the directivity of the speakers 27-1 and 27-2 and outputs a sound of presence toward user A2.

[0114] As shown in the fourth row of the table in Figure 10, when the situation at point B is such that user B2 is outside the camera's field of view but is not paying attention to the image at point A, the telecommunication device 11a performs output as in a normal video communication system.

[0115] As shown in the fifth row of the table in FIG. 10, when the situation at point B is such that no user is present outside the camera's angle of view, the telecommunication device 11a performs output in a normal video communication system.

[0116] In this way, the greater the degree of attention of user B2, the more emphasized the presence sound heard from outside the camera's angle of view is output to user A2.

[0117] This allows user A2 to notice, by the presence sound output from telecommunication device 11a, that there is someone outside the camera angle of view of connected telecommunication device 11b (or the camera 24 installed on it) who is paying attention to user A2.

[0118] · Telecommunications equipment 11 configuration FIG. 11 is a block diagram showing a second configuration example of the telecommunication device 11. As shown in FIG.

[0119] In Fig. 11, the same components as those of the telecommunication device 11 in Fig. 6 are denoted by the same reference numerals. Duplicate explanations will be omitted as appropriate. The same applies to Fig. 16, which will be described later.

[0120] The configuration of the processing unit 32 shown in FIG. 11 differs from the configuration described with reference to FIG. 6 in that a connection destination attention level analysis unit 51 is provided after the environment information analysis unit .

[0121] The destination attention level analysis unit 51 is supplied with the destination video information, environmental information, and environmental sound information, as well as information indicating the results of analysis by the environmental information analysis unit 42, from the environmental information analysis unit 42.

[0122] The connection destination attention level analysis unit 51 supplies the video information of the connection destination and the sound information of the environmental sound supplied from the environmental information analysis unit 42 to the output unit 35 .

[0123] Further, the connection destination attention level analysis unit 51 analyzes the attention level of users who are outside the angle of view of the camera of the connection destination based on the environmental information of the connection destination supplied from the environmental information analysis unit .

[0124] The destination attention analysis unit 51 analyzes the level of attention of the destination user based on, for example, the position at which the destination user is gazing, the length of time that the user directs his / her gaze at the display 22, the number of times that the user directs his / her gaze at the display 22, and the direction of the destination user's head.

[0125] For example, the attention level of a user outside the viewing angle of the connected camera is analyzed by eye tracking based on information acquired by an infrared sensor of the connected camera. Also, the attention level of a user outside the viewing angle of the connected camera may be analyzed by a method of detecting the direction of the face of the connected user based on a video showing a scene outside the viewing angle of the connected camera.

[0126] Note that information acquired by an infrared sensor and images showing the situation outside the angle of view of the camera of the connected device, which are used to analyze the attention level of the connected user, are information included in the environmental information of the connected device.

[0127] The attention level information of the destination user, which is information representing the attention level of the destination user analyzed by the destination attention level analysis unit 51, is supplied to the presence sound output method selection unit 43 together with sound information of the environmental sound of the destination and information representing the result of the analysis by the environmental information analysis unit 42.

[0128] The presence sound output method selection unit 43 selects a method for outputting a presence sound based on the attention level information of the destination user supplied from the destination attention level analysis unit 51. For example, a threshold determination is made of the attention level of the destination user, and the presence sound output method is selected based on the determination result.

[0129] ·Operation of telecommunications equipment 11 Now, the operation of the telecommunications device 11 having the configuration shown in FIG. 11 will be described with reference to the flow chart shown in FIG.

[0130] The process described with reference to FIG. 12 also starts, for example, after video information and environmental sound information are transmitted from the connected telecommunication device 11 via the network 12.

[0131] The processing of the destination attention analysis unit 51 for analyzing the attention of users outside the viewing angle of the destination camera is appropriately performed in parallel with the processing of Fig. 12. The video information, environmental information, and sound information of environmental sounds acquired by the input unit 31 are transmitted by the transmission unit 33 to the destination telecommunication device 11 via the network 12.

[0132] The processes in steps S51 and S52 are similar to the processes in steps S1 and S2 in Fig. 7. That is, sounds heard from within the camera's field of view and sounds heard from outside the camera's field of view, which are included in the environmental sounds of the connection destination, are analyzed.

[0133] In step S53 , the presence sound output method selection unit 43 selects a method for outputting a presence sound in accordance with the attention level information of the connected user analyzed by the connected user attention level analysis unit 51 .

[0134] The processing in steps S54 and S55 is similar to the processing in steps S4 and S5 in Fig. 7. That is, sound information of a presence sound is generated, and the presence sound is output together with the environmental sound and image of the connection destination.

[0135] Through the above process, the telecommunication device 11 can make only a user who is paying attention to a person outside the angle of view of the connected camera aware of the presence of the person outside the angle of view.

[0136] <4. Third output control> 13 and 14 are diagrams showing an overview of the third output control of the telecommunication device 11. In FIG.

[0137] In this example, the output of a presence sound, which is a sound around a user who is outside the angle of view of the camera, is controlled according to the degree of attention of the user who is outside the angle of view of the connected camera.

[0138] In the example of Fig. 13, as in the case described with reference to Fig. 9, it is assumed that user A2 is in front of telecommunication device 11a at point A. Meanwhile, it is assumed that user B2 is in range #3 at point B. As shown in the balloon in Fig. 13, user B2 is paying attention to the video of point A.

[0139] In this way, when user B2, who is outside the angle of view of the camera of the connection destination, is paying attention to the video of the connection destination, the telecommunication device 11a outputs an emphasized presence sound toward user A2, who is the attention of user B2. In this example, a sound heard from within range #11, which is the range around user B2, is output as a presence sound.

[0140] As shown in FIG. 14, even when user B2, who is paying attention to user A2, is in range #1, the telecommunication device 11a outputs a sound heard from within range #11 as a presence sound.

[0141] In addition, in the third output control, the configuration and processing flow of the telecommunication device 11 are basically the same as those in the second output control, i.e., the configuration shown in Figure 11 and the processing flow shown in Figure 12, so their explanation is omitted.

[0142] As described above, the user A2 can become aware that a person who is paying attention to the user A2 himself / herself is outside or inside the camera's angle of view, by the sound presence output from the telecommunication device 11a.

[0143] <5. Regarding the fourth output control> Output control overview FIG. 15 is a diagram showing an outline of the fourth output control of the telecommunication device 11. In FIG.

[0144] In this example, sound output from telecommunication device 11b is controlled according to the secrecy of the actions of a user in the space in which telecommunication device 11a is installed.

[0145] In the example of Fig. 15, it is assumed that two users, users A11 and A12, are present in front of the telecommunication device 11a at point A. Meanwhile, it is assumed that only user B2 is present in range #3 at point B. As shown in the speech bubble of Fig. 15, users A11 and A12 are not aware that user B2 is present outside the angle of view of the camera at point B, and are engaged in a conversation about highly confidential matters such as personal matters.

[0146] In such a situation, when the telecommunication device 11b detects a highly confidential conversation between the users A11 and A12 at point A, the telecommunication device 11b processes the confidential sound collected at point A by degrading the sound quality, etc., based on the confidentiality of the conversation between the users A11 and A12, and outputs the processed confidential sound. The confidential sound is a sound such as a conversation that should be kept confidential, among sounds included in the environmental sound.

[0147] User B2 will hear voice with degraded sound quality, making it difficult to hear the content of the conversation.

[0148] This enables the telecommunication device 11b to output the voice of a highly confidential conversation between users in the vicinity of the connected telecommunication device 11a in a quality that is difficult for the user at point B to hear.

[0149] In the example of Figure 15, a case is shown in which the telecommunication device 11b detects a highly confidential conversation between users A11 and A12 at point A, but the same processing can be performed when the telecommunication device 11a detects a highly confidential conversation between multiple users at point B.

[0150] · Telecommunications equipment 11 configuration FIG. 16 is a block diagram showing a third example of the configuration of the telecommunication device 11. As shown in FIG.

[0151] The configuration of the processing unit 32 shown in FIG. 16 differs from the configuration described with reference to FIG. 6 in that a secrecy level analysis unit 71, a connection destination secrecy level acquisition unit 72, an encrypting sound output method selection unit 73, and an encrypting sound control unit 74 are provided.

[0152] The confidentiality analysis unit 71 receives the video information, the environmental information, and the sound information of the environmental sound from the input unit 31 .

[0153] The confidentiality analysis unit 71 analyzes the confidentiality of the voice of a user in the space where the telecommunication device 11 is installed, based on the environmental information and the sound information of the environmental sound supplied from the input unit 31 .

[0154] The confidentiality analysis unit 71 determines, for example, that the voice of a conversation about content that is not known to the public or the voice of a conversation about content that may hurt others is a voice with a high confidentiality level.

[0155] In addition, a learning model generated by machine learning using techniques such as neural networks and deep learning may be used to analyze the level of confidentiality.

[0156] The confidentiality information, which is information indicating the confidentiality of the user's voice analyzed by the confidentiality analysis unit 71, is supplied to the transmission unit 33 together with the video information, environmental information, and sound information of the environmental sound supplied from the input unit 31.

[0157] The transmission unit 33 transmits the video information, the environmental information, the sound information of the environmental sound, and the confidentiality information supplied from the confidentiality analysis unit 71 to the connected telecommunication device 11 via the network 12.

[0158] The receiving unit 34 receives the video information, environmental information, sound information of environmental sounds, and confidentiality information of the connected destination transmitted from the connected telecommunication device 11 via the network 12, and supplies them to the environmental information analysis unit 42 of the processing unit 32.

[0159] The environment information analysis unit 42 supplies information indicating the result of the analysis by the environment information analysis unit 42 to the connection destination secrecy level acquisition unit 72 together with the connection destination's video information, connection destination's sound information, and connection destination's secrecy level information.

[0160] The connection destination confidentiality level acquisition unit 72 supplies the video information of the connection destination and the sound information of the environmental sound supplied from the environmental information analysis unit 42 to the output unit 35 .

[0161] In addition, the connection destination confidentiality acquisition unit 72 supplies the connection destination confidentiality information, sound information of the environmental sound, and information representing the results of the analysis by the environmental information analysis unit 42 supplied from the environmental information analysis unit 42 to the confidential sound output method selection unit 73.

[0162] The encrypting sound output method selection unit 73 selects a method for outputting the encrypting sound based on the encrypting level information supplied from the connection destination encrypting level acquisition unit 72 and information representing the result of the analysis by the environmental information analysis unit 42. For example, the encrypting sound output method selection unit 73 selects a method of distorting the encrypting sound or a method of raising the level of only a specific frequency.

[0163] The information indicating the output method of the concealed sound is supplied to the concealed sound control unit 74 together with the sound information of the environmental sound of the connection destination and the concealment level information.

[0164] The concealed sound control unit 74 generates sound information of the concealed sound by processing sound information of the environmental sound of the connection destination based on the confidentiality information of the connection destination supplied from the concealed sound output method selection unit 73. For example, the concealed sound control unit 74 generates sound information of the concealed sound by extracting sound information of a conversation with a high confidentiality from the sound information of the environmental sound of the connection destination.

[0165] Moreover, the encrypting sound control unit 74 processes the encrypting sound based on the information indicating the output method of the encrypting sound supplied from the encrypting sound output method selection unit 73. The encrypting sound control unit 74 supplies sound information of the processed encrypting sound to the output unit 35, causing the output unit 35 to output the processed encrypting sound.

[0166] ·Operation of telecommunications equipment 11 Here, the operation of the telecommunication device 11 having the configuration shown in FIG. 16 will be described with reference to the flow chart shown in FIG.

[0167] The process described with reference to FIG. 17 is started, for example, when video information, sound information of environmental sounds, and confidentiality information are transmitted from the connected telecommunication device 11 via the network 12.

[0168] The video and environmental sound information and confidentiality information acquired by the input unit 31 are transmitted by the transmission unit 33 via the network 12 to the connected telecommunication device 11 .

[0169] In step S101, the receiving unit 34 receives the video information, confidentiality information, and sound information of the environmental sound of the destination transmitted from the destination telecommunication device 11.

[0170] In step S102, the environmental information analysis unit 42 and the connection destination secrecy acquisition unit 72 supply sound information of the environmental sound of the connection destination to the concealed sound output method selection unit 73 together with secrecy information and information representing the result of the analysis by the environmental information analysis unit 42.

[0171] In step S103, the encrypting sound output method selection unit 73 selects an output method of the encrypting sound based on the encrypting level information of the connection destination.

[0172] In step S104, the concealed sound control unit 74 processes the sound information of the environmental sound of the connection destination based on the confidentiality information of the connection destination to generate the concealed sound. The concealed sound control unit 74 processes the concealed sound based on the information indicating the output method of the concealed sound. As described above, the generation of the concealed sound and the processing of the concealed sound are processes performed when there is a user having a highly confidential conversation.

[0173] In step S105, the concealed sound control unit 74 causes the output unit 35 to output the processed concealed sound together with the environmental sound and video of the connected party.

[0174] By the above process, it is possible to prevent the audio of a highly confidential conversation from being presented to a user outside the angle of view of the camera of the connected telecommunication device 11.

[0175] In addition, sounds of the surroundings of users outside the angle of view of the connected camera may be output to a user who is engaged in highly confidential behavior, so as to make the user aware of the presence of users outside the angle of view of the connected camera.

[0176] FIG. 18 is a diagram showing an outline of another output control performed by the telecommunication device 11. In FIG.

[0177] In the example of Fig. 18, it is assumed that a user A11 is in front of the telecommunication device 11a at point A. Meanwhile, it is assumed that only a user B2 is present within range #3 at point B. The user A11 is a user who is performing highly confidential behavior such as personal behavior, without noticing the user B2 who is outside the camera's angle of view.

[0178] When it is detected that the user A11 is performing a highly confidential behavior, the telecommunication device 11a outputs an emphasized sound presence toward the user A11. The output of the sound presence outside the angle of view of the connected camera is controlled according to the level of confidentiality of the behavior of the user A11 at point A. It should be noted that the notification to the user A11 may be given by using other methods other than the sound presence.

[0179] Since the sound presence is output toward the user A11 who is performing highly confidential behavior, the telecommunication device 11 can make only the user A11 aware of the presence of a person outside the camera's angle of view.

[0180] For example, when a user A11 notices that there is a person outside the angle of view of the connected camera, the user A11 can stop engaging in highly confidential behavior.

[0181] In this way, by presenting a sound presence only to a user who is performing a highly confidential behavior, it is possible to make the user aware of the presence of a person outside the camera's angle of view in a more appropriate manner.

[0182] <6. Variations> About the light source The presence of a person outside the angle of view of the connected camera may be indicated by light emitted from a light emitter provided in the telecommunication device 11.

[0183] FIG. 19 is a front view showing an example of a telecommunication device 11a provided with a light emitter.

[0184] For example, light emitters are provided at the top, bottom, left and right edges of the display 22 of the telecommunication device 11a.

[0185] When user B2 is outside the angle of view of the camera of telecommunication device 11b, the image of user B2 is not displayed on display 22 of telecommunication device 11a, as shown in Fig. 19. In Fig. 19, user B2 is shown by a dashed line to the right of telecommunication device 11a, which indicates that user B2 is outside the angle of view of the camera at point B.

[0186] In this case, the telecommunication device 11a illuminates the light emitter provided on the right edge, which is the light emitter on the edge toward the direction in which the user B2 is located. In Fig. 19, the colored part on the right edge of the display 22 indicates that the light emitter is emitting light.

[0187] The light emission to indicate the presence of a person outside the camera's viewing angle is performed when there is only one user around the telecommunication device 11a, or when the presence of a person outside the camera's viewing angle is to be notified simultaneously to multiple users around the telecommunication device 11a. The intensity of the light from the light emitter may represent the attention level of user B2 to the image of point A.

[0188] FIG. 20 is a diagram showing an example of the relationship between light intensity and attention level.

[0189] In FIG. 20, the horizontal axis represents the attention level of user B2, and the vertical axis represents the light intensity.

[0190] As shown in FIG. 20, the telecommunication device 11a emits a stronger light as the attention of a user B2 who is outside the angle of view of the connected camera increases.

[0191] The user of the telecommunication device 11a can know the degree of attention of users outside the angle of view of the connected camera based on the intensity of light from the light emitter.

[0192] The attention level of a user outside the angle of view of the camera of the connection destination may be notified by vibrating a mobile terminal such as a smartphone carried by the user of the telecommunication device 11a. In this case, the presence sound control unit 44 of the telecommunication device 11a controls the mobile terminal carried by the user of the telecommunication device 11a to vibrate according to the attention level of user B2 outside the angle of view of the camera of the connection destination.

[0193] About Camera 24 and Sensor 25 In the above, the camera 24 and the sensor 25 are provided in the sensor unit 23, but the camera 24 and the sensor 25 may be provided in other forms.

[0194] FIG. 21 is a diagram showing a schematic example of the configuration of the camera 24 and the sensor 25. In FIG.

[0195] As shown in A of Fig. 21, it is possible to provide the camera 24 and the sensor 25 separately. In the example of A of Fig. 21, sensors 25-1 and 25-2 configured as cameras are provided on the left and right ends of the frame 21, respectively.

[0196] An image of range #51, which is the range of the angle of view of camera 24, is displayed on display 22 of connected telecommunication device 11. In addition, images of ranges #52-1 and #52-2 are used to detect conditions outside the display angle of view (outside the angle of view of camera 24).

[0197] Ranges #52-1 and #52-2 are the ranges of the angles of view of sensors 25-1 and 23-2, respectively, excluding the range that overlaps with range #51.

[0198] 21B, it is also possible to use the camera 24 itself as the sensor 25. For example, the camera 24 is configured as a wide-angle camera and is provided in the center of the telecommunications device 11.

[0199] An image of range #61, which represents a portion of the range within the angle of view of camera 24, is displayed on display 22 of connected telecommunication device 11. Also, an image of range #62 is used to detect conditions outside the display angle of view.

[0200] Range #62 is the range of the angle of view of camera 24 excluding range #61.

[0201] 21C, it is also possible to provide the sensor 25 outside the telecommunications device 11. For example, a camera constituting the sensor 25 is provided on a wall different from the wall on which the telecommunications device 11 is installed.

[0202] 21A, an image of range #51, which is the range of the angle of view of camera 24, is displayed on display 22 of connected telecommunication device 11. Also, an image of range #71 is used to detect conditions outside the display angle of view.

[0203] Range #71 is the range of the angle of view of sensor 25 excluding the area overlapping with range #51.

[0204] System configuration In FIG. 6 and other figures, the processing unit 32 is provided in the telecommunication device 11 , but all or part of the functions of the processing unit 32 may be realized in a server connected to the network 12 .

[0205] FIG. 22 is a diagram showing another example of the configuration of a video communication system.

[0206] The video communication system shown in Fig. 22 is configured by connecting a telecommunication device 11a, a telecommunication device 11b, and a server 101 via a network 12 such as the Internet. The server 101 is provided with all or part of the configuration of the processing unit 32 described with reference to Fig. 6, Fig. 11, or Fig. 16.

[0207] In this way, the processing unit 32 may be provided outside the telecommunications device 11 .

[0208] For example, various types of information such as video information, environmental information, and sound information of environmental sounds are transmitted from the telecommunication device 11 to the server 101 .

[0209] For example, various types of information such as video information, environmental information, and sound information of the connected device are transmitted from the server 101 to the telecommunication device 11 .

[0210] A home server for controlling the telecommunication device 11 may be provided for the telecommunication device 11 .

[0211] FIG. 23 is a diagram showing yet another example of the configuration of the video communication system.

[0212] The video communication system shown in FIG. 23 is configured by connecting a telecommunication device 11a and a telecommunication device 11b via a network 12.

[0213] The telecommunication device 11a is composed of an input / output unit 111a and a home server 112a. The input / output unit 111a is provided with a configuration of an input unit 31 and an output unit 35. The home server 112a is provided with a configuration of a processing unit 32, a transmitting unit 33, and a receiving unit 34. The home server 112a is connected to a home server 112b of the telecommunication device 11b via the network 12.

[0214] Similarly, the telecommunication device 11b is composed of an input / output unit 111b and a home server 112b. The input / output unit 111b is provided with a configuration of an input unit 31 and an output unit 35. The home server 112b is provided with a configuration of a processing unit 32, a transmitting unit 33, and a receiving unit 34. The home server 112b is connected to the home server 112a via the network 12.

[0215] In this way, components such as the processing unit 32 may be provided outside the input / output unit 111 having the display 22 and the like.

[0216] Note that a part of the configuration of the processing unit 32 may be provided in the home servers 112a and 112b, and the remaining part of the processing unit 32 may be provided in the input / output units 111a and 111b.

[0217] As described above, in this technology, when an image corresponding to an image captured by a camera 24 in one space between a space at point A where a telecommunication device 11a is installed and a space at a space at point B where a telecommunication device 11b is installed is displayed in real time on a display 22 in the other space, if a user is detected in a position outside the angle of view of the camera 24 installed at point B, processing is performed to notify a user at point A of the presence of the user.

[0218] This makes it possible for a user at point A who is watching video of point B to be made aware of the presence of a person outside the angle of view of the video of point B in a more appropriate manner.

[0219] In particular, in the present technology, a process is performed to notify a specific target user among users at point A of a presence sound outside the angle of view of camera 24 installed at point B that is included in the sound collected by microphone 26 installed at point B.

[0220] This makes it possible, when multiple users are at point A looking at the situation at point B, to make only some of the users who are concerned about what is outside the camera's angle of view aware that there is a person outside the camera's angle of view.

[0221] Computer Example Note that the processes described with reference to the above flowcharts do not necessarily have to be processed in chronological order according to the order described in the flowcharts, and include processes executed in parallel or individually (for example, parallel processing or object-based processing). Also, the programs may be processed by a single CPU, or may be processed in a distributed manner by multiple CPUs.

[0222] Furthermore, the series of processes by the above-mentioned telecommunication device 11 and the like can be executed by hardware or software. When the series of processes are executed by software, the programs constituting the software are installed from a program recording medium on which the programs are recorded, into a computer built into dedicated hardware, or into, for example, a general-purpose personal computer capable of executing various functions by installing various programs.

[0223] FIG. 24 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0224] In the computer, a CPU 201 , a ROM 202 , and a RAM 203 are connected to each other via a bus 204 .

[0225] An input / output interface 205 is further connected to the bus 204. Connected to the input / output interface 205 are an input unit 206 including a keyboard, a mouse, a microphone, etc., an output unit 207 including a display, a speaker, etc., a storage unit 208 including a hard disk or a non-volatile memory, etc., a communication unit 209 including a network interface, etc., and a drive 210 that drives removable media 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0226] In the computer configured as above, the CPU 201 loads, for example, a program stored in the storage unit 208 into the RAM 203 via the input / output interface 205 and the bus 204, and executes the program, thereby performing the series of processes described above.

[0227] The program executed by the computer (CPU 201) is provided, for example, by being recorded on removable media 211, which is a package medium consisting of a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor memory, or via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0228] The program can be installed in the storage unit 208 via the input / output interface 205 by mounting the removable medium 211 in the drive 210. The program can also be received by the communication unit 209 via a wired or wireless transmission medium and installed in the storage unit 208. Alternatively, the program can be installed in the ROM 202 or the storage unit 208 in advance.

[0229] ·others In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0230] The effects described herein are merely examples and are not limiting, and other effects may also be present.

[0231] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0232] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0233] <Examples of configuration combinations> The present technology can also be configured as follows. (1) In a first space in which a first photographing device and a first display device are installed, and a second space in which a second photographing device and a second display device are installed, an image photographed by the photographing device in one space is displayed in real time by a display device in the other space, a processing unit that, when detecting a second user in a position outside the angle of view of the second image capturing device, performs processing to notify a first user in the first space that the second user is outside the angle of view of the second image capturing device. Information processing device. (2) A process is performed to notify a first specific user, who is a specific target, of a first user in the first space of a presence sound that is included in a sound collected in the second space and is outside the angle of view of the second imaging device. The information processing device according to (1). (3) The processing unit includes: an output control unit that causes a first sound output device installed in the first space to output the sound collected in the second space in real time; an analysis unit that analyzes the presence sounds outside the angle of view of the second image capturing device among the sounds collected in the second space; having The output control unit causes the first sound output device to output the presence sound in real time based on an analysis result of the presence sound. The information processing device according to (2). (4) The processing unit further has an output method selection unit that selects an output method of the presence sound based on an analysis result of the presence sound, The output control unit causes the first sound output device to output the presence sound using the selected output method. The information processing device according to (3). (5) The processing unit further includes a first attention level analysis unit configured to analyze an attention level of a first attention user who is in the first space and focuses on an area outside an angle of view of the second image capture device, to the area outside an angle of view of the second image capture device; The output method selection unit selects an output method of the presence sound according to the attention degree of the first noted user. The information processing device according to (4). (6) The output control unit outputs the presence sound to the first target user among first users in the first space. The information processing device according to (5). (7) The first attention level analysis unit analyzes the attention level of the first noted user based on a position outside the angle of view of the second photographing device to which the first noted user is directed, a length of time during which the first noted user directs his / her gaze outside the angle of view of the second photographing device, a head direction of the first noted user, and a voice uttered by the first noted user. The information processing device according to (5) or (6). (8) the output method selection unit selects, as the output method of the presence sound, a method of adjusting a directivity of the presence sound and outputting the presence sound, The output control unit outputs the presence sound with adjusted directivity toward the first noted user. The information processing device according to (6). (9) The processing unit further includes a second attention analysis unit configured to analyze an attention of the second user, who is located outside an angle of view of the second image capturing device in the second space, to the first space; The output control unit selects a method for outputting the presence sound in accordance with the attention level of a second attention user who is paying attention to the first space. The information processing device according to (4). (10) The output control unit outputs the presence sound to a first noted user who is noted by the second noted user among first users in the first space. The information processing device according to (9). (11) The second attention level analysis unit analyzes the attention level of the second noted user based on a position in the first space where the second noted user is looking, a length of time during which the second noted user is looking at the first noted user, a number of times the second noted user is looking at the first noted user, and a direction of the head of the second noted user. The information processing device according to (10). (12) the output method selection unit selects, as the output method of the presence sound, a method of adjusting a directivity of the presence sound and outputting the presence sound, The output control unit outputs the presence sound with adjusted directivity toward the first attended user. The information processing device according to (10). (13) The output control unit causes the first sound output device to output a sound around the second user of interest as a presence sound. The information processing device according to any one of (9) to (12). (14) The output control unit causes a light-emitting body installed in the first space to emit light in accordance with the attention level of the second attention user. The information processing device according to (9). (15) The output control unit vibrates a device held by a first noted user who has been noted by the second noted user among first users in the first space, according to the attention level of the second noted user. The information processing device according to (9). (16) The processing unit includes: an output control unit that causes a first sound output device installed in the first space to output the sound collected in the second space in real time; a confidentiality level acquisition unit that acquires a confidentiality level of a behavior performed by a second user in the second space, The output control unit outputs a concealment sound to the first user in the first space according to the confidentiality level. The information processing device according to (1). (17) The processing unit includes: an output control unit that causes a first sound output device installed in the first space to output the sound collected in the second space in real time; a confidentiality analysis unit that analyzes confidentiality of an action performed by a first user in the first space; having The output control unit outputs a sound of presence to the first user in the first space according to the degree of secrecy when the second user is outside an angle of view of the second imaging device in the second space. The information processing device according to (1). (18) The first image capturing device and the first display device installed in the first space, or the second image capturing device and the second display device installed in the second space are integrally configured, and are mutually connected to an information processing device installed in the other space via a network. 18. The information processing device according to any one of (1) to (17). (19) An information processing device, In a first space in which a first photographing device and a first display device are installed, and a second space in which a second photographing device and a second display device are installed, an image photographed by the photographing device in one space is displayed in real time by a display device in the other space, When a sound outside the angle of view of the second image capturing device is detected from the sounds collected in the second space, a process is performed to notify a first user in the first space of the sound outside the angle of view. Information processing methods. (20) On the computer, In a first space in which a first photographing device and a first display device are installed, and a second space in which a second photographing device and a second display device are installed, an image photographed by the photographing device in one space is displayed in real time by a display device in the other space, A program for executing a process for notifying a first user in the first space of a sound outside the angle of view of the second imaging device when the sound outside the angle of view of the second imaging device is detected from the sounds collected in the second space. [Explanation of symbols]

[0234] 1 video communication system, 11 telecommunication device, 21 frame, 22 display, 23 sensor unit, 24 camera, 25-1, 25-2 sensor, 26 microphone, 27-1, 27-2 speaker, 31 input unit, 32 processing unit, 33 transmission unit, 34 reception unit, 35 output unit, 41 attention level analysis unit, 42 environmental information analysis unit, 43 presence sound output method selection unit, 44 presence sound control unit, 51 connection destination attention level analysis unit, 71 confidentiality level analysis unit, 72 connection destination confidentiality level acquisition unit, 73 confidential sound output method selection unit, 74 confidential sound control unit, 101 server, 111a, 111b home server, 112a, 112b input / output unit

Claims

1. In a first space in which a first image capturing device and a first display device are installed, and a second space in which a second image capturing device and a second display device are installed, an image captured by the image capturing device in one space is displayed in real time by a display device in the other space, a processing unit that, when detecting a second user in a position outside the angle of view of the second imaging device, performs processing to notify a first user in the first space that the second user is outside the angle of view of the second imaging device. Information processing device.

2. The processing unit performs processing for notifying a first specific user who is a specific target among first users in the first space of a presence sound outside an angle of view of the second imaging device, the presence sound being included in a sound collected in the second space. The information processing device according to claim 1 .

3. The processing unit includes: an output control unit that causes a first sound output device installed in the first space to output the sound collected in the second space in real time; an analysis unit that analyzes the presence sounds outside the angle of view of the second image capturing device among the sounds collected in the second space; having The output control unit causes the first sound output device to output the presence sound in real time based on an analysis result of the presence sound. The information processing device according to claim 2 .

4. The processing unit further has an output method selection unit that selects an output method of the presence sound based on an analysis result of the presence sound, The output control unit causes the first sound output device to output the presence sound using the selected output method. The information processing device according to claim 3 .

5. The processing unit further includes a first attention level analysis unit configured to analyze an attention level of a first attention user who is in the first space and focuses on an area outside an angle of view of the second image capture device, to the area outside an angle of view of the second image capture device; The output method selection unit selects an output method of the presence sound according to the attention degree of the first noted user. The information processing device according to claim 4.

6. The output control unit outputs the presence sound to the first target user among first users in the first space. The information processing device according to claim 5 .

7. The first attention level analysis unit analyzes the attention level of the first noted user based on a position outside the angle of view of the second imaging device to which the first noted user is directed, a length of time during which the first noted user directs his / her gaze outside the angle of view of the second imaging device, a head direction of the first noted user, and a voice uttered by the first noted user.

7. The information processing device according to claim 5 or 6.

8. the output method selection unit selects, as the output method of the presence sound, a method of adjusting a directivity of the presence sound and outputting the presence sound, The output control unit outputs the presence sound with adjusted directivity toward the first noted user. The information processing device according to claim 6.

9. The processing unit further includes a second attention level analysis unit configured to analyze an attention level of the second user who is in a position outside an angle of view of the second imaging device in the second space with respect to the first space, and the output control unit selects an output method of the sound of presence according to the attention level of the second user who is paying attention to the first space. The information processing device according to claim 4.

10. The output control unit outputs the presence sound to a first attended user who is attended by the second attended user among first users in the first space. The information processing device according to claim 9.

11. The second attention level analysis unit analyzes the attention level of the second noted user based on a position in the first space where the second noted user is looking, a length of time during which the second noted user is looking at the first noted user, a number of times the second noted user is looking at the first noted user, and a direction of the head of the second noted user. The information processing device according to claim 10.

12. the output method selection unit selects, as the output method of the presence sound, a method of adjusting a directivity of the presence sound and outputting the presence sound, The output control unit outputs the presence sound with adjusted directivity toward the first attended user. The information processing device according to claim 10.

13. The output control unit causes the first sound output device to output a sound around the second target user as a presence sound.

13. The information processing device according to claim 9.

14. The output control unit causes a light-emitting body installed in the first space to emit light in accordance with the attention level of the second attention user. The information processing device according to claim 9.

15. The output control unit vibrates a device held by a first noted user who has been noted by the second noted user among first users in the first space, according to the attention level of the second noted user. The information processing device according to claim 9.

16. The processing unit includes: an output control unit that causes a first sound output device installed in the first space to output the sound collected in the second space in real time; a confidentiality level acquisition unit that acquires a confidentiality level of a behavior performed by a second user in the second space, The output control unit outputs a concealment sound to the first user in the first space according to the confidentiality level. The information processing device according to claim 1 .

17. The processing unit includes: an output control unit that causes a first sound output device installed in the first space to output the sound collected in the second space in real time; a confidentiality analysis unit that analyzes a confidentiality of an action performed by a first user in the first space, The output control unit outputs a sound of presence to the first user in the first space according to the degree of secrecy when the second user is outside an angle of view of the second imaging device in the second space. The information processing device according to claim 1 .

18. The first image capturing device and the first display device installed in the first space, or the second image capturing device and the second display device installed in the second space are integrally configured, and are mutually connected to an information processing device installed in the other space via a network.

18. An information processing apparatus according to claim 1.

19. An information processing device, In a first space in which a first image capturing device and a first display device are installed, and a second space in which a second image capturing device and a second display device are installed, an image captured by the image capturing device in one space is displayed in real time by a display device in the other space, When a second user is detected in a position outside the angle of view of the second image capturing device, a process is performed to notify a first user in the first space that the second user is outside the angle of view of the second image capturing device. Information processing methods.

20. On the computer, In a first space in which a first image capturing device and a first display device are installed, and a second space in which a second image capturing device and a second display device are installed, an image captured by the image capturing device in one space is displayed in real time by a display device in the other space, A program for executing a process for notifying a first user in the first space that the second user is outside the field of view of the second photographing device when a second user is detected in a position outside the field of view of the second photographing device.

Citation Information

Patent Citations

  • User interface of image communication, terminal device

    JP1996251561A

  • Video display system and video display method

    JP2016192686A

  • Image display system and image display method

    WO2016159163A1