Remote communication system and remote communication method

Abstract

A remote communication system (100) according to the present disclosure comprises: a terminal device (2) that has a display unit (22) and a camera (23) provided to the display unit (22) and that can be operated by a second user who performs remote communication with a first user; an augmented monitor (3), which has a display surface, and on which the terminal device (2) is disposed on a display surface side such that the display unit (22) overlaps the display surface as viewed from the second user; and a video processing device (4) that uses a captured video of the first user to generate a first video, which is part of the video and is displayed on the terminal device (2), and a second video, which is at least part of the video. The first video, when it is assumed that the video is to be displayed on the augmented monitor (3), is a partial video that is displayed in an area corresponding to the display unit (22).

Description

Remote communication system and remote communication method

[0001] The present disclosure relates to a remote communication system and a remote communication method.

[0002] In recent years, remote communication, which is communication between users located in different locations, has been used in a variety of situations. In remote communication, for example, a first user, who is the communication partner, is projected onto a monitor, a second user located in front of the monitor is photographed from the monitor side, and the photographed image is projected onto the monitor in front of the second user. Since photographed images are used in remote communication, there are restrictions on the position of the photographing device, making it difficult for users to align their gazes.

[0003] Patent Literature 1 discloses a technique for attaching a suction cup to the back of an image capturing device and using the suction cup to fix the image capturing device to the display surface of a monitor in order to capture an image of a user from a position close to the line of sight of the other party in remote communication. This allows the image capturing device to be fixed at any position on the monitor display screen, making it possible to capture an image from a position close to the line of sight of the other party.

[0004] Japanese Unexamined Patent Publication No. 7-154661

[0005] However, with the technology described in Patent Document 1, the user cannot see the image of the part where the camera is fixed. To make eye contact with the other person in remote communication, the camera must be fixed to a part of the image that corresponds to the position of the eyes. In this case, the image of the eyes of the other person in remote communication is hidden by the camera. This poses a problem in that the user cannot make eye contact with the other person.

[0006] The present disclosure has been made in view of the above, and aims to provide a remote communication system that allows a user to make eye contact with a remote communication partner.

[0007] In order to solve the above-mentioned problems and achieve the object, the present disclosure provides a remote communication system including a terminal device having a display unit and a camera provided on the display unit and operable by a second user who is to remotely communicate with a first user, and an extended monitor having a display surface and the terminal device being positioned on the display surface side so that the display unit overlaps the display surface as seen by the second user. The remote communication system further includes an image processing device that uses a captured image of the first user to generate a first image that is a part of the image and is displayed on the terminal device, and a second image that is at least a part of the image, the first image being a partial image that would be displayed in an area corresponding to the display unit if the image were to be displayed on the extended monitor.

[0008] The remote communication system according to the present disclosure has the effect of allowing a user to make eye contact with a remote communication partner.

[0009] FIG. 1 is a diagram showing a configuration example of a remote communication system according to a first embodiment. FIG. 1 is a diagram showing an example of the arrangement of a terminal device and an extended monitor according to the first embodiment. FIG. 2 is a diagram showing an example of the arrangement of a terminal device and an extended monitor according to the first embodiment. FIG. 3 is a diagram for explaining the effect of the first embodiment. FIG. 4 is a flowchart showing an example of a video processing procedure according to the first embodiment. FIG. 1 is a diagram showing a first alternative configuration example of a remote communication system according to the first embodiment. FIG. 2 is a diagram showing a second alternative configuration example of a remote communication system according to the first embodiment. FIG. 3 is a diagram showing a third alternative configuration example of a remote communication system according to the first embodiment. FIG. 4 is a diagram showing an example of a configuration of a computer system realizing a video processing device according to the first embodiment. FIG. 5 is a diagram showing a configuration example of a remote communication system according to the second embodiment. FIG. 6 is a diagram showing an example of a video shift in the second embodiment.

[0010] Hereinafter, a remote communication system and a remote communication method according to an embodiment will be described in detail with reference to the accompanying drawings.

[0011] 1 is a diagram showing an example of the configuration of a remote communication system according to a first embodiment. The remote communication system 100 of this embodiment supports remote communication between a first user located at a first point 5 and a second user located at a second point 6 that is different from the first point 5. A terminal device 1, which is a first terminal device, is provided at the first point 5. The terminal device 1 captures an image of the first user and transmits the captured image to the second point 6.

[0012] The remote communication system 100 of this embodiment includes a terminal device 2 that is a second terminal device, an extended monitor 3, and a video processing device 4. While Fig. 1 shows an example in which the terminal device 2, extended monitor 3, and video processing device 4 are installed at the second location 6, it is sufficient that the terminal device 2 and extended monitor 3 are installed at the second location 6, and the video processing device 4 may be installed at a location other than the second location 6. Note that the terminal device 1 may be included in the remote communication system 100.

[0013] The terminal device 1 is, for example, but not limited to, a smartphone, a tablet, a personal computer, etc. The terminal device 1 includes a communication unit 11, a display unit 12, a camera 13, a microphone 14, an input reception unit 15, a speaker 16, and a processing unit 17.

[0014] The processing unit 17 performs various processes in the terminal device 1. For example, the processing unit 17 controls the transmission and reception of video and audio for remote communication between a first user of the terminal device 1 and a second user of the terminal device 2. The communication unit 11 communicates with other devices. For example, under the control of the processing unit 17, the communication unit 11 transmits video captured by the camera 13 and audio collected by the microphone 14 to the terminal device 2. For example, under the control of the processing unit 17, the communication unit 11 receives video (video captured of the second user, described below) and audio (electrical audio signals including audio emitted by the second user) from the terminal device 2. The video received by the communication unit 11 is displayed on the display unit 12, and the audio received by the communication unit 11 is output by the speaker 16. Note that, here, an example will be described in which audio and video are transmitted and received between the terminal device 1 and the terminal device 2; however, as long as video is transmitted and received, audio transmission and reception is not essential. Furthermore, instead of sound, text data or the like may be transmitted and received between the terminal device 1 and the terminal device 2.

[0015] The display unit 12 displays various types of information. For example, the display unit 12 displays video received by the communication unit 11. Furthermore, when the communication unit 11 receives data such as text for remote communication from the terminal device 2, the display unit 12 may display the data. The camera 13 captures video. For example, the camera 13 captures a picture of the first user and outputs the captured video to the communication unit 11. The microphone 14 converts sound into an electrical signal. For example, the microphone 14 converts sound, including the voice uttered by the first user, into an electrical signal and outputs the sound converted into an electrical signal to the communication unit 11.

[0016] The input accepting unit 15 accepts input from the first user and outputs input information indicating the content of the accepted input to the processing unit 17. The input accepting unit 15 accepts, for example, an input instructing the start of remote communication with the second user and outputs input information indicating the content of the input to the processing unit 17. As a result, the processing unit 17 starts controlling the operation of the terminal device 1 for performing remote communication. Furthermore, when input information such as text is transmitted from the terminal device 1 to the terminal device 2 during remote communication, the input information accepted by the input accepting unit 15 is transmitted to the terminal device 2 via the communication unit 11. The speaker 16 outputs, as sound, voice input as an electrical signal. For example, the speaker 16 outputs, as sound, an electrical sound signal received by the communication unit 11 from the terminal device 2.

[0017] The terminal device 2 at the second location 6 is a terminal device operable by a second user who remotely communicates with the first user. Like the terminal device 1, the terminal device 2 may be, for example, a smartphone, a tablet, a personal computer, or the like, but is not limited to these. The terminal device 2 includes a communication unit 21, a display unit 22, a camera 23, a microphone 24, an input reception unit 25, a speaker 26, and a processing unit 27. The basic functions of the communication unit 21, the display unit 22, the camera 23, the microphone 24, the input reception unit 25, the speaker 26, and the processing unit 27 are similar to those of the communication unit 11, the display unit 12, the camera 13, the microphone 14, the input reception unit 15, the speaker 16, and the processing unit 17 of the terminal device 1, respectively. However, the user of the terminal device 2 is changed from the first user to the second user, and the communication partner of the terminal device 2 becomes the terminal device 1. That is, for example, the camera 23 captures video of the second user, and the microphone 24 converts sound, including the voice of the second user, into an electrical signal. Furthermore, the communication unit 21 receives video (video captured of the first user) and sound (sound including the voice emitted by the first user) from the terminal device 1, and the speaker 26 outputs the electrical sound signal received by the communication unit 21 from the terminal device 1 as sound. The communication unit 21 transmits the video received from the terminal device 1 to the video processing device 4, and receives video for terminal device 2, which will be described later, from the video processing device 4. The video for terminal device 2 is part of the video transmitted from the terminal device 1. The video for terminal device 2 received by the communication unit 21 is displayed by the display unit 22.

[0018] The communication line between the terminal device 1 and the terminal device 2 may be a wireless communication line or a wired communication line. The communication line between the terminal device 2 and the video processing device 4 may also be a wireless communication line or a wired communication line.

[0019] As described above, remote communication between a first user and a second user can be realized by using the terminal device 1 and the terminal device 2. Any processing other than the video processing described below may be used for the detailed processing related to realizing remote communication between the first user and the second user. For example, general processing for remote communication may be used, and therefore detailed description will be omitted. Furthermore, there are no particular restrictions on the purpose of the remote communication, and it may be a meeting, education, leisure, communication related to remote control, or other purposes.

[0020] Note that, although an example in which the video of the first user is captured by the terminal device 1 will be described here, the present invention is not limited to this, and the video of the first user may be captured by an external camera separate from the terminal device 1, or may be video captured by a camera module virtually placed in a VR (Virtual Reality) space. For example, if the first user exists in the VR space as an avatar or the like, the video of the first user may be video captured by a camera module virtually placed in the VR space.

[0021] When the image captured by the camera 13 of the terminal device 1 is displayed on the display unit 22 of the terminal device 2, the second user can see the first user in the image. On the other hand, remote communication is performed via image, making eye contact with the other party difficult. To achieve eye contact, the camera 23 must be placed at a position corresponding to the line of sight. However, if the camera 23 is placed in front of the display unit 22, the image from the camera 23 will be hidden. Using a half mirror is also an option, but this increases the device size. To solve these problems, a camera (called an embedded camera), which is placed under a display such as an LCD panel, can be used. That is, the camera 23 is provided in the display unit 22 and is capable of capturing images of the display unit 22 from inside the display unit 22, for example. In this embodiment, a camera placed under the display is used as the camera 23. That is, in this embodiment, a camera 23 embedded under the display unit 22 is used as the camera 23. This solves the above problem.

[0022] For example, many smartphones, tablets, and the like are equipped with built-in cameras, and it is possible to use such cameras, but these built-in cameras are generally provided at the edge of the display unit 22. Generally, remote communication is often carried out with the other party's face displayed near the center of the display unit 22, and therefore, if the camera 23 is provided at the edge of the display unit 22, it would be impossible to make eye contact.

[0023] In this embodiment, even when the camera 23 is provided at the end of the display unit 22, the extended monitor 3 is provided so that the second user can make eye contact with the first user. The extended monitor 3 has a display surface, and the terminal device 2 is placed on the display surface side of the extended monitor 3 so that the display unit 22 overlaps the display surface as seen by the second user. The video processing device 4 then generates a video for the terminal device 2 and a video for the extended monitor 3 so that seamless videos are projected on the display unit 22 of the terminal device 2 and the extended monitor 3. Specifically, the video processing device 4 generates a video for the terminal device 2 (first video) and a video for the extended monitor 3 (second video) from the video received from the terminal device 1, and the display unit 22 displays the first video, and the extended monitor 3 displays the second video. The first video is a partial video that would be displayed in an area corresponding to the display unit 22 if the video received from the terminal device 1 were displayed on the extended monitor 3. The second video is at least a part of the video received from the terminal device 1 and is seamlessly connected to the first video. As a result, the remote communication system 100 of this embodiment can achieve natural eye contact.

[0024] The video processing device 4 includes a communication unit 41 and a video processing unit 42. The communication unit 41 receives video from the terminal device 2 and outputs it to the video processing unit 42. The video processing unit 42 generates a first video that is a video for the terminal device 2 and a second video that is a video for the extended monitor 3 from the received video, and outputs the first video to the communication unit 41 and the second video to the extended monitor 3. The communication unit 41 transmits the first video to the terminal device 2. Note that FIG. 1 shows an example in which the video processing device 4 is connected to the extended monitor 3 by a video transmission cable or the like, and the second video is transmitted (output) from the video processing unit 42 to the extended monitor 3 via the cable or the like, but this is not limiting. The extended monitor 3 may have a communication function, and the second video may also be transmitted to the extended monitor 3 via the communication unit 41.

[0025] 2 and 3 are diagrams showing an example of the arrangement of the terminal device 2 and the extended monitor 3 according to this embodiment. FIG. 2 schematically shows the terminal device 2 and the extended monitor 3 as viewed from the display surface side of the extended monitor 3, and FIG. 3 schematically shows the terminal device 2 and the extended monitor 3 as viewed from the side of the extended monitor 3. In FIG. 3, the left side shows the display surfaces of the extended monitor 3 and the display unit 22 of the terminal device 2. As shown in FIGS. 2 and 3, the terminal device 2 is arranged above the display surface of the extended monitor 3. The lens 231 in FIG. 2 is the lens of the camera 23 of the terminal device 2.

[0026] FIG. 4 is a diagram illustrating an example of the first image and the second image according to the present embodiment. In the example illustrated in FIG. 4 , as illustrated in FIGS. 2 and 3 , the terminal device 2 is placed on the display surface of the extended monitor 3. The terminal device 2 displays a partial image 200, which is an example of the first image, and the extended monitor 3 displays a partial image 300, which is an example of the second image, with the partial image 200 and the partial image 300 seamlessly connected. Note that in FIG. 3 , the back surface of the terminal device 2 is in contact with the display surface of the extended monitor 3, but the back surface of the terminal device 2 may be separated from the display surface of the extended monitor 3. However, if the back surface of the terminal device 2 is too far from the display surface of the extended monitor 3, the images may not be connected. Therefore, it is preferable that the distance between the back surface of the terminal device 2 and the display surface of the extended monitor 3 be equal to or less than a certain value. Note that FIG. 4 illustrates an example in which the terminal device 2 is a bezel-less terminal device, such as a bezel-less smartphone or bezel-less tablet, in which the display unit 22 of the terminal device 2 extends across the entire surface of the terminal device 2, with no or very small borders around the display unit 22. Although the terminal device 2 is bezel-less, which means that less of the image is hidden, the terminal device 2 is not limited to being bezel-less and may have a border around the display unit 22 .

[0027] Partial video 200 is a division of the video received by terminal device 2 from terminal device 1, i.e., a portion of the video received by terminal device 2 from terminal device 1. Partial video 300 is the video received by terminal device 2 from terminal device 1, excluding the portion of partial video 200. The second video, which is the video for extended monitor 3, may be the entire video received by terminal device 2 from terminal device 1. When the video received by terminal device 2 from terminal device 1 is displayed on display unit 22, the video may be enlarged by an operation of the second user. In this case, the first video is a portion of the enlarged video, and the second video may be the enlarged video excluding the portion of the first video, or may be the entire enlarged video.

[0028] FIG. 5 is a diagram illustrating the effects of this embodiment. The upper part of FIG. 5 shows an example in which the video received from terminal device 1 is displayed as is by terminal device 2, and the lower part of FIG. 5 shows an example in which the video received from terminal device 1 is displayed separately as a partial video 301 for terminal device 2 and a partial video 300 for extended monitor 3, similar to FIG. 4 . As shown in the upper part of FIG. 5 , the lens 231 of the camera 23 of terminal device 2 is located near the upper end of terminal device 2. In this case, when remotely communicating with the first user, the second user must direct their gaze toward the vicinity of the first user's eyes displayed as an image on terminal device 2 in order to make eye contact with the first user. However, because the lens 231 of terminal device 2 is located near the upper end of terminal device 2, the second user is photographed from a position above the position corresponding to the second user's line of sight, and the first and second users' eye contact cannot be achieved. Furthermore, if the image is displayed so that the position of the first user's eyes is at the position of the lens 231, the first user's face will be positioned at the top end and the entire face will not be displayed.

[0029] 5, the face of the first user is displayed near the center of the overall image 302, which is a combination of the partial image 301 displayed on the terminal device 2 and the partial image 300 displayed on the extended monitor 3, and the lens 231 is positioned at a position corresponding to the position of the first user's eyes. Therefore, for example, by placing the terminal device 2 on the extended monitor 3 so that the position of the camera 23 of the terminal device 2, i.e., the position of the lens 231, is near the center of the extended monitor 3, the second user can align their gaze with the first user. In this embodiment, it is assumed that the position of the terminal device 2 on the extended monitor 3 is determined in advance. For example, on the adjustment screen, a frame for placing the terminal device 2 may be displayed on the extended monitor 3, or a mark indicating the placement position of the terminal device 2 may be applied to the extended monitor 3.

[0030] In this embodiment, the second user being photographed from the position of the other user's eyes projected as an image (for example, the center of the other user's eyes) is referred to as "gaze alignment." To achieve two-way gaze alignment, it is desirable for the first user to also photograph the first user from the position of the other user's eyes projected as an image (for example, the center of the other user's eyes), but in this embodiment, this restriction does not apply to the first user's photographing.

[0031] Next, the operation of the video processing device 4 of this embodiment will be described. Fig. 6 is a flowchart showing an example of a video processing procedure of this embodiment. As shown in Fig. 6, the video processing device 4 receives video (step S1). In detail, the communication unit 41 receives video captured by the first user from the terminal device 2 and outputs the received video to the video processing unit 42. The video received by the communication unit 41 may be the video itself received by the terminal device 2 from the terminal device 1, or may be a video enlarged based on an input from the terminal device 2 instructing the second user to enlarge the video.

[0032] Next, the video processing device 4 generates a video (first video) for the terminal device 2 (step S2). Specifically, the video processing unit 42 generates the first video by dividing the video received from the communication unit 41 into a first video, which is a video of a portion corresponding to the position of the terminal device 2 on the extended monitor 3, and the remaining portion of the video. The position of the terminal device 2 on the extended monitor 3 is, for example, predetermined, and the video processing unit 42 holds position information indicating the position of the terminal device 2 on the extended monitor 3. The position of the terminal device 2 on the extended monitor 3 is determined so that the lens 231 of the terminal device 2 is positioned at the position of the eyes of the first user when displayed as an image. The eye position is, for example, the center of both eyes, but is not limited thereto. The eye position is determined in advance on the assumption that the first user will be displayed approximately in the center of the extended monitor 3, but is not limited thereto. The position information may be, for example, indicated by coordinate values ​​within the video displayed on the extended monitor 3, measured in units of pixels. Alternatively, the position information may be information indicating the physical size of the display surface of the extended monitor 3 and coordinates indicating the physical position of the terminal device 2 on the display surface of the extended monitor 3. Furthermore, the position information may be information indicating the positions of the four corners of the terminal device 2, or information indicating the position of the terminal device 2 in pixel units.

[0033] Next, the video processing device 4 generates a video (second video) for the extended monitor 3 (step S3). In detail, the video processing unit 42 determines the video other than the portion corresponding to the position of the terminal device 2 on the extended monitor 3, which was divided in step S2, as the second video. Alternatively, the video itself received by the communication unit 41 may be determined as the second video.

[0034] Next, the video processing device 4 transmits the first video and the second video (step S4). Specifically, the video processing unit 42 transmits the first video to the terminal device 2 via the communication unit 41, and transmits the second video to the extended monitor 3.

[0035] Next, the video processing device 4 determines whether to end the video display (step S5), and ends the process if the video display is to be ended (step S5 Yes). If the video display is not to be ended (step S5 No), the video processing device 4 repeats the process from step S1. In step S5, for example, the video processing unit 42 may determine to end the video display when it receives an instruction to end the video display from the terminal device 2 via the communication unit 41, or may determine to end the video display when it does not receive any video from the communication unit 41 for a certain period of time or more.

[0036] Through the above processing, the terminal device 2 receives the first video from the video processing device 4 and displays the received first video. Furthermore, the extended monitor 3 receives the second video from the video processing device 4 and displays the received second video. This allows the second user to view the seamlessly connected first video and second video as a single video, as illustrated in FIG. 4 . Furthermore, as described with reference to FIGS. 4 and 5 , even if the lens 231 of the terminal device 2 is provided at the end of the terminal device 2, the second user can see the entire face of the first user while making eye contact with the first user.

[0037] FIG. 1 illustrates an example in which the terminal device 2 is a smartphone, tablet, or the like. However, the terminal device 2 only needs to include a display unit 22 and a camera 23, and other components may be provided outside the terminal device 2. FIG. 7 is a diagram showing a first alternative configuration example of the remote communication system 100 of this embodiment. Components having the same functions as those in FIG. 1 are assigned the same reference numerals as in FIG. 1, and redundant explanations will be omitted. In the example shown in FIG. 7, the terminal device 2 includes a display unit 22 and a camera 23, and the function of the communication unit 21 is provided by a communication unit 41 of the video processing device 4. That is, the video processing device 4 receives video transmitted from the terminal device 1 and outputs the received video to a video processing unit 42.

[0038] 7 , the input reception unit 25, the microphone 24, and the speaker 26 are provided separately from the terminal device 2 and the video processing device 4, and the video processing device 4 includes a processing unit 43, which controls the terminal device 2, the input reception unit 25, the microphone 24, and the speaker 26. For example, the processing unit 43 transmits input information indicating the content of the input received by the input reception unit 25 and the sound collected by the microphone 24 to the terminal device 1 via the communication unit 41, and outputs the sound received by the communication unit 41 to the speaker 26. Similar to the configuration example shown in FIG. 1 , the video processing device 4 generates a first image for the terminal device 2 and a second image for the extended monitor 3 from the image transmitted from the terminal device 1, outputs the second image to the extended monitor 3, and outputs the first image to the communication unit 41. 7 shows that the input reception unit 25, the microphone 24, and the speaker 26 are all provided externally, but this is not limiting, and at least one of the input reception unit 25, the microphone 24, and the speaker 26 may be provided within the video processing device 4 or the terminal device 2. Also, a communication terminal having a communication unit 41 may be provided separately from the terminal device 2 and the video processing device 4, and the communication terminal may receive video from the terminal device 1 and output the received video to the video processing device 4. Also, while FIG. 7 shows an example in which the terminal device 2 is connected to the video processing device 4 by a cable or the like, the terminal device 2 may include the communication unit 21, and communication between the video processing device 4 and the terminal device 2 may be performed via the communication unit 41 and the communication unit 21.

[0039] While FIG. 1 illustrates an example in which the terminal device 2 and the video processing device 4 are provided separately, the video processing device 4 and the terminal device 2 may be integrated. FIG. 8 illustrates a second alternative configuration example of the remote communication system 100 of this embodiment. In the example illustrated in FIG. 8, the video processing device 4 is provided in the processing unit 27 of the terminal device 2. Components having the same functions as those in FIG. 1 are assigned the same reference numerals as in FIG. 1, and redundant description will be omitted. In the example illustrated in FIG. 8, the video processing device 4 is provided in the processing unit 27, but the video processing device 4 may be provided in the terminal device 2 separately from the processing unit 27. In the example illustrated in FIG. 8, the second video generated by the video processing device 4 may also be transmitted to the extended monitor 3 via the communication unit 21.

[0040] 8 shows an example in which the video processing device 4 and the terminal device 2 are integrated, but the extended monitor 3 and the video processing device 4 may also be integrated. For example, the video processing device 4 may be connected to and integrated with the extended monitor 3 shown in FIG. 1 , or the extended monitor 3 may be an information processing device with a monitor, such as a tablet or a personal computer, and the information processing device may have the function of the video processing device 4.

[0041] In this embodiment, by using an in-camera or front camera of a smartphone, tablet, or the like, eye contact can be achieved with a simple device configuration at the second point 6. It is preferable that measures are taken to achieve eye contact at the first point 5 as well. This measure may be a method using an extended monitor 3, as with the second point 6, or another method. There are no particular restrictions on the device configuration for achieving eye contact at the first point 5. As with the second point 6, the first point 5 may also be provided with a terminal device such as a smartphone, an extended monitor, and a video processing device, and the video processing device may similarly generate a first image for the terminal device and a second image for the extended monitor. In this case, the video processing device 4 that processes the image displayed at the second point 6 and the video processing device that processes the image displayed at the first point 5 may be the same device. By using a device configuration similar to that of the second point 6, eye contact can be achieved with a simple device configuration at the first point 5.

[0042] FIG. 1 also illustrates an example in which the terminal device 1, an information processing device with a monitor such as a smartphone, is provided at the first location 5. However, the device provided at the first location 5 may be divided into multiple devices. FIG. 9 illustrates a third alternative configuration example of the remote communication system 100 according to the present embodiment. Components having the same functions as those in FIG. 1 are designated by the same reference numerals as in FIG. 1, and redundant description will be omitted. For example, the terminal device 1 illustrated in FIG. 1 may be divided into a terminal device 1a, which is a monitor with a camera, and a terminal device 1b, as illustrated in FIG. 9. The terminal device 1a includes a display unit 12 and a camera 13. The camera 13 is installed, for example, at a predetermined location on the display unit 12 where an image of the second user's eyes is displayed. For example, a hole may be provided on the surface of the display unit 12 at a predetermined location, and the camera 13 may be provided behind the hole. FIG. 9 also illustrates an example in which the camera 13 is embedded in the terminal device 1a. However, the monitor, which is the display unit 12, and the camera 13 may be provided separately. For example, the camera 13 may be positioned so that the line of sight of the first user and the second user can be aligned using a half mirror.

[0043] Next, the hardware configuration of each device of this embodiment will be described. The video processing device 4 of this embodiment shown in FIG. 1 functions as the video processing device 4 by executing a computer program, which is a computer program that describes the processing in the video processing device 4, on the computer system. FIG. 10 is a diagram showing an example of the configuration of a computer system that realizes the video processing device 4 of this embodiment. As shown in FIG. 10, this computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107. The control unit 101 and the storage unit 103 form a processing circuit.

[0044] In FIG. 10 , the control unit 101 is a processor such as a CPU (Central Processing Unit) and executes a program describing the processing of the video processing device 4 of this embodiment. Note that a portion of the control unit 101 may be implemented by dedicated hardware such as a GPU (Graphics Processing Unit) or FPGA (Field-Programmable Gate Array). The input unit 102 may be an input device such as a button, keyboard, mouse, joystick, touchpad, or game controller, or may be an input device that detects a user's gestures based on a captured image of the user or the operator's muscle movements and accepts input. The storage unit 103 includes various types of memory such as RAM (Random Access Memory) and ROM (Read Only Memory) and a storage device such as a hard disk, and stores programs to be executed by the control unit 101, necessary data obtained during processing, and the like. The storage unit 103 is also used as a temporary storage area for programs. The display unit 104 is, as described above, a display, for example. Note that the display unit 104 and the input unit 102 may be integrated and implemented as a touch panel or the like. The communication unit 105 is a receiver and transmitter that perform communication processing. The output unit 106 is a speaker or the like. Note that Fig. 10 is an example, and the configuration of the computer system is not limited to the example of Fig. 10. For example, in this embodiment, the computer system that realizes the video processing device 4 does not need to include the display unit 104 and the output unit 106.

[0045] Here, an example of the operation of the computer system until the program of this embodiment is ready to be executed will be described. In the computer system having the above configuration, for example, a computer program is installed in storage unit 103 from a CD-ROM or DVD-ROM inserted in a CD (Compact Disc)-ROM drive or DVD (Digital Versatile Disc)-ROM drive (not shown). Then, when the program is executed, the program read from storage unit 103 is stored in the main storage area of ​​storage unit 103. In this state, control unit 101 executes processing as video processing device 4 of this embodiment in accordance with the program stored in storage unit 103.

[0046] In the above explanation, a program describing the processing in the video processing device 4 is provided using a CD-ROM or DVD-ROM as a recording medium, but this is not limited to this, and depending on the configuration of the computer system, the capacity of the program to be provided, etc., it is also possible to use a program provided via a transmission medium such as the Internet.

[0047] The video processing unit 42 shown in Fig. 1 is realized by the control unit 101 shown in Fig. 10 executing a computer program stored in the storage unit 103 shown in Fig. 10. The storage unit 103 shown in Fig. 10 is also used to realize the video processing unit 42 shown in Fig. 1. The communication unit 41 shown in Fig. 1 is realized by the communication unit 105 shown in Fig. 10.

[0048] Terminal device 1 and terminal device 2 shown in FIG. 1 are each realized, for example, by a computer system having the configuration shown in FIG. 10. In the example shown in FIG. 7, video processing device 4 is realized, for example, by a computer system having the configuration shown in FIG. 10. Also, in the example shown in FIG. 7, terminal device 2 only needs to be equipped with a display unit 22 such as a monitor or display and a camera 23, and does not need to be a computer system. In the example shown in FIG. 8, terminal device 2 is realized, for example, by a computer system having the configuration shown in FIG. 10. Also, in the example shown in FIG. 9, terminal device 1b is realized, for example, by a computer system having the configuration shown in FIG. 10. Also, in the example shown in FIG. 9, terminal device 1a only needs to be equipped with a display unit 22 such as a monitor or display and a camera 23, and does not need to be a computer system.

[0049] As described above, the remote communication system 100 of this embodiment displays a first image that is part of the image of the first user on the terminal device 2, and displays a second image that is seamlessly connected to the first image on the extended monitor 3 provided on the back of the terminal device 2. The camera 23 is embedded in the display unit 22 of the terminal device 2, and the camera 23 is placed at a position corresponding to the position of the eyes of the first user. This allows the second user to make eye contact with the first user, who is the partner in remote communication, using the terminal device 2 with the camera 23 embedded at its end.

[0050] Second Embodiment Fig. 11 is a diagram showing an example of the configuration of a remote communication system according to a second embodiment. The remote communication system 100a of this embodiment is similar to the remote communication system 100 of the first embodiment, except that it includes a video processing device 4a instead of the video processing device 4. Components having the same functions as those of the first embodiment are given the same reference numerals as those of the first embodiment, and redundant explanations will be omitted. Below, differences from the first embodiment will be mainly explained.

[0051] The video processing device 4a is similar to the video processing device 4 of embodiment 1 except that it includes an eye detection unit 44 and an image shifting unit 45. The communication unit 41 outputs the video received from the terminal device 2 to the eye detection unit 44. The eye detection unit 44 detects the position of the first user's eyes in the video received from the communication unit 41 and outputs the detection result together with the video to the image shifting unit 45. The eye position may be, for example, but is not limited to, the center of both eyes. The image shifting unit 45 shifts the video received from the eye detection unit 44 using the detection result received from the eye detection unit 44 and outputs the shifted video to the video processing unit 42. Specifically, the image shifting unit 45 shifts the video so that the position of the eyes detected by the eye detection unit 44 coincides with the position corresponding to the camera 23. The video processing unit 42 generates a first video and a second video using the video after being shifted by the image shifting unit 45, as in embodiment 1.

[0052] For example, a reference position for the eye position in the video is determined in advance, and the video shifter 45 shifts the video by the difference between the reference position and the detection result. For example, if the coordinates of the reference position are X0, Y0 and the coordinates of the detection result are Xe, Ye, the video shifter 45 shifts the video by X0-Xe, Y0-Ye. The processing in the video processor 42 is the same as in the first embodiment, but the position of the terminal device 2 is determined so that the position of the lens 231 is the above-mentioned reference position.

[0053] Fig. 12 is a diagram showing an example of image shifting according to this embodiment. The upper part of Fig. 12 shows a display example when image shifting is not performed, and the lower part of Fig. 12 shows a display example when image shifting is performed. As shown in the upper part of Fig. 12, when image shifting is not performed, eye position 310 is slightly deviated from the position of lens 231 of terminal device 2, i.e., the position of camera 23 of terminal device 2. In this embodiment, the image is shifted as shown in the lower part of Fig. 12 using the detection result of eye position 310 in the image, so that eye position 310 can be aligned with the position of camera 23 of terminal device 2.

[0054] 13 is a flowchart showing an example of a video processing procedure of this embodiment. Step S1 is the same as in embodiment 1, except that communication unit 41 outputs the received video to eye detection unit 44. Next, eye detection unit 44 detects the positions of eyes in the video (step S11). Eye detection unit 44 outputs the detection result together with the video to video shifting unit 45. Next, video shifting unit 45 shifts the video (step S12). In detail, video shifting unit 45 shifts the video received from eye detection unit 44 using the detection result received from eye detection unit 44, and outputs the shifted video to video processing unit 42. Step S2 and subsequent steps are the same as in embodiment 1.

[0055] 11, eye detection unit 44 and image shift unit 45 are added to video processing device 4 of remote communication system 100 shown in FIG. 1, but this is not limiting, and eye detection unit 44 and image shift unit 45 may be added to video processing device 4 of the configuration examples described as the first to third alternative configuration examples in embodiment 1 or the other configuration examples described in embodiment 1. Furthermore, eye detection unit 44 and image shift unit 45 may be provided in terminal device 2, and video processing device 4 may receive the image after the image shift and generate the first image and the second image.

[0056] Like the video processing device 4 of the first embodiment, the video processing device 4a in the remote communication system 100a of the present embodiment is also realized by, for example, the computer system illustrated in Fig. 10. The eye detection unit 44 and the video shifting unit 45 shown in Fig. 11 are realized by the control unit 101 shown in Fig. 10 executing a computer program stored in the storage unit 103 shown in Fig. 10. The storage unit 103 shown in Fig. 10 is also used to realize the eye detection unit 44 and the video shifting unit 45 shown in Fig. 11.

[0057] Third Embodiment Fig. 14 is a diagram showing an example of the configuration of a remote communication system according to a third embodiment. A remote communication system 100b according to this embodiment is similar to the remote communication system 100a according to the second embodiment, except that it includes a video processing device 4b instead of a video processing device 4a, and a position detection device 8 is added. Components having the same functions as those in the second embodiment are assigned the same reference numerals as those in the second embodiment, and redundant explanations will be omitted. Differences from the second embodiment will be mainly explained below.

[0058] The position detection device 8 detects the position of the terminal device 2. Specifically, the position detection device 8 detects the position of the terminal device 2 on the extension monitor 3 and notifies the video processing device 4b of position information indicating the detected position. If the extension monitor 3 has a touch panel, the position detection device 8 may be integrated with the extension monitor 3 and implemented as a function of the extension monitor 3. In this case, for example, a touch panel-sensitive sticker may be attached to the back surface of the terminal device 2. Alternatively, the position detection device 8 may be a camera that captures an image of the back surface of the terminal device 2, or may be a device that detects the position using a LiDAR (Light Detection and Ranging) and an IMU (Inertial Measurement Unit) provided on the back surface of the terminal device 2. If the position detection device 8 is a camera that captures an image of the back surface of the terminal device 2, for example, an adjustment screen including position detection marks, numbers, marks, etc. is displayed on the extension monitor 3, and the image shift unit 45a of the terminal device 2 or the video processing device 4b analyzes the image captured by the camera to calculate the position of the terminal device 2. Furthermore, a camera may be installed on the extended monitor 3 as the position detection device 8, and the image shifting unit 45a of the terminal device 2 or the image processing device 4b may analyze the image captured by the camera to calculate the position of the terminal device 2. The position detection device 8 is not limited to the above example as long as it can detect the position of the terminal device 2 on the extended monitor 3.

[0059] The video processing device 4b is similar to the video processing device 4a of the second embodiment, except that it includes a video shifting unit 45a instead of the video shifting unit 45. The video shifting unit 45a shifts the video received from the eye detection unit 44 using the detection result received from the eye detection unit 44 and the position information received from the position detection device 8. That is, the video shifting unit 45a determines the camera position, which is the position of the camera 23, using the position of the terminal device 2 detected by the position detection device 8, and shifts the video so that the position of the eyes detected by the eye detection unit 44 matches the camera position. For example, the video shifting unit 45a calculates the position of the camera on the extended monitor 3 using the position information received from the position detection device 8 and information indicating the position of the camera 23 on the terminal device 2. If the coordinates of the calculated camera position are Xc, Yc and the coordinates of the detection result received from the eye detection unit 44 are Xe, Ye, the video shifting unit 45a shifts the video by Xc - Xe, Yc - Ye.

[0060] 15 is a flowchart showing an example of a video processing procedure according to the present embodiment. Steps S1 and S11 are the same as those in the second embodiment. The video shifter 45a shifts the video based on the detected eye positions and the detected position of the terminal device 2 (the position of the terminal device 2 detected by the position detection device 8) (step S21). Steps S2 and onward are the same as those in the first embodiment.

[0061] By the above operations, in this embodiment, even if the placement position of the terminal device 2 on the extended monitor 3 is not predetermined or if the placement position of the terminal device 2 on the extended monitor 3 is shifted from the predetermined position, the position of the eyes of the first user in the video can be made to match the position of the camera 23 of the terminal device 2, as in embodiment 2.

[0062] 14 shows an example in which a position detection device 8 is added to the remote communication system 100 shown in FIG. 1 and an image shifting unit 45a is provided instead of the image shifting unit 45, but this is not limiting, and the position detection device 8 may be added and an image shifting unit 45a may be provided instead of the image shifting unit 45 in the remote communication system 100 of the configuration examples described as the first to third alternative configuration examples in embodiment 1 or the other configuration examples described in embodiment 1. Furthermore, the eye detection unit 44 and the image shifting unit 45a may be provided in the terminal device 2, and the image processing device 4 may receive the image after the image shift and generate the first image and the second image.

[0063] Like the video processing devices 4 and 4a of the first and second embodiments, the video processing device 4b in the remote communication system 100b of the present embodiment is realized by, for example, the computer system illustrated in Fig. 10. The video shifting unit 45a shown in Fig. 14 is realized by the control unit 101 shown in Fig. 10 executing a computer program stored in the storage unit 103 shown in Fig. 10. The storage unit 103 shown in Fig. 10 is also used to realize the video shifting unit 45a shown in Fig. 14.

[0064] Fourth Embodiment The remote communication systems 100, 100a, and 100b described in the first to third embodiments can be applied to remote communication for various purposes. In the fourth embodiment, an example in which the remote communication systems are used in combination with remote operation will be described.

[0065] Fig. 16 is a diagram showing an example of the configuration of a remote machine operation system 500 according to the fourth embodiment. Components having the same functions as those in the first embodiment are given the same reference numerals as those in the first embodiment, and redundant explanations will be omitted. As shown in Fig. 16, the remote machine operation system 500 includes the terminal device 2, extension monitor 3, and video processing device 4 that constitute the remote communication system 100 according to the first embodiment. In other words, the remote machine operation system 500 includes the remote communication system 100 according to the first embodiment.

[0066] In the example shown in FIG. 16 , an operator 403, who is a first user located at a first location 5, remotely controls a remote machine 401 located at a second location 6 using the input receiving unit 15 of the terminal device 1 as an operation means. Operation means such as a joystick, touchpad, button, keyboard, mouse, or game controller may be provided separately from the terminal device 1, and the terminal device 1 may accept input from these operation means. The terminal device 1 transmits operation information indicating the content of the operation input by the operator 403 and the video and sound described in the first embodiment to the control device 402. The control device 402 transmits a control signal to the remote machine 401 using the operation information to operate the remote machine 401, and transmits the video and sound to the terminal device 2. The remote machine 401 operates in accordance with the control signal. The remote machine 401 may be a machine with a manipulator, a moving vehicle, a machine with a manipulator and a moving machine, or other types of machine.

[0067] 16 , the control device 402 receives operation information, video, and the like from the terminal device 1, but this is not limiting, and the terminal device 2 may receive operation information, video, and the like from the terminal device 1 and transmit the operation information to the control device 402.

[0068] If a second user, the recipient 404, is present near the remote machine 401, a sudden movement of the remote machine 401 may cause anxiety to the recipient 404. When the operator 403 and the recipient 404 perform remote communication, the operator 403 can notify the recipient 404 of the details of the movement in advance, or the recipient 404 can inform the operator 403 of his or her own situation, thereby reducing the recipient 404's anxiety. In this embodiment, as described in embodiment 1, images are displayed on the terminal device 2 and the extended monitor 3, allowing the recipient 404 to make eye contact with the operator 403, thereby promoting smooth remote communication. FIG. 16 shows an example in which the remote communication system 100 described in embodiment 1 is used in the remote machine operation system 500. However, this is not limiting, and the remote communication systems 100a and 100b described in embodiment 2 or 3 may also be used in the remote machine operation system 500.

[0069] 17 and 18 are diagrams illustrating an example of how the terminal device 2 and the extended monitor 3 of this embodiment are used. In the example shown in FIGS. 17 and 18 , a communication avatar for communication is placed around the remote machine 401, and the extended monitor 3 is used as part of the communication avatar. The communication avatar resembles at least a part of a human or animal's body, such as a face, waist, arms, or legs, and may be extremely deformed. In the example shown in FIGS. 17 and 18 , the extended monitor 3 is provided on top of a base 501, which corresponds to the torso or the like. As a result, the extended monitor 3 corresponds to the face of the communication avatar, and by using the terminal device 2 in conjunction with the extended monitor 3, the recipient 404 can make eye contact with the operator 403 via the communication avatar. By providing not only the terminal device 2 and the extended monitor 3 but also a base 501 simulating a torso and arms as shown in Fig. 17 or a base 501 simulating a torso, arms, and legs as shown in Fig. 18, the recipient 404 can more smoothly communicate with the operator 403. Note that the shape of the base 501 is not limited to the examples shown in Figs. 17 and 18.

[0070] Furthermore, in the above description, an example has been described in which the terminal device 2 and the extended monitor 3 are used as part of a communication avatar, but the base 501 may be the remote machine 401 itself. That is, for example, the extended monitor 3 may be provided on top of the remote machine 401. The position of the extended monitor 3 on the remote machine 401 is not limited to the examples shown in Figures 17 and 18. In this case, too, by using the terminal device 2 in combination with the extended monitor 3, the recipient 404 can make eye contact with the operator 403, enabling smooth communication and reducing anxiety about the movements of the remote machine 401.

[0071] As described in the first to third embodiments, the use of remote communication is not limited to the use in the present embodiment.

[0072] The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. It is also possible to omit or modify parts of the configurations as long as they do not deviate from the gist of the invention.

[0073] 1, 1a, 1b, 2 Terminal device, 3 Extended monitor, 4, 4a, 4b Image processing device, 5 First location, 6 Second location, 8 Position detection device, 11, 21, 41, 105 Communication unit, 12, 22, 104 Display unit, 13, 23 Camera, 14, 24 Microphone, 15, 25 Input reception unit, 16, 26 Speaker, 17, 27, 43 Processing unit, 42 Image processing unit, 44 Eye detection unit, 45, 45a Image shift unit, 100, 100a, 100b Remote communication system, 101 Control unit, 102 Input unit, 103 Memory unit, 106 Output unit, 107 System bus, 231 Lens, 401 Remote machine, 402 Control device, 403 Operator, 404 Recipient, 500 Remote machine operation system.

Claims

1. A remote communication system used for communication between a first user at a first location and a second user at a second location different from the first location, comprising: a first terminal device provided at the first location, using an image of the first user from a first camera that captures an image of the first user, or a camera module that is virtually arranged in a virtual reality space that captures an image of the first user existing as an avatar in the virtual reality space; a video processing device that uses a video of the first user from the first terminal device to generate a first video from a part of the video of the first user, and a second video from at least a part of the video of the first user that is seamlessly connected to the first video; a second terminal device provided at the second location and configured to display the first image; an extension monitor provided on a rear surface of the second terminal device and configured to display the second image; A remote communication system comprising:

2. The second terminal device includes a second camera that photographs the second user; 2. The remote communication system according to claim 1, wherein the second camera is provided at a position corresponding to a line of sight of the first user displayed on the second terminal device.

3. The second terminal device includes a second camera that photographs the second user, and further includes: an eye detection unit that detects a position of the eyes of the first user in an image of the first user; an image shifting unit that shifts an image of the first user so that the positions of the eyes detected by the eye detection unit correspond to positions corresponding to the second camera; Equipped with The remote communication system according to claim 1 , characterized in that the image processing device generates the first image and the second image using the image of the first user after it has been shifted by the image shifting unit.

4. a position detection device that detects the position of the second terminal device; Equipped with The remote communication system described in claim 3, characterized in that the image shifting unit determines a camera position, which is the position of the second camera, using the position of the second terminal device detected by the position detection device, and shifts the image so that the eye position detected by the eye detection unit matches the camera position.

5. 5. The remote communication system according to claim 1, wherein the second terminal device is a smartphone or a tablet.

6. The remote communication system according to claim 5 , wherein the second terminal device is bezel-less.

7. A remote communication method for communication between a first user at a first location and a second user at a second location different from the first location, comprising: A step of photographing the first user with a first camera provided at the first location, or photographing the first user existing as an avatar in a virtual reality space with a camera module virtually arranged in the virtual reality space; A step of generating, using an image of the first user captured by the first camera or the camera module, a part of the image of the first user as a first image, and generating, as a second image, an image which is at least a part of the image of the first user and is seamlessly connected to the first image; displaying the first image on a terminal device provided at the second location; displaying the second image on an extension monitor provided on a rear surface of the terminal device; A remote communication method comprising: