Communication terminal device

Abstract

To provide a communication terminal apparatus which realizes satisfactory conversation between users in different remote places.SOLUTION: A communication terminal apparatus according to the present invention acquires a sound including a voice of a user using the own apparatus, acquires visual information allowing three-dimensional data of an object in a predetermined user space to be formed, specifies a view position of an own user based on the visual information of the user space, transmits, to another apparatus, image transmission information based on the visual information of the user space and audio transmission information based on a sound, displays, based on the image transmission information transmitted from the other apparatus, an image of a shared virtual space viewed from the viewpoint of the user on a display screen fixed with a relative position relative to the shared virtual space in which a first virtual space where the user space is fixedly disposed and a second virtual space where a user space of the other apparatus is fixedly disposed are disposed at predetermined relative positions and at relative angles, and outputs sounds based on the audio transmission information transmitted from the other apparatus.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a communication terminal device that enables good interaction between users located remotely.

Background Art

[0002] As means for remote communication, there are web conferences conducted using general-purpose personal computers and the like, video conferences using dedicated devices, and video calls available on smartphones and the like. All of them display a video mainly of the person on the remote side taken from the front on a flat screen and conduct a call by voice. Therefore, it is difficult to obtain a feeling as if actually having a face-to-face conversation in the same space. In contrast, various technologies have been proposed to share a virtual space and improve the quality of communication.

[0003] Patent Document 1 discloses a remote conferencing system in which a plurality of members share the same space. The remote conferencing system of Patent Document 1 is composed of a server device and a plurality of client devices. The client device includes a headset having an acceleration sensor and a display device worn by the user, and a three-dimensional shape acquisition sensor having an infrared sensor, imaging means, and sound collection means. The server device analyzes the user's motion using the depth signal transmitted from the client device, generates CG (computer graphics) based on the motion analysis, displays real object data, synthesizes an augmented reality space, and transmits information on the synthesized augmented reality space to the client device. The client device displays an image on the display device of the headset and outputs an audio signal to the speaker based on the augmented reality space information received from the server device. Patent Document 1 also discloses attaching a face texture acquired in advance to the shape information of the CG.

[0004] Patent Document 2 discloses a technique for sharing a virtual space among a plurality of users and associating the position of each user in the real space with the position in the virtual space. The virtual space sharing system disclosed in Patent Document 2 includes means for detecting position information and visual field information at the respective locations of a plurality of users who are in remote locations, user control means for transmitting the position information and visual field information and receiving the virtual space information transmitted to each user, information processing means for receiving the position information and visual field information of each user, arranging virtual objects representing each user at positions corresponding to the position information of each user within a single virtual space shared by a plurality of users, generating virtual space information for displaying the corresponding virtual space from the viewpoints of each user according to the position information and visual field information of each user, and transmitting the virtual space information to each user, and display means for inputting the virtual space information received by the respective user control means for each user and displaying the virtual space as seen from the viewpoint of each user.

[0005] Patent Document 3 discloses a technique for reducing communication load in a virtual reality space sharing system. The technique disclosed in Patent Document 3 is a virtual reality space sharing system for sharing a virtual reality space between a first terminal used by a first user and a second terminal used by a second user, and includes a line-of-sight information acquisition unit for acquiring line-of-sight information including the position and line-of-sight direction of the second user in the virtual reality space from the second terminal, an image generation unit for generating a virtual reality space image for display on the first terminal based on the line-of-sight information from the second terminal, and an image supply unit for supplying the virtual reality space image generated by the image generation unit to the first terminal. Further, Patent Document 3 discloses that an object such as an avatar is displayed in the virtual reality space instead of the other user, and the direction of the face of the object is made to coincide with the direction of the face of the other user.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] In communication between people face-to-face, such as in communication and building trust relationships, each other's actions, postures, expressions, complexions, gazes, etc. are said to play an important role as non-verbal information transmission means. In communication carried out while sharing a virtual space, the transmission of these non-verbal information is also important.

[0008] However, in the technology of Patent Document 1, the other user appearing in the virtual space shared by multiple users is an image of CG such as an avatar, or an image obtained in advance and pasted on the shape of the other user's face texture. In the technology of Patent Document 2, the other user appearing in the virtual space shared by multiple users is a virtual object. In the technology of Patent Document 3, the other user appearing in the virtual space shared by multiple users is an avatar. Therefore, communication using the technologies of Patent Documents 1 to 3 cannot be said to have sufficient non-verbal information transmission.

[0009] Also, in any of Patent Documents 1 to 3, wearing a head-mounted display on the head is uncomfortable for the user and gives a sense of discomfort different from face-to-face conversation.

[0010] An object of the present invention is to provide a technology for realizing good conversation between users located at remote locations. [Means for Solving the Problems]

[0011] According to one aspect of the present disclosure, a communication terminal device includes: an audio acquisition unit that acquires audio including the voice of the user of the own device; a visual information acquisition unit that acquires visual information capable of forming three-dimensional data of a person on a predetermined user space; a person capture unit that specifies the viewpoint position of the own user based on the visual information of the user space; an information transmission unit that transmits image transmission information based on the visual information of the user space and audio transmission information based on the audio to another device; an image display unit that displays an image of the shared virtual space as seen from the viewpoint position of the own user on a display surface whose relative position is fixed with respect to a shared virtual space in which a first virtual space fixedly arranged in the user space and a second virtual space in which the user space of another device is fixedly arranged are arranged at a predetermined relative position and relative angle, based on the image transmission information transmitted from the other device; and an audio output unit that outputs audio based on the audio transmission information transmitted from the other device.

Advantages of the Invention

[0012] According to one aspect of the present disclosure, it is possible to provide a communication terminal device that enables good interaction between users located at remote locations.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Mode for Carrying Out the Invention

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0015] FIG. 1 is a block diagram of a communication system according to the present embodiment. The communication system 1 includes a plurality of communication terminal devices 10. The communication system 1 is a system that reproduces an experience as if a plurality of users 91 who are far apart are having a face-to-face conversation in the same space. The plurality of communication terminal devices 10 basically have the same configuration and functions and can be interconnected via a communication network 90.

[0016] Here, attention is paid to a certain communication terminal device 10, and that communication terminal device 10 is referred to as the own device, the user 91 who uses that own device is referred to as the own user, a communication terminal device 10 different from the own device is referred to as the other device, and the user 91 who uses the other device may be referred to as the other user. The other device may be one or more, and may be plural.

[0017] FIG. 2 is a block diagram of a communication terminal device according to the present embodiment. The communication terminal device 10 includes an audio acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, and an audio output unit 16.

[0018] In the present embodiment, a predetermined range of the real space where the user 91 who uses the communication terminal device 10 exists is defined as the user space, and it is assumed that the user 91 moves within that user space.

[0019] The voice acquisition unit 11 acquires voices including the voice of the user 91 in the user space. For example, the voice acquisition unit 11 includes a microphone (not shown) and acquires voice information with the microphone.

[0020] The visual information acquisition unit 12 acquires visual information that enables generation of three-dimensional data of an object in the user space. The three-dimensional data may be data representing a three-dimensional object in a three-dimensional space, and the specific method is not particularly limited. The object in the user space may include the user 91 using the communication terminal device 10. The visual information includes, for example, information on an image and depth information. Also, the visual information is acquired at a predetermined frame rate such as 30 to 60 fps (frames per second). The visual information acquisition unit 12 includes an image sensor (not shown) and a depth sensor (not shown), and may acquire an image in real time with the image sensor and acquire depth in real time with the depth sensor.

[0021] The person capture unit 13 specifies the viewpoint position of the user 91 based on the image and / or depth of the user space acquired by the visual information acquisition unit 12. Here, the viewpoint refers to the position where the eyeballs are located. The viewpoint position is a position in a three-dimensional space and can be represented by coordinates in, for example, a three-axis orthogonal coordinate system. For example, the person capture unit 13 detects a person in the user space in real time based on the image and / or depth information, further specifies the head of the detected person, and sets the center of the head as the viewpoint position of the user 91. Note that the viewpoint position may be the position of the viewpoint of the user 91 or a position that can be regarded as the viewpoint of the user 91, and is not limited to the example of this embodiment. The center of the face region recognized by face recognition processing or person recognition processing, or the center of the recognized both eyes may be set as the viewpoint position of the user 91.

[0022] The information transmission unit 14 transmits image transmission information based on the image and depth of the user space acquired by the visual information acquisition unit 12 and voice transmission information based on the voice acquired by the voice acquisition unit 11 to another device.

[0023] The image transmission information is information that enables the reproduction of an image by the image display unit 15 of another device. The image transmission information may include, for example, the image and depth information as acquired by the visual information acquisition unit 12, or the compressed information thereof, or the information with a part of them reduced, or the image and depth information of only the feature points of a person. The feature points are the feature points of the face and body of the user 91, and indicate, for example, the positions of the joints of the arms, the positions of the eyes, the positions of the mouths, etc. From the information of these feature points, the posture, movement, and expression of a person can be reproduced.

[0024] The audio transmission information is information that enables the output of audio by the audio output unit 16 of another device. The audio transmission information may include, for example, the audio information as acquired by the audio acquisition unit 11, or the compressed information thereof, or the information with a part of them reduced, or the processed information thereof.

[0025] The image display unit 15 forms a shared virtual space in which a first virtual space fixedly arranged in the user space of the own device and a second virtual space in which the user space of another device is fixedly arranged are arranged at a predetermined relative position, and on the display surface with a fixed relative position with respect to the virtual space, an image of the shared virtual space visible from the viewpoint position of the own user is displayed based on the image transmission information transmitted from another device. Note that the location where there is a physical entity that performs the process of generating an image to be displayed on the display surface of another device based on the visual information acquired by the own device is not particularly limited. This process may be performed by the own device, or by another device, or by cloud computing between the own device and another device.

[0026] The viewpoint position of the own user is updated in real time by the person capture unit 13, and the image display unit 15 uses the information of the viewpoint position of the own user updated in real time and displays the video of the shared virtual space visible from the real-time viewpoint position on the display surface. As a result, a video with a motion parallax is projected on the display surface.

[0027] The display surface on which the image is displayed is not particularly limited, and includes the display surface for displaying the image of the display device, the display surface on which the image of the projector is projected, and the like. The display surface may be composed of one or more planes. If the display surface is rectangular and flat, instead of representing the display surface by the position in the three-dimensional space for each point included in the display surface, the position of the entire display surface (for example, the position of the center), the posture (for example, the roll angle, the pitch angle, and the yaw angle), and the size (for example, the lengths in the vertical and horizontal directions respectively) can be used to represent it. Further, the display surface may be composed of one or more curved surfaces. For example, if it is a curved surface extending over a predetermined angular range in the circumferential direction of the inner surface of a cylinder, it is uniquely determined by the radius of the cross-section of the cylinder, the height of the cylinder, and the angular range over which the curved surface extends as seen from the center of the circle. Also, as another example, if it is a curved surface extending over a predetermined angular range on the inner surface of a sphere, it is uniquely determined by the radius of the sphere and the angular range (roll angle, pitch angle, and yaw angle) as seen from the center of the sphere. The curved display surface can generally be represented by the position of each point included in the display surface in the three-dimensional space. Furthermore, the display surface may be configured to include one or more curved surfaces and one or more planes.

[0028] As described above, as a process to make the image look convincingly natural so that the other users on the shared virtual space seem to exist there for the self-user, in this embodiment, when generating the data of the three-dimensional object (hereinafter, three-dimensional data) in the shared virtual space based on the image and depth information transmitted from another device and generating the image to be displayed on the display surface, a projective transformation is performed to project the three-dimensional object in the shared virtual space defined in the three-dimensional data onto the display surface, that is, a two-dimensional surface. However, other processing methods that obtain the same effect may be used. As an example of other processing methods, there is a method of using the shared virtual space as a reference coordinate space and projecting each point of the three-dimensional data in the reference coordinate space to the point where the straight line connecting each point and the viewpoint position intersects the display surface. Also, as another processing method for generating the image to be displayed on the display surface, specific matrix or arithmetic operation processing according to empirical rules may be performed on the image or the three-dimensional parameters of the image.

[0029] FIG. 3 is a conceptual diagram for explaining the process of displaying an image. In FIG. 3, a first virtual space 21, a second virtual space 31, and a shared virtual space 41 are shown in a plan view as seen from above. In FIG. 3, for the sake of illustration, the space is shown as a plane, but it also extends in the height direction.

[0030] The image display unit 15 defines a first virtual space 21 in which the user space 22 of the own device is fixedly arranged. In the example of FIG. 3, it is assumed that the visual information acquisition unit 12 has an imager 40 that integrally constitutes an image sensor for acquiring an image and a depth sensor for acquiring depth, and the user space 22 is a space that extends within the angular range of the imager 40. However, the size and shape of the user space 22 are not particularly limited. The size and shape of the first virtual space 21 are not particularly limited. The size of the first virtual space 21 may be infinite. In the user space 22 of the own device, there exists a viewpoint position 24 of the own user 23 specified by the person capture unit 13.

[0031] Also, the image display unit 15 defines a second virtual space 31 in which the user space 32 of another device is fixedly arranged. The size and shape of the second virtual space 31 are also not particularly limited. The size of the second virtual space 31 may be infinite. In the example of FIG. 3, an example is shown in which the second virtual space 31 and the first virtual space 21 have the same size and shape. However, they do not necessarily have to be equal. In the user space 32 of another device, there exists another user 33, and the image and depth of the other user 33 are acquired by the other device, and the image transmission information based on the image and depth information is transmitted to the own device.

[0032] The image display unit 15 defines a shared virtual space 41 in which the first virtual space 21 and the second virtual space 31 are arranged at a predetermined relative position. Here, the shared virtual space 41 is formed by overlapping the first virtual space 21 and the second virtual space 31. In the shared virtual space 41, there will exist the viewpoint position 24 of the own user 23 and the other user 33.

[0033] Furthermore, the image display unit 15 displays an image of the shared virtual space visible from the viewpoint position 24 of the own user 23 on the display surface 42 with a fixed relative position to the shared virtual space 41, based on the image transmission information transmitted from another device.

[0034] FIG. 4 is a diagram showing a state in which an image of the shared virtual space is displayed on the display surface.

[0035] An image of the shared virtual space 41 visible from the viewpoint position 24 of the own user 23 specified based on the image and / or depth acquired by the imager 40 is displayed on the display surface 42. To the own user 23, it appears as if another user 33 is in front of him / her.

[0036] Note that the image display unit 15 may configure not only objects actually existing in the user space of the communication terminal devices 10 connected to each other, but also virtual objects common to the plurality of communication terminal devices 10 in the shared virtual space 41, and display an image thereof on the display surface. For example, when a user 91 holds a certain real object in front of the imager of the visual information acquisition unit 12 in a certain communication terminal device 10, the visual information acquisition unit 12 acquires three-dimensional data of the object, and the image display unit 15 in the plurality of communication terminal devices 10 may be enabled to configure a virtual object based on the three-dimensional data on the shared virtual space 41.

[0037] The voice output unit 16 outputs voice based on the voice transmission information transmitted from another device. The voice output unit 16 includes, for example, a speaker (not shown) and outputs voice through the speaker. At this time, the voice output unit 16 may reproduce the voice based on the voice transmission information transmitted from another device as a voice audible from the direction where the user fixed position of another user in the shared virtual space 41 is arranged, by a stereo method, a surround method, a three-dimensional audio method, or the like. A plurality of people in a remote location can communicate with each other's live-action images and stereo sound in the virtual space.

[0038] FIG. 5 is a diagram showing a state of change of the image displayed on the display surface.

[0039] As described above, the person capture unit 13 identifies the viewpoint position 24 of the self-user 23, and the image display unit 15 displays, on the display surface 42, an image of the shared virtual space as seen from the viewpoint position 24 of the self-user 23 based on the image transmission information transmitted from another device. Therefore, as shown in FIG. 5, even if the other user 33 does not move, if the viewpoint position 24 of the self-user 23 changes, the image displayed on the display surface 42 changes due to motion parallax. For example, if the self-user 23 shifts the viewpoint position 24 from the front to the right direction, it is also possible to peek at the side face of the other user 33 on the upper left diagonal.

[0040] As described above, according to the present embodiment, the communication terminal device 10 shares a virtual space with another communication terminal device 10, and on the display surface 42 whose relative position is fixed in the virtual space, the actual appearance of the other user 33 sharing the virtual space is displayed so that it can be seen from the viewpoint position 24 of the self-user 23. Therefore, in the virtual space shared by a plurality of people in a remote location, good communication can be realized while viewing each other's actual images.

[0041] Note that the processes executed by the voice acquisition unit 11, the visual information acquisition unit 12, the information transmission unit 14, the person capture unit 13, the voice output unit 16, and the image display unit 15 shown in FIG. 2 may be defined by a software program, and a computer having a memory and a processor may execute the software program.

[0042] Hereinafter, more specific examples will be described.

Example

[0043] In the first embodiment, a remote conferencing system that simulates a meeting in which a plurality of users face each other in the same conference room is exemplified. The basic configuration of the remote conferencing system of the first embodiment is the same as that of the communication system 1 according to the embodiment shown in FIGS. 1 to 4.

[0044] FIG. 6 is a block diagram of the communication terminal device according to the first embodiment. The communication terminal device 10 according to the present embodiment includes an audio acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, an audio output unit 16, and an arrangement specification reception unit 17. The audio acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, and the audio output unit 16 are basically the same as those in the embodiment shown in FIG. 2. The arrangement specification reception unit 17 receives, based on a specification from the user 91, the arrangements of the user space 22 of its own device and the user space 32 of another device with respect to the shared virtual space 41, and shares them with the other device. In the present embodiment, the visual information acquisition unit 12 includes three imagers 40A, 40B, and 40C, and the image display unit 15 includes three displays 42A, 42B, and 42C. In the present embodiment, the visual information acquisition unit 12 and the image display unit 15 perform the operations described below based on the arrangements received by the arrangement specification reception unit 17.

[0045] FIG. 7 is a schematic plan view of the communication terminal device according to the first embodiment. FIG. 8 is a schematic perspective view of the communication terminal device of the first embodiment.

[0046] As shown in FIG. 7, in the first embodiment, a predetermined position in the user space 22 is defined as the user-defined position 51, and a predetermined direction from the user-defined position 51 is defined as the user-defined direction 52. For example, the user-defined position 51 is a position where the user 91 using the communication terminal device 10 will mainly be present. For example, the user-defined direction 52 is a direction in which the user 91 using the communication terminal device 10 will mainly be facing.

[0047] As shown in FIG. 8, the displays 42A to 42C are all rectangular and have a flat display surface, and are display devices arranged with the display surface facing the user-defined position 51. The display 42B is arranged in the user-defined direction 52 as viewed from the user-defined position 51. The display 42C is arranged on the right side of the user-defined direction 52 as viewed from the user-defined position 51. The display 42A is arranged on the left side of the user-defined direction 52 as viewed from the user-defined position 51. The displays 42A to 42C are all arranged with the display surface facing the user-defined position 51. The display 42C preferably has an angle of 90° or more and less than 180° between the display surfaces of the display 42C and the display 42B. In this embodiment, the angle is 90°, and it is close to the right side of the display 42B. However, as long as the user can comfortably use the device, it may be arranged at an angle less than 90°. The display 42A preferably has an angle of 90° or more and less than 180° between the display surfaces of the display 42A and the display 42B. In this embodiment, the angle is 90°, and it is close to the left side of the display 42B. However, as long as the user can comfortably use the device, it may be arranged at an angle less than 90°. Also, the displays 42A to 42C are all shown in an example where they are erected at 90° with respect to the horizontal plane, but they may be angled between 0° and 180° with respect to the horizontal plane.

[0048] The image pick-up devices 40A to 40C are arranged such that their relative positions with respect to the user-defined position 51 are fixed, their relative directions with respect to the user-defined direction 52 are fixed, they are isolated from each other, and the imaging direction is directed to the user-defined position 51 so that at least a part of the user space 22 is imaged. Preferably, the image pick-up device 40B is arranged near the center of the upper side of the display surface of the display 42B. The image pick-up device 40A is arranged on the upper side of the display surface of the display 42A or on the left side of the display surface of the display 42A. The image pick-up device 40C is arranged on the upper side of the display surface of the display 42C or on the right side of the display surface of the display 42C. In this embodiment, as shown in FIG. 8, the image pick-up device 40A is arranged on the left side of the display surface of the display 42A, and the image pick-up device 40C is arranged on the right side of the display surface of the display 42C.

[0049] FIG. 9 is a flowchart of the image display process in the first embodiment. The image display process is a process for the self-device to realize a remote conference through mutual communication with other devices, and is mainly executed by the information transmission unit 14, the image display unit 15, and the arrangement specification reception unit 17.

[0050] First, in step S101, the arrangement specification reception unit 17 receives the specification of the arrangements of the self-device and other devices in the shared virtual space 41. Since the user space 22 and the user fixed positions 51 are fixedly defined for the displays 42A to 42C and the imagers 40A to 40C of the self-device, and the user space 32 and the user fixed positions of other devices are fixedly defined for the displays 42A to 42C and the imagers 40A to 40C of other devices, by receiving the arrangements of the self-device and other devices in the shared virtual space 41, the arrangements of the user spaces 22 and 32, the user fixed position 51, and the user fixed positions of other devices in the shared virtual space 41 can be determined.

[0051] At this time, the arrangement specification reception unit 17 may display, for example, an arrangement specification screen in the form of a plan view looking down from above the shared space where the object corresponding to the self-device and the object corresponding to the other device can be movably arranged on the display surface of the display 42B, and receive the specification of the arrangements of the user spaces 22 and 32 in the shared virtual space 41 based on the positions of the specified objects in the plan view.

[0052] FIG. 10 is a diagram showing an example of an arrangement specification screen in Embodiment 1. The arrangement specification screen 60 includes a conference room area 61 and a decision button 63. The conference room area 61 is a plan view simulating the shared virtual space 41 corresponding to a virtual conference room. In the conference room area 61, desk icons 62, which are objects corresponding to the communication terminal devices 10 used by the members participating in the conference, are arranged. The desk icons 62 can be moved or fixed, for example, by mouse operation, touch operation on the display 42B, or gesture operation described later. The shape and size of the conference room area 61 may be changeable. Here, as an example, the conference room area 61 simulates the shared virtual space 41, so the shape and size of the shared virtual space 41 may be determined according to the shape and size of the conference room area 61. When the decision button 63 is pressed, based on the arrangement of the desk icons 62 with respect to the conference room area 61 at that time, the arrangement of the user spaces 22, 32 of each communication terminal device 10, the user fixed positions 51, and the user fixed positions of other devices in the shared virtual space 41 is determined.

[0053] There is a scene of a meeting where multiple people face each other. There is a typical form of a meeting where multiple people face multiple people in negotiations, etc. Various arrangements such as one-on-one and one-on-multiple are possible, but in the example of FIG. 10, an arrangement where six users face each other in a three-on-three configuration is illustrated. Also, here, Mr. A, Mr. B, and Mr. C belonging to Company X and Mr. D, Mr. E, and Mr. F belonging to Company Y are arranged facing each other, but this relationship between affiliation and arrangement is an illustration.

[0054] Also, here, the arrangement specification reception unit 17 shows an example where the user 91 freely moves the desk icon 62 to specify the arrangement, but other examples are also possible. For example, the arrangement specification reception unit 17 may assist in specifying the arrangement, or the arrangement specification reception unit 17 may automatically set or propose the arrangement.

[0055] When the arrangement designation reception unit 17 assists in designating an arrangement, a plurality of typical arrangement pattern may be prepared in advance and the user 91 may be allowed to select one of them. For example, when the number of users who first participate in a meeting is input, the arrangement designation reception unit 17 may present the patterns corresponding to the number to the user and allow the user to select one of them.

[0056] Also, when the arrangement designation reception unit 17 automatically sets or proposes an arrangement, for example, when this meeting is a place for some presentation, the main presenter may be arranged at a position where all the meeting participants can see well (in the example of FIG. 10, the positions of Mr. B of Company X and Mr. E of Company Y). Further, for example, the main presenter may be specified based on the amount of speech in the meeting and the arrangement may be set.

[0057] In this embodiment, it can be realized by arranging the user fixed positions 51 of the plurality of communication terminal devices 10 on each of two straight lines parallel to each other, and setting the user fixed direction 52 of the communication terminal devices 10 on each straight line to the direction perpendicular to the other straight line. Here, the arrangement of the user fixed position 51 on the straight line does not mean strict positioning, and the perpendicular orientation of the user fixed direction 52 to the straight line does not mean strict orientation determination. In either case, it is sufficient that the state where the users of the plurality of communication terminal devices 10 are arranging desks side by side in the meeting can be simulated.

[0058] In step S102, based on the relative angle between the imaging directions of the imagers 40A to 40C of its own device in the shared virtual space and the straight line connecting the user fixed position of the other device and the user fixed position of its own device, the information transmission unit 14 selects an imager so that an image of a visible part when an object in the predetermined range (for example, user space) of the user fixed position of the other device is seen from the predetermined range (for example, user space) of the user fixed position of its own device is acquired, and transmits the image transmission information including the image and depth information acquired by the selected imager to the other device.

[0059] When the user-defined position of another device is arranged on the right side of the user-defined direction 52 from the user-defined position 51 of its own device in the shared virtual space 41, the information transmission unit 14 may transmit image transmission information including the information by the imager 40C on the right side and not including the information by the imager 40A on the left side to the other device. Further, when the user-defined position of another device is arranged on the left side of the user-defined direction 52 from the user-defined position 51 of its own device in the shared virtual space 41, the information transmission unit 14 may transmit image transmission information including the information by the imager 40A on the left side and not including the information by the imager 40C on the right side to the other device.

[0060] For example, focusing on the combination where the own user 23 is Mr. A of Company X and the other user 33 is Mr. F of Company Y, even without the image and depth data of Mr. A of Company X acquired by the imager 40A on the left side of the own device, an image of Mr. A of Company X within the range visible from the viewpoint position of Mr. F of Company Y can be generated in the other device. Therefore, the information transmission unit 14 of the own device used by Mr. A of Company X transmits image transmission information based on the images and depth data acquired by the imagers 14B and 14C to the other device used by Mr. F of Company Y. Further, focusing on the combination where the own user 23 is Mr. A of Company X and the other user 33 is Mr. B of Company X, even without the image and depth data of Mr. A of Company X acquired by the imager 40A on the left side of the own device, an image of Mr. A of Company X within the range visible from the viewpoint position of Mr. B of Company X can be generated in the other device. The information transmission unit 14 of the own device used by Mr. A of Company X transmits image transmission information based on the images and depth data acquired by the imagers 14B and 14C to the other device used by Mr. B of Company X.

[0061] Here, the information transmission unit 14 has been exemplified as performing a process of selecting an image pickup device based on the relative angle between the image pickup direction of the image pickup device among the plurality of image pickup devices 40A to 40C and the straight line connecting the user-defined position of the other device and the user-defined position of its own device. However, other examples are also possible. For example, the information transmission unit 14 may select an image pickup device based on, in addition to the relative angle between the image pickup direction of the image pickup device among the plurality of image pickup devices 40A to 40C and the straight line connecting the user-defined position of the other device and the user-defined position of its own device, the position, angle, and size of the display surface of the other device. In that case, the information transmission unit 14 may select an image pickup device such that an image of the portion visible when an object within a predetermined range (e.g., user space) of the user-defined position of the other device is seen through the display surface of the other device and within a predetermined range (e.g., user space) of the user-defined position of its own device is acquired. Here, an example has been shown in which the information transmission unit 14 of the own user pre-selects the image transmission information to be sent to the communication network 90 and then sends it. However, it is not limited to this method. The information transmission unit 14 may take a method of transmitting the image transmission information obtained from all the image pickup devices at once to a server on the communication network 90 and then performing a process of selecting and deselecting the image transmission information on the server side and sending it to other users. Also, the selection of the transmission data in step S102 is not necessarily performed. In an environment where a sufficient data communication bandwidth can be ensured and reduction of the data amount on the network is unnecessary, this step may be skipped.

[0062] Next, in step S103, the image display unit 15 displays, on the display surfaces of the displays 42A, 42B, 42C, an image of the shared virtual space 41 visible from the viewpoint position 24 of the own user 23 based on the image transmission information transmitted from the other device. Since the user spaces 22, 32 of each user are fixedly arranged in the shared virtual space 41, other users located at positions corresponding to the arrangement can be seen from the viewpoint position 24 of the own user 23.

[0063] FIG. 11 is a conceptual diagram showing a state of a remote meeting in which a meeting where a plurality of users face each other in the same conference room is simulated. FIG. 11 shows an example in which the arrangement shown in FIG. 10 is simulated. The image display unit 15 arranges the user fixed positions 51 of at least one of the own device and the other device on each of a first straight line and a second straight line parallel to each other, and sets the user fixed direction 52 of the device with the user fixed position arranged on the first straight line as a direction perpendicular to the first straight line and toward the second straight line, and sets the user fixed direction 52 of the device with the user fixed position 51 arranged on the second straight line as a direction perpendicular to the second straight line and toward the first straight line, and a shared virtual space may be formed in this way.

[0064] On the front side in FIG. 11, Mr. A, Mr. B, and Mr. C of Company X are lined up horizontally side by side on a straight line with their backs turned. On the back side, Mr. D, Mr. E, and Mr. F of Company Y are lined up horizontally side by side on a straight line facing forward. Mr. A, Mr. B, Mr. C, Mr. D, Mr. E, and Mr. F are actually using the communication terminal devices 10 at different locations. On the displays 42A to 42C of the communication terminal devices 10 of the respective users, images of the shared virtual space 41 as seen from the viewpoint positions of the respective users are displayed in the arrangement shown in FIG. 11. Thereby, all users can obtain the view from each seat when six people are having a meeting with desks arranged in a conference room in the arrangement shown in FIG. 11.

[0065] Next, in step S104, the arrangement designation reception unit 17 determines whether there is a request to change the arrangement of the own device and the other device with respect to the shared virtual space 41. If there is a request to change the arrangement, the arrangement designation reception unit 17 returns to step S101 and receives the arrangement designation.

[0066] The own user or the other user can change the shape and size of the shared virtual space 41 corresponding to the conference room and the arrangement of the user space with respect to the shared virtual space 41 corresponding to the arrangement of each user in the conference room by operating the arrangement designation screen 60 illustrated in FIG. 10.

[0067] FIG. 12 is a conceptual diagram showing a state of a remote conference with another configuration. In the example of FIG. 12, Mr. A, Mr. B, and Mr. C of Company X who are actually using the communication terminal device 10 at different locations are facing each other so as to surround a round table in the shared virtual space 41. The image display unit 15 may form the shared virtual space 41 by arranging the user-defined positions 51 of the own device and other devices on a predetermined circle or polygon, and setting the user-defined directions of the own device and other devices to the directions toward the center of the circle or polygon.

[0068] FIG. 13 is a conceptual diagram showing a state of a remote conference with yet another configuration. In the example of FIG. 13, Mr. A, Mr. B, and Mr. C of Company X who are actually using the communication terminal device 10 at different locations are standing side by side in the shared virtual space 41 and facing an image screen 65 capable of sharing images. The image display unit 15 may form the shared virtual space 41 by arranging the user-defined positions 51 of the own device and other devices on a predetermined straight line, and setting the user-defined directions of the own device and other devices to the same direction perpendicular to the straight line. A plurality of users can stand side by side and watch the video projected onto a huge screen in the virtual space together. For example, it is a video on a personal computer connected to the communication terminal device 10, or a video obtained by streaming and playing back in real time a video acquired at the shooting site. Also, it is possible to define the shared virtual space as a sphere large enough to include all users inside, and paste a streaming-playing or already-shot 360° video inside the sphere to watch together while sharing the entire space. Note that the image screen 65 in FIG. 13 is configured in the shared virtual space 41 as a virtual object common to a plurality of communication terminal devices 10, and the image is displayed on the display surface. In the example of FIG. 13, the image displayed on the image screen 65 is the image displayed on the personal computer 68 of Mr. C.

[0069] If there is no request for changing the arrangement in step S104, in step S105, the image display unit 15 determines whether there is a request to end the remote conference. If there is no request to end the remote conference, the image display unit 15 returns to step S103. If there is a request to end the remote conference, the communication terminal device 10 ends the remote conference.

[0070] According to this embodiment, the virtual spaces of the own device and other devices are superimposed and shared, and on the display surface with a fixed relative position in the virtual space, the real-life images of other users in the virtual space are displayed so as to be visible from the viewpoint position of the own user. Therefore, good communication is realized in which multiple people at remote locations view each other's real-life images while in the shared virtual space.

[0071] Also, according to this embodiment, among the three image pick-up devices 40A to 40C, for only the image pick-up device selected based on the relative angle between the imaging direction and the straight line connecting the user fixed position of the other device and the user fixed position 51 of the own device, the image transmission information based on the visual information acquired by the image pick-up device can be transmitted to the other device. Therefore, the image transmission information transmitted from the own device to the other device can be reduced, and the data amount on the network can also be reduced.

[0072] Also, according to this embodiment, as shown in FIGS. 7 and 8, since the display surface is arranged so as to surround the user fixed position 51, a configuration for displaying the shared virtual space 41 in a wide angular range can be realized with a compact occupied space.

[0073] Also, according to this embodiment, the positional relationship between the own device and other devices can be easily set on the screen with the feeling of moving the conference table in the floor plan.

[0074] Also, in this embodiment, the image display unit 15 may generate three-dimensional data of the shared virtual space 41 that represents the objects in the second virtual space as a point cloud or a polygon mesh based on the image and depth information transmitted from another device, and display, on the display surface, an image of the shared virtual space 41 visible from the viewpoint position 24 of the own user 23 based on the generated three-dimensional data. At this time, the image display unit 15 may enable the points of the point cloud to be processed. For example, the image display unit 15 may make a three-dimensional object in front of the own user in the three-dimensional data appear translucent by reducing the display size of the points representing the three-dimensional object, so that the three-dimensional object behind can also be seen. By making the person displayed by the point cloud appear translucent, it is possible to transmit the information behind the person while leaving the non-verbal information by the person's face or body. For example, by making the hand writing the text appear translucent and making the text hidden by the hand visible when writing, it is possible to transmit the non-verbal information by the movement of the hand and the information of the written text at the same time. Also, for example, by making the person writing the text appear translucent and making the text hidden by the person's figure visible when writing, it is possible to transmit the non-verbal information by the movement of the person and the information of the written text at the same time. FIG. 14 is a conceptual diagram showing a state where a screen with the point cloud processed is displayed on the display surface. FIG. 14 shows a state where the characters on the whiteboard 35 in the virtual space behind the other user 33, which is processed from the point cloud and displayed translucent, can be seen on the display surfaces of the displays 42A, 42B, and 42C. Here, the process of changing the display size of the points of the point cloud is exemplified, but various other processes are also possible. For example, it is also possible to perform a process of changing the color of each point constituting the point cloud, blinking the displayed points, or replacing each point with a three-dimensional object having a specific shape and then drawing. By such processing, for example, it becomes possible to convert to a single-color person maintaining only the outer shell shape, and to perform a person expression suitable for use by a female user without makeup or for the purpose of not specifying the expression or clothing.In addition, in order to reduce the amount of data to be transmitted and avoid realistic representation by actual photographed images, for parts of the human body other than the face, for information representing the skeleton obtained by calculation from the acquired image and / or depth information, by using virtual 3D CG objects to add flesh, it is also possible to perform processing such as representing the posture and movement of a person. Alternatively, it is also possible to selectively display the parts of the user's body to be displayed based on the information representing the skeleton, such as not drawing parts other than the face or not drawing parts other than the face and one hand. In addition, in order to make a certain user stand out, light can be made to surround the user to make them conspicuous, while on the other hand, in order to make others not stand out, appropriate coloring is performed with a color close to the background color corresponding to the viewpoint position for each point so that they blend into the background and are not made conspicuous, that is, it is also possible to implement so-called optical camouflage processing.

[0075] In addition, in this embodiment, an example is shown in which the person capture unit 13 constantly and continuously specifies the viewpoint position 24 of the user 91, and the image display unit 15 tracks the viewpoint position 24 and displays an image of the shared virtual space 41 visible from the viewpoint position 24 on the display surfaces of the displays 42A to 42C. This configuration gives a visual sense of presence to the user 91 due to motion parallax when the user 91 moves. However, in a scene such as a meeting, the movement of the viewpoint position 24 of the user 91 may not be large. Considering such a case, a configuration that does not constantly track the viewpoint position 24 of the user 91 is also possible.

[0076] As another example, once the image display unit 15 has identified the viewpoint position 24 of the user 91 by the person capture unit 13, it may fix the viewpoint position 24 and display an image of the shared virtual space 41 visible from there on the display surfaces of the displays 42A to 42C. For example, in a prior sensing operation, the user 91 may be made to assume a natural posture, during which the person capture unit 13 may identify the viewpoint position 24. Alternatively, the person capture unit 13 may execute the identification of the viewpoint position 24 a plurality of times within a certain period of time, use the average value as the average viewpoint position, and the image display unit 15 may display an image of the shared virtual space 41 visible from the average viewpoint position on the display surfaces of the displays 42A to 42C. Alternatively, the person capture unit 13 may continuously calculate the viewpoint position 24 of the user 91 for a certain period of time, the image display unit 15 may track the viewpoint position 24 and display an image of the shared virtual space 41 visible from the viewpoint position 24 on the display surfaces of the displays 42A to 42C, and during that time, the person capture unit 13 may calculate the average viewpoint position, and the image display unit 15 may converge from the viewpoint position 24 specified by the person capture unit 13 to the average viewpoint position over a predetermined time.

[0077] As yet another example, the image display unit 15 may define a reference viewpoint position in advance and display an image of the shared virtual space 41 visible from the reference specified position on the display surfaces of the displays 42A to 42C. In that case, the process of the person capture unit 13 identifying the viewpoint position 24 of the user 91 becomes unnecessary. As the reference viewpoint position, for example, the viewpoint position when a person of standard build uses the communication terminal device 10 may be used. Alternatively, the viewpoint positions of a plurality of users who have used the communication terminal device 10 may be measured, and the average value may be used as the reference specified position. The viewpoint position fixed by such calculations and definitions may reuse the viewpoint position determined when the device was previously used by reading the information stored in the server on the communication network 90 when each user connects to the communication network 90, or by reading the information stored in the communication terminal device of the user himself / herself.

[0078] Note that the communication terminal device 10 of this embodiment constitutes, for example, a conference booth for one person. A plurality of such conference booths may be arranged in the same room. In addition, the conference booth may be arranged in an individual's residence for telecommuting. In such a case, it is preferable to take noise countermeasures so that the voice of the user 91 does not leak to the surroundings. Therefore, the voice output unit 16 may be capable of outputting voice by means of earphones, a neck speaker, or the like. Further, the voice output unit 16 may include a speaker having directivity toward the user fixed position 51. Furthermore, the communication terminal device 10 may reduce the voice of the user 91 leaking to the surroundings by means of a noise-canceling speaker. Furthermore, the output from the voice output unit 16 may be completely cut off, and the voices from other users may be converted into characters by voice recognition technology and output to the image display unit 15 instead of the output from the voice output unit 16.

[0079] In addition, in this embodiment, an example in which the image display unit 15 includes three displays 42A to 42C has been shown, but other configurations are also possible. For example, in addition to the displays 42A to 42C, the image display unit 15 may further include a display having a display surface on a rectangular upper surface disposed close to the bottom sides of the displays 42A, 42B, and 42C. Further, the image display unit 15 may include a display having a display surface on a rectangular lower surface disposed close to the sides in the ceiling direction of the displays 42A, 42B, and 42C. In any of these cases, the image display unit 15 uses the information on the viewpoint position of the self-user updated in real time, and displays the video of the shared virtual space visible from the real-time viewpoint position on the display surfaces of the four displays. As shown in FIG. 13, in the usage form in which the user fixed positions 51 and the user fixed directions 52 of each device are arranged and a plurality of users watch the screen video together, the video of the shared virtual space can be viewed in a field of view that extends to the lower surface and the upper surface. Further, the communication terminal device 10 has the function of a computer capable of executing application software. For example, the screen of the application software may be displayed on the display surface of a display having a display surface on the upper surface. In that case, the display may be a touch panel type display, and operations of the application software by the user 91 may be accepted. Note that the arrangements of the display having a display surface on the upper surface and the display having a display surface on the lower surface described here may be arranged at an angle from 0° to 180° with respect to the horizontal plane.

[0080] In addition, in this embodiment, an example is shown in which the imager 40A is disposed above or to the left of the display surface of the display 42A, and the imager 40C is disposed above or to the right of the display surface of the display 42C. However, the present configuration is not limited thereto. As another example, the imager 40A may be disposed at a position where the upper side of the display surface of the display 42A and the upper side of the display surface of the display 42B are adjacent to each other, and the imager 40C may be disposed at a position where the upper side of the display surface of the display 42C and the upper side of the display surface of the display 42B are adjacent to each other, or may not be at a position directly above or in the vicinity of the sides of those displays. The purpose of arranging the imagers 40A to 40C is to widely acquire the image and depth information of the self-user 23 so as to obtain an overall image of the self-user, or to acquire specific parts such as the facial expression or palm of the self-user with higher resolution. In accordance with that purpose, the imagers 40A to 40C may be arranged so as to be distant from the user fixed position 51, or may be arranged so as to be close thereto. Further, even if each imager and each display are not directly or indirectly connected via a connector or the like, as long as the relative positions and relative angles thereof are fixed, for example, each display may be fixed to a fixing bracket standing on the floor of a room where the image display unit 15 is fixedly arranged, or a fixing bracket suspended from the ceiling.

[0081] In addition, the communication terminal device 10 of this embodiment has an external connection terminal, and a personal computer may be connected to the external connection terminal and controlled from the personal computer. The communication terminal device 10 outputs control data for receiving control regarding the voice acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, the voice output unit 16, and the arrangement designation reception unit 17 from the external connection terminal, and inputs control information therefor from the external connection terminal. For example, the arrangement designation reception unit 17 outputs the data of the arrangement designation screen 60 shown in FIG. 10 to the personal computer via the external connection terminal, and acquires control information such as moving the desk icon 62 from the personal computer via the external connection terminal, and may receive designation of the arrangement of the user spaces 22 and 32 with respect to the shared virtual space 41 based on the acquired control information. Further, the image display unit 15 may configure the image screen 65 illustrated in FIG. 13 in the shared virtual space 41 and display the image on the display surface based on the control information acquired from the personal computer via the external connection terminal. The external connection terminal is, as an example, a wired LAN terminal or a USB terminal. The control information is directly exchanged between the personal computer and the communication terminal device 10, or is exchanged via a server on the communication network 90. At this time, it may be a wireless connection such as a wireless LAN or Bluetooth even if it is not a physical connection terminal. As another example, it is a terminal conforming to the communication standard of HDMI (High-Definition Multimedia Interface) (HDMI is a registered trademark). The control information is inserted into the HDMI capture information and transmitted. Also, in any case, it may be one-way transmission of a control signal from the personal computer to the communication terminal device 10, that is, control in a situation where control data is not sent from the communication terminal device 10 to the personal computer.

[0082] In addition, the communication terminal device 10 of this embodiment may be controllable by the user 91 through gestures. The communication terminal device 10 stores gesture setting information in which a predetermined operation using the hand (hereinafter referred to as a control operation) is associated in advance with predetermined control related to the voice acquisition unit 11, the visual information acquisition unit 12, the information transmission unit 14, the image display unit 15, the voice output unit 16, and the arrangement designation reception unit 17. When the person capturing unit 13 monitors the movement of the person captured and detects that the person has performed a control operation, the control associated with the control operation in the gesture setting information is instructed to the voice acquisition unit 11, the visual information acquisition unit 12, the information transmission unit 14, the image display unit 15, the voice output unit 16, or the arrangement designation reception unit 17. As an example, an operation of configuring a shared virtual space 41 as a virtual object shared by a plurality of communication terminal devices 10 and moving an object whose image is displayed on a display surface will be described. In the gesture setting information, a first control operation of holding a hand over an object and a first control of setting the object in a selected state are associated with each other, and a second control operation of moving a hand held toward the object and a second control of moving the object within the shared virtual space 41 are associated with each other. Based on the visual information acquired by the visual information acquisition unit 12, the person capture unit 13 detects the first control operation. When the first control operation is detected by the person capture unit 13, the image display unit 15 sets the object in a selected state. At this time, the person capture unit 13 determines that the object selected by the user 41 is an object on a straight line connecting the viewpoint position 24 on the shared virtual space 41 and the center position of the open hand, or on a straight line connecting the joint position of a specific part of the body such as the elbow or shoulder joint and the center position of the hand, or in the vicinity of either of the straight lines, and the image display unit 15 sets the object in a selected state. For example, the fact that the object is selected may be indicated by highlighting the object by, for example, color, light, shadow, or the like. The fact that the first control operation has been detected or the object has been set in a selected state may be notified to other devices from the information transmission unit 14. In that state, when the person capture unit 13 detects the second control operation, the image display unit 15 moves the object on the shared virtual space 41. At this time, the image display unit 15 moves the object when the second control operation is detected after the first control operation is detected, but does not move the object when the second control operation is detected without the first control operation being detected. The fact that the object has moved in the shared virtual space 41 is also transmitted to other devices by the image transmission information, and the object also moves in the display of other devices. Here, the movement of a virtual object has been exemplified, but the object that moves by a gesture is not particularly limited. For example, the image screen 65 shown in FIG. 13 may be moved, or the size, position, and angle may be changed.

[0083] Also, in this embodiment, the communication terminal device 10 is configured to display an image of the shared virtual space 41 as viewed from one viewpoint position 24 on the display surface, but it is not limited to this example. As a modification, the communication terminal device 10 may display an image composed of two images of the shared virtual space 41 with a parallax between the left and right eyes (hereinafter, each is referred to as a left-eye image and a right-eye image). The image display unit 15 includes a display device that displays an image on the display surface and a three-dimensional glasses device worn by the user 91. The image display unit 15 generates the left-eye image and the right-eye image and displays them on the display surface. The three-dimensional glasses device shows the left-eye image to the left eye of the user 91 and the right-eye image to the right eye of the user 91. By giving a parallax to the left and right eyes, a stereoscopic effect can be given to the image, and a higher sense of presence and immersion can be given to the user 91. Note that the method of showing an image with a parallax between the left and right eyes is not particularly limited. For example, it may be an anaglyph type, a polarization type, or a liquid crystal shutter type. Generally, three-dimensional glasses do not give a sense of uneasiness or discomfort like a head-mounted display.

[0084] Note that in this embodiment, an example is shown in which the same shared virtual space 41 is shared between the own device and the other device, but it is not limited to this. As another example, the arrangements of the user space 22 of the own device and the user space 32 of the other device with respect to the shared virtual space 41 may be different between the own device and the other device. For example, considering the arrangement and size of the display of each device, the arrangements may be adjusted so that the expressions and movements of the other user can be easily seen from the own user within a range that does not cause discomfort to the user 91. When using shared virtual spaces 41 with different arrangements between the own device and the other device, all the information on the arrangements used by each device may be shared among all the devices, and each device may select an imager that should transmit visual information to each of the other devices. Alternatively, each device may specify in which direction it wants the visual information captured by each of the other devices, and each device may send the visual information captured from the direction specified by that device to each device.

[0085] Also, in this embodiment, an example where the communication terminal device 10 displays a real-time live image of the face portion has been shown, but it is not limited to this example. As another example, the device on the transmission side may notify the device on the reception side of the mental state of the user 91, and the device on the reception side may process the facial expression of the user to be displayed based on the notified mental state. The information transmission unit 14 of the own device estimates the mental state of the own user from the facial expression of the own user by the person capture unit 13 of the own device. For example, it is estimated whether the own user is in a happy state, an angry state, a sad state, a happy state, a state of agreeing with the conversation content, a state of not agreeing, or a state of not understanding. The information transmission unit 14 of the own device transmits the information on the mental state estimated by the person capture unit 13 to the device on the reception side as image transmission information instead of the image of the face portion. The device on the reception side generates and displays an image of the face portion of the user on the transmission side based on the notified mental state information. According to this, it is possible to visualize qualitative information such as the mental state of the other party, which is difficult to visualize, and it can be an aid for realizing more strategically effective communication. Also, the amount of data transmitted between devices can be reduced.

[0086] Furthermore, as still another example, the device on the transmission side may be able to process the facial expression in the visual information of the user 91 into an expression representing the mental state desired by the user 91 according to the desire of the user 91. When the person capture unit 13 of the own device receives a designation of a desired mental state from the user 91, it processes the facial expression of the user 91 into an expression representing the designated mental state. The information transmission unit 14 transmits the image transmission information in which the facial expression has been processed by the person capture unit 13 to another device. Since the facial expression can be processed according to the desire of the user 91, the user 91 can intentionally use the information conveyed from the facial expression to achieve effective communication.

Embodiment

[0087] In the first embodiment, the communication terminal device 10 selects an imager that acquires an image to be transmitted to another device, and transmits image transmission information including the image acquired by the selected imager and depth information to the other device. Thereby, the data amount of the image transmission information transmitted from the own device to the other device is reduced. In the second embodiment, an example of further reducing the data amount of the image transmission information transmitted from the own device to the other device is shown.

[0088] The configuration of the communication system 1 in the second embodiment is basically the same as that of the first embodiment shown in FIG. 1. Also, the configuration of the communication terminal device 10 in the second embodiment is the same as that of the first embodiment shown in FIGS. 6 to 8. The image display process executed by the communication terminal device 10 in the second embodiment is basically the same as that of the first embodiment shown in FIG. 9. However, in the second embodiment, the specific processes in steps S102 and S103 are different from those in the first embodiment.

[0089] In step S102, in the second embodiment, the person capture unit 13 detects the own user in the user space 22 based on the image and / or depth of the user space 22 acquired by the visual information acquisition unit 12, and the information transmission unit 14 uses the image of the user space 22 acquired by the visual information acquisition unit 12 and / or Generates information representing the face portion of the own user detected from the depth information, and information representing the skeleton of the own user, and transmits these information to another device as image transmission information. Here, the information representing the skeleton is information that enables the identification of the body posture of a person, and may be, for example, information indicating the positions of characteristic parts of the body such as the head, torso, buttocks, shoulders, elbows, hands, knees, and heels. The body posture of a person here is a human body form to the extent that non-verbal information such as poses and movements can be obtained by human eyes and is useful for communication. There is no particular limitation on how detailed and accurate the identification of the body posture is required.

[0090] In step S103, in the second embodiment, the image display unit 15 converts a three-dimensional object in the shared virtual space including a person obtained by synthesizing a face image based on the information representing the face part received from another device and an image of the body posture of the person based on the information representing the skeleton received from another device into a two-dimensional image appropriately according to the viewpoint position of the user and displays it on the display surfaces of the displays 42A to 42C. The image of the body posture of the person based on the information representing the skeleton may be an image obtained by, for example, adding flesh to the information representing the skeleton using a virtual 3DCG object and performing processing to represent the posture and movement of the person.

[0091] According to the present embodiment, since it is not necessary to send all the information captured by the transmitting communication terminal device 10 to the receiving communication terminal device 10, the amount of data transmitted between the communication terminal devices 10 can be reduced. In addition, since parts other than the face of the human body are represented by virtual objects, it is possible to avoid a realistic display of a real image.

Embodiment

[0092] In the first embodiment, as shown in FIGS. 6 to 8, a configuration in which the communication terminal device has three displays each having a rectangular and flat display surface and the display surfaces are arranged facing the user-defined positions was exemplified. On the other hand, in the third embodiment, a configuration in which the communication terminal device 10 has one display having a curved display surface that curves so as to surround the user-defined position is exemplified.

[0093] FIG. 15 is a block diagram of the communication terminal device according to the third embodiment. The communication terminal device 10 according to the present embodiment includes an audio acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, an audio output unit 16, and an arrangement designation reception unit 17. The audio acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, and the audio output unit 16 are basically the same as those in the embodiment shown in FIG. 2. The arrangement designation reception unit 17 receives a designation by the user of the arrangement of the user space 22 of the own device and the user space 32 of another device with respect to the shared virtual space 41.

[0094] In this embodiment, unlike Embodiment 1, the image display unit 15 includes one display 42B.

[0095] FIG. 16 is a schematic plan view of the communication terminal device according to Embodiment 3. FIG. 17 is a schematic perspective view of the communication terminal device of Embodiment 3.

[0096] As shown in FIG. 16, in Embodiment 4, similar to Embodiment 1, a predetermined position in the user space 22 is set as the user-defined position 51, and a predetermined direction from the user-defined position 51 is set as the user-defined direction 52. For example, the user-defined position 51 is a position where the self-user 23 using the communication terminal device 10 is likely to be present. For example, the user-defined direction 52 is a direction in which the self-user 23 using the communication terminal device 10 is likely to be facing.

[0097] As shown in FIG. 17, the display 42B is a display device having a display surface that is curved so as to surround the user-defined position 51. More specifically, the display surface of the display 42B is a curved surface that extends over a predetermined angular range (e.g., 180°) in the circumferential direction of the inner surface of a cylindrical shape. The display 42B may be configured to project an image onto the curved screen using a plurality of projectors, or may be an organic EL display arranged with a curved shape. The center in the left-right direction of the display surface of the display 42B is arranged in the user-defined direction 52 as viewed from the user-defined position 51.

Embodiment

[0098] In Embodiment 1, as shown in FIGS. 6 to 8, an example of the configuration of the communication terminal device having three displays each having a rectangular and flat display surface and arranged with the display surface facing the user-defined position, and three image pick-up devices arranged with the imaging directions facing the user-defined position and isolated from each other in the horizontal direction was illustrated. In contrast, in Embodiment 4, an example of the configuration of the communication terminal device 10 having one display having a rectangular and flat display surface and arranged with the display surface facing the user-defined position, and one image pick-up device arranged with the imaging direction facing the user-defined position is illustrated.

[0099] FIG. 18 is a block diagram of the communication terminal device according to the fourth embodiment. The communication terminal device 10 according to this embodiment includes an audio acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, an audio output unit 16, and an arrangement designation reception unit 17. The audio acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, and the audio output unit 16 are basically the same as those in the embodiment shown in FIG. 2. The arrangement designation reception unit 17 receives a designation by the user of the arrangement of the user space 22 of its own device and the user space 32 of another device in the shared virtual space 41.

[0100] In this embodiment, unlike the first embodiment, the visual information acquisition unit 12 includes one imager 40B, and the image display unit 15 includes one display 42B.

[0101] FIG. 19 is a schematic plan view of the communication terminal device according to the fourth embodiment. FIG. 20 is a schematic perspective view of the communication terminal device of the fourth embodiment.

[0102] As shown in FIG. 19, in the fourth embodiment, as in the first embodiment, a predetermined position in the user space 22 is defined as the user-defined position 51, and a predetermined direction from the user-defined position 51 is defined as the user-defined direction 52. For example, the user-defined position 51 is a position where the own user 23 using the communication terminal device 10 is likely to be present. For example, the user-defined direction 52 is a direction in which the own user 23 using the communication terminal device 10 is likely to be facing.

[0103] As shown in FIG. 20, the display 42B is a rectangular display device having a flat display surface, and the display surface is arranged facing the user-defined position 51. The display 42B is arranged in the user-defined direction 52 as viewed from the user-defined position 51. The imager 40B has a fixed relative position with respect to the user-defined position 51 and a fixed relative direction with respect to the user-defined direction 52, and is arranged with its imaging direction facing the user-defined position 51. Preferably, the imager 40B is arranged near the center of the upper side or the lower side of the display surface of the display 42B.

[0104] FIG. 21 is a flowchart of the image display process in Example 4. The image display process in Example 4 is different from that in Example 1 shown in FIG. 9 in that there is no process in step S102 of selecting transmission data, but other than that, it is basically the same as that in Example 1. As described above, in Example 4, the visual information acquisition unit 12 has one imager 40B, and the information transmission unit 14 transmits image transmission information including the image and depth information acquired by the imager 40B to other devices.

[0105] Note that the communication terminal device 10 in this embodiment can be configured as a device in which the voice acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, and the voice output unit 16 are built into the housing, the imager 40B of the visual information acquisition unit is disposed at a predetermined position of the housing, and an external output terminal for connecting to a display having a flat display surface and outputting the image data by the image display unit 15 to the display is provided on the housing. In this configuration, the image display unit 15 receives the designation of the size of the display surface, the relative position, and the relative posture of the display surface with respect to the housing, and based on the designation, generates the data of the image of the shared virtual space visible from the viewpoint position of the self-user on the display surface and outputs it from the external output terminal. By connecting a general-purpose flat display externally, the communication terminal device 10 that realizes communication sharing the space can be easily configured. Note that in this embodiment, a configuration including only one display 42B is shown, but a configuration including a plurality of external output terminals from the housing and a plurality of displays may be used. In this case, the designation of the size of the display surface, the relative position, and the relative posture of the display surface with respect to the housing received by the image display unit 15 is required for the number of connected displays.

[0106] Also, as described above, since the imaging device 40B included in the visual information acquisition unit 12 of the communication terminal device 10 of this embodiment is single, the three-dimensional data that can be generated from the image transmission information that can be transmitted to other devices is limited. Also, in the configurations of the first and third embodiments represented by the configurations shown in FIGS. 8 and 16, depending on the arrangement of the imaging devices of the visual information acquisition unit 12 to be installed, the three-dimensional data that can be generated may be limited and may not be complete. Therefore, for example, in the own device or other devices, it may be decided to complement the portions not included in the visual information or image transmission information of the user 91 of the communication terminal device 10 of this embodiment. When complementing in the own device, when the person capture unit 13 captures a person from the visual information from the visual information acquisition unit 12, the information transmission unit 14 complements the portions not included in the visual information of the captured person to generate image transmission information. Specifically, the information transmission unit 14 may save in advance the complementary visual information of the side or back of the user 91 acquired by the visual information acquisition unit 12, and complement the side or back portions of the person not included in the visual information acquired by the visual information acquisition unit 12 with the complementary visual information. Alternatively, the person capture unit 13 may complement the side or back portions of the person not included in the visual information acquired by the visual information acquisition unit 12 based on the information of the portions included in the visual information. For example, if the user 91 is wearing black clothes, the visual information of the user 91's back may be created in black. When complementing in other devices, the information transmission unit 14 and / or the server on the communication network 90 complement the portions not included in the image transmission information of the person included in the image transmission information transmitted from the own device (the other device for the other device). Specifically, the information transmission unit 14 may acquire and save in advance the complementary visual information of the side or back of the user 91 from the own device (the other device for the other device), and complement the side or back portions of the person not included in the image transmission information with the complementary visual information. Alternatively, the person capture unit 13 may complement the side or back portions of the person not included in the image transmission information based on the information of the portions included in the image transmission information.

Example

[0107] As shown in FIGS. 7 to 8, Example 1 illustrated a configuration in which the communication terminal device includes a display having a rectangular and flat display surface. In Example 5, a configuration in which the communication terminal device includes a display having a curved display surface will be illustrated.

[0108] FIG. 22 is a block diagram of the communication terminal device according to Example 5. The communication terminal device 10 according to the present embodiment includes an audio acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, an audio output unit 16, and an arrangement designation reception unit 17. The audio acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, the audio output unit 16, and the arrangement designation reception unit 17 are basically the same as those in Example 1 shown in FIG. 6.

[0109] In the present embodiment, the visual information acquisition unit 12 includes three image pick-up devices 40A, 40B, and 40C, and the image display unit 15 includes one display 42D.

[0110] FIG. 23 is a schematic plan view of the communication terminal device according to Example 5. FIG. 24 is a schematic perspective view of the communication terminal device of Example 5.

[0111] As shown in FIGS. 23 and 24, in Example 5, the display 42D of the image display unit 15 has a configuration in which the inner peripheral surface of a vertical cylindrical shape is the display surface. The inside of the cylinder of the display 42D is the user space 22. A predetermined position in the user space 22 is defined as the user-defined position 51. For example, the user-defined position 51 is a position where the self-user 23 using the communication terminal device 10 is likely to be present. Since the display 42D has a cylindrical shape, the user-defined position 51 may be set near its central axis.

[0112] The display 42D is a display device having a cylindrical inner peripheral surface surrounding the user-defined position 51 as the display surface.

[0113] The imaging devices 40A to 40C are each close to the display surface of the display 42D, have a fixed relative position with respect to the user-defined position 51, and are arranged with the imaging direction facing the user-defined position 51. For example, the imaging devices 40A to 40C may be arranged such that the imaging device 40A is in a direction forming an angle of 110° with the imaging device 40B around the central axis of the display 42D, and the imaging device 40C is in a direction forming an angle of -110° with the imaging device 40B around the central axis of the display 42D. Also, the height of the imaging devices 40A to 40C may be, for example, assumed to be the height of a person of standard height as the self-user 23, above the head, at the feet, or at the height of the face.

[0114] The flowchart of the image display process in the fifth embodiment is the same as that of the first embodiment shown in FIG. 9.

[0115] FIG. 25 is a diagram showing an example of the arrangement specification screen in the fifth embodiment. The arrangement specification screen 60 includes a shared space area 64 and a determination button 63. The shared space area 64 is a plan view simulating the shared virtual space 41 corresponding to the virtual space shared by the self-user 23 and the other user 33. In the shared space area 64, booth icons 66, which are objects corresponding to the communication terminal devices 10 used by the members participating in the conversation, are arranged. The booth icons 66 can be moved or fixed, for example, by mouse operations, touch operations on the display surface of the display 42D, or gesture operations detected based on the visual information acquired by the imaging devices 40A to 40C. The shape and size of the shared space area 64 may be changeable. Here, as an example, the shared space area 64 simulates the shared virtual space 41, so the shape and size of the shared virtual space 41 may be determined according to the shape and size of the shared space area 64. When the determination button 63 is pressed, based on the arrangement of the booth icons 66 with respect to the shared space area 64 at that time, the arrangement of the user spaces of the communication terminal devices 10, the user-defined position 51, and the user-defined positions of other devices in the shared virtual space 41 is determined. In the example of FIG. 25, an arrangement in which three users form the vertices of a triangle is illustrated.

[0116] FIG. 26 is a conceptual diagram showing a state in which face-to-face interaction among a plurality of users in a shared space is virtually realized. FIG. 26 shows an example in which the arrangement shown in FIG. 25 is virtually realized. In FIG. 26, Mr. A, Mr. B, and Mr. C of Company X are standing at positions forming the vertices of a triangle and having a conversation. On the display 42D of the communication terminal device 10 of each user, an image of the shared virtual space 41 as seen from the viewpoint position of each user is displayed in the arrangement shown in FIG. 26. Thereby, all users can obtain a view of interacting in the arrangement as shown in FIG. 26.

[0117] According to this embodiment, an image of a virtual space shared with other users is displayed in all directions of 360 degrees as seen from the own user 23, and a high sense of presence can be given to the own user 23. For example, a virtual space composed of CG or a three-dimensional space composed of real-shot video can be used as the shared virtual space, and a state in which a plurality of users are together and having a conversation in the shared virtual space can be reproduced. The real-shot video may be a real-time video or a recorded video taken with a 360-degree camera at a location different from the installation location of any of the communication terminal devices 10.

[0118] Note that, in this embodiment, an example in which the visual information acquisition unit 12 includes three image pick-up devices 40A to 40C is shown, but the number of image pick-up devices included in the visual information acquisition unit 12 is not limited to three. As another example, two may be sufficient, or four or more may be sufficient. Also, the position where the image pick-up device is arranged is not limited to the arrangement described in this embodiment.

Embodiment

[0119] In Embodiment 5, a communication terminal device having a display surface on the inner peripheral surface of a cylindrical shape is exemplified. In contrast, in Embodiment 6, a communication terminal device having a display surface on the outer peripheral surface of a cylindrical shape is exemplified.

[0120] FIG. 27 is a block diagram of the communication terminal device according to Example 6. The communication terminal device 10 according to this example includes an audio acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, an audio output unit 16, and an arrangement specification reception unit 17. The audio acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, the audio output unit 16, and the arrangement specification reception unit 17 are basically the same as those in Example 1 shown in FIG. 6.

[0121] In this example, the visual information acquisition unit 12 includes eight imagers 40A to 40H, and the image display unit 15 includes one display 42E.

[0122] FIG. 28 is a schematic plan view of the communication terminal device according to Example 6. FIG. 29 is a schematic perspective view of the communication terminal device of Example 6.

[0123] As shown in FIGS. 28 and 29, in Example 6, the display 42E of the image display unit 15 has a configuration in which the outer peripheral surface of a vertical cylindrical shape is the display surface.

[0124] The display 42E is a display device having the outer peripheral surface of a cylindrical shape as the display surface. The outside of the cylinder of the display 42E is the user space 22. The imagers 40A to 40H are each close to the display surface of the display 42E, fixed to the outside of the cylinder, and arranged with the imaging direction facing the user fixed position 51. The imagers 40A to 40H may be arranged, for example, in directions forming an angle of 45° with each other around the central axis of the display 42E. Also, the height of the imagers 40A to 40H may be, for example, the height of the head, feet, or face assuming a person of standard height as the self-user 23.

[0125] The communication terminal device 10 of Example 6 is suitable for performing an interaction in combination with the communication terminal device 10 of Example 5. Hereinafter, the user of the communication terminal device 10 of Example 6 will be referred to as user 91(5), and the user of the communication terminal device 10 of Example 5 will be referred to as user 91(4).

[0126] Based on the size of the real space where the communication terminal device 10 of Example 6 is installed, the size of the shared virtual space 41 is set. For example, the shared virtual space 41 in the communication terminal device 10 of Example 6 may be set to infinite. The relative position between the cylindrical central axis of the communication terminal device 10 of Example 6 and the cylindrical central axis of the communication terminal device 10 of Example 5 is made to coincide in the shared virtual space 41. Moreover, for example, a 360° video of the real space where the user 91(5) is located, where the communication terminal device 10 is installed, which is created by synthesizing the image videos obtained from the imagers 40A to 40H in real time, is projected onto the display of the communication terminal device 10 of the user 91(4). As a result, the user 91(4) can obtain a view as if he himself has entered the real space where the user 91(5) is located. The user 91(5) can obtain a view as if the user 91(4) has appeared in the real space where he himself is located. FIG. 30 is a conceptual diagram showing the view obtained from the user of Example 6. In the real indoor space 71 where the user 91(5) actually is, the user 91(4) who is actually remotely located appears, and a state where the user 91(5) and the user 91(4) are directly facing each other and having a conversation is simulated.

[0127] According to this embodiment, the user himself can simultaneously view the real space where he actually is and the shared virtual space displayed by the display surface. Therefore, for the user himself, communication with other users can be achieved with the feeling that other users have appeared at his location. Also, for other users using the communication terminal device 10 of Example 5, communication can be achieved with the feeling of visiting the location where the communication terminal device 10 of Example 6 is installed together with the user himself. For example, a use case such as an inspection of a remote location is assumed.

[0128] Note that in this embodiment, a configuration in which the outer peripheral surface of the cylindrical shape serves as the display surface is illustrated, but the shape of the display surface is not limited to this configuration. For example, a configuration in which the outer surface of a polygonal prism or a spherical shape serves as the display surface may be used, or a configuration in which the outer surface of an oval shape serves as the display surface may be used.

Explanation of Reference Numerals

[0129] 1…Communication system, 10…Communication terminal device, 11…Voice acquisition unit, 12…Visual information acquisition unit, 13…Person capture unit, 14…Information transmission unit, 14C…Imaging device, 14B…Imaging device, 15…Image display unit, 16…Voice output unit, 17…Arrangement designation reception unit, 21…Virtual space, 22…User space, 23…Self-user, 24…Viewpoint position, 31…Virtual space, 32…User space, 33…Other user, 35…Whiteboard, 40…Imaging device, 41…Shared virtual space, 42…Display surface, 42A~42E…Display, 51…User-defined position, 52…User-defined direction, 60…Arrangement designation screen, 61…Conference room area, 62…Desk icon, 63…Decision button, 64…Shared space area, 65…Image screen, 66…Booth icon, 71…Real space, 90…Communication network, 91…User

Claims

1. A plurality of communication terminal devices interconnected via a communication line, a visual information acquisition unit that acquires visual information that enables formation of three-dimensional data of a person in a predetermined user space; an information transmission unit that transmits image transmission information based on the visual information in the user space to another device; an image display unit that displays on a display surface a display image that is generated by projecting three-dimensional data of a person in a shared virtual space in which a user space of the user's own device and a user space of the other device are arranged based on image transmission information transmitted from the other device onto a display surface whose relative position is fixed with respect to the shared virtual space, the display image being as if the person in the shared virtual space were viewed through the display surface from the viewpoint position of the user of the own device; having The image display unit generates an image for a left eye according to a viewpoint position of the left eye, generates an image for a right eye according to a viewpoint position of the right eye, displays the image for the left eye on the display surface so as to be seen by the left eye of the user, and displays the image for the right eye on the display surface so as to be seen by the right eye of the user. Communications terminal equipment.

2. The device further includes a voice acquisition unit and a voice output unit, The voice acquisition unit acquires a voice including a voice of the user, The information transmission unit further transmits voice transmission information based on the voice to the other device, The audio output unit outputs audio based on audio transmission information transmitted from the other device. The communication terminal device according to claim 1.

Images