Communication terminal device

Abstract

To realize good conversation between users located at remote places.SOLUTION: A communication terminal device acquires voice including the voice of a user using the own device, acquires visual information enabling formation of three-dimensional data of an object in a predetermined user space, specifies the viewpoint position of the user on the basis of the visual information of the user space, transmits image transmission information based on the visual information of the user space and voice transmission information based on the voice to another device, displays, on a display surface whose relative position is fixed with respect to a shared virtual space in which a first virtual space with the user space fixedly arranged and a second virtual space with the user space of the other device fixedly arranged are arranged at a predetermined relative position and relative angle, an image of the shared virtual space visible from the viewpoint position of the user on the basis of the image transmission information transmitted from the other device, and outputs voice on the basis of the voice transmission information transmitted from the other device.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a communication terminal device that enables good communication between users in remote locations. [Background technology]

[0002] There are several methods of remote communication, including web conferencing using general-purpose personal computers, video conferencing using dedicated equipment, and video calls available on smartphones. All of these methods display a video of the remote party, primarily taken from the front, on a flat screen, and communicate via audio. As a result, it is difficult to get the feeling that you are actually in the same space and having a face-to-face conversation. In response to this, various technologies have been proposed to improve the quality of communication by sharing virtual spaces.

[0003] Patent Document 1 discloses a teleconferencing system in which multiple members share the same space. The teleconferencing system in Patent Document 1 comprises a server device and multiple client devices. The client device includes a headset worn by a user, which includes an acceleration sensor and a display device, and a 3D shape acquisition sensor, which includes an infrared sensor, an imaging means, and a sound collection means. The server device analyzes the user's movements using depth signals transmitted from the client device, generates CG (computer graphics) based on the movement analysis, displays real object data, synthesizes an augmented reality space, and transmits information about the synthesized augmented reality space to the client device. The client device displays an image on the headset's display device and outputs an audio signal to a speaker based on the augmented reality space information received from the server device. Patent Document 1 also discloses applying a previously acquired facial texture to the CG shape information.

[0004] Patent Document 2 discloses a technology for sharing a virtual space among multiple users and for matching each user's position in the real space with their position in the virtual space. The virtual space sharing system disclosed in Patent Document 2 includes: means for detecting position information and field of view information at the respective locations of multiple users who are in remote locations; user control means for transmitting the position information and field of view information and receiving the virtual space information transmitted to each user; information processing means for receiving each user's position information and field of view information, placing virtual objects representing each user at positions corresponding to each user's position information within a single virtual space shared by the multiple users, generating virtual space information that displays the corresponding virtual space from each user's viewpoint according to each user's position information and field of view information, and transmitting the generated virtual space information to each user; and display means for inputting the received virtual space information into each user control means for each user and displaying the virtual space as seen from each user's viewpoint.

[0005] Patent Document 3 discloses a technology for reducing communication load in a virtual reality space sharing system. The technology disclosed in Patent Document 3 is a virtual reality space sharing system in which a virtual reality space is shared between a first terminal used by a first user and a second terminal used by a second user. The technology includes an image generation device that generates a virtual reality space image to be displayed on the first terminal, the image generation device including: a gaze information acquisition unit that acquires gaze information including the position and gaze direction of the second user in the virtual reality space from the second terminal; an image generation unit that generates a virtual reality space image to be displayed on the first terminal based on the gaze information from the second terminal; and an image supply unit that supplies the virtual reality space image generated by the image generation unit to the first terminal. Patent Document 3 also discloses displaying an object such as an avatar in the virtual reality space in place of a partner user and aligning the facial orientation of the object with the facial orientation of the partner user. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-184986 [Patent Document 2] Japanese Patent Application Laid-Open No. 2002-149581 [Patent Document 3] Japanese Patent Application Publication No. 2017-078891 Summary of the Invention [Problem to be solved by the invention]

[0007] In face-to-face communication between people, it is said that non-verbal communication such as movements, posture, facial expressions, complexion, and gaze plays an important role in communicating with each other and building trust. The transmission of non-verbal information is also important in communication shared in virtual spaces.

[0008] However, in the technology of Patent Document 1, the other user who appears in the virtual space shared by multiple users is a CG image such as an avatar, or an image in which a pre-acquired facial texture is pasted onto the shape of the other user. In the technology of Patent Document 2, the other user who appears in the virtual space shared by multiple users is a virtual object. In the technology of Patent Document 3, the other user who appears in the virtual space shared by multiple users is an avatar. Therefore, it cannot be said that the communication using the technologies of Patent Documents 1 to 3 is sufficient in terms of non-verbal information transmission.

[0009] Furthermore, in all of Patent Documents 1 to 3, wearing a head-mounted display on the head is uncomfortable for the user and gives an unnatural feeling that is different from face-to-face conversation.

[0010] An object of the present invention is to provide a technology that enables good communication between users in remote locations. [Means for solving the problem]

[0011] A communication terminal device according to one aspect of the present disclosure has an audio acquisition unit that acquires audio including the voice of the user using the device, a visual information acquisition unit that acquires visual information that enables the formation of three-dimensional data of a person in a specified user space, a person capture unit that identifies the viewpoint position of the user based on the visual information of the user space, an information transmission unit that transmits image transmission information based on the visual information of the user space and audio transmission information based on audio to other devices, an image display unit that displays an image of the shared virtual space as seen from the viewpoint position of the user based on image transmission information transmitted from the other devices on a display surface that has a fixed relative position with respect to a shared virtual space in which a first virtual space in which the user space is fixedly arranged and a second virtual space in which the user space of the other devices is fixedly arranged are arranged at a specified relative position and relative angle, and an audio output unit that outputs audio based on audio transmission information transmitted from the other devices. [Effects of the Invention]

[0012] According to one aspect of the present disclosure, it is possible to provide a communication terminal device that enables good communication between users in remote locations. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram of a communication system according to an embodiment. [Figure 2] FIG. 2 is a block diagram of a communication terminal device according to an embodiment. [Figure 3] FIG. 10 is a conceptual diagram for explaining a process for displaying an image. [Figure 4] FIG. 10 is a diagram showing an image of a shared virtual space displayed on a display screen. [Figure 5] 10A and 10B are diagrams showing how an image displayed on a display surface changes; [Figure 6] 1 is a block diagram of a communication terminal device according to a first embodiment. [Figure 7] 1 is a schematic plan view of a communication terminal device according to a first embodiment. [Figure 8] 1 is a schematic perspective view of a communication terminal device according to a first embodiment. [Figure 9]10 is a flowchart of an image display process in the first embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a layout designation screen in the first embodiment. [Figure 11] FIG. 1 is a conceptual diagram showing a remote conference simulated as a face-to-face conference held by multiple users in the same conference room. [Figure 12] FIG. 10 is a conceptual diagram showing a remote conference in another configuration. [Figure 13] FIG. 10 is a conceptual diagram showing a remote conference in yet another configuration. [Figure 14] FIG. 10 is a conceptual diagram showing a screen in which a processed point cloud is displayed on a display surface. [Figure 15] FIG. 10 is a block diagram of a communication terminal device according to a third embodiment. [Figure 16] FIG. 10 is a schematic plan view of a communication terminal device according to a third embodiment. [Figure 17] FIG. 11 is a schematic perspective view of a communication terminal device according to a third embodiment. [Figure 18] FIG. 10 is a block diagram of a communication terminal device according to a fourth embodiment. [Figure 19] FIG. 10 is a schematic plan view of a communication terminal device according to a fourth embodiment. [Figure 20] FIG. 10 is a schematic perspective view of a communication terminal device according to a fourth embodiment. [Figure 21] 10 is a flowchart of an image display process in the fourth embodiment. [Figure 22] FIG. 10 is a block diagram of a communication terminal device according to a fifth embodiment. [Figure 23] FIG. 10 is a schematic plan view of a communication terminal device according to a fifth embodiment. [Figure 24] FIG. 10 is a schematic perspective view of a communication terminal device according to a fifth embodiment. [Figure 25] FIG. 13 is a diagram showing an example of a layout designation screen in the fifth embodiment. [Figure 26] FIG. 1 is a conceptual diagram showing a situation in which a plurality of users are engaged in a simulated face-to-face conversation in a shared space. [Figure 27] FIG. 13 is a block diagram of a communication terminal device according to a sixth embodiment. [Figure 28]FIG. 13 is a schematic plan view of a communication terminal device according to a sixth embodiment. [Figure 29] FIG. 13 is a schematic perspective view of a communication terminal device according to a sixth embodiment. [Figure 30] FIG. 20 is a conceptual diagram showing the field of view obtained from the user in the sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0015] 1 is a block diagram of a communication system according to this embodiment. The communication system 1 includes a plurality of communication terminal devices 10. The communication system 1 is a system that recreates the experience of a plurality of remote users 91 conversing face-to-face in the same space. The plurality of communication terminal devices 10 basically have the same configuration and functions, and are connectable to each other via a communication network 90.

[0016] Here, focusing on a certain communication terminal device 10, the communication terminal device 10 may be referred to as the own device, a user 91 using the own device may be referred to as the own user, a communication terminal device 10 different from the own device may be referred to as another device, and a user 91 using the other device may be referred to as another user. There may be one or more other devices, and there may be more than one.

[0017] 2 is a block diagram of a communication terminal device according to this embodiment. The communication terminal device 10 includes a voice acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, and a voice output unit 16.

[0018] In this embodiment, a predetermined range in the real space where a user 91 using the communication terminal device 10 is likely to be present is defined as a user space, and it is assumed that the user 91 moves within this user space.

[0019] The voice acquisition unit 11 acquires voice including the voice of the user 91 in the user space. For example, the voice acquisition unit 11 includes a microphone (not shown) and acquires voice information through the microphone.

[0020] The visual information acquisition unit 12 acquires visual information that enables generation of three-dimensional data of an object in a user space. The three-dimensional data may be data that represents a three-dimensional object in a three-dimensional space, and the specific method is not particularly limited. The object in the user space may include a user 91 using the communication terminal device 10. The visual information includes, for example, image information and depth information. The visual information is acquired at a predetermined frame rate, for example, 30 to 60 fps (frames per second). The visual information acquisition unit 12 may include an image sensor (not shown) and a depth sensor (not shown), and may acquire images in real time using the image sensor and depth in real time using the depth sensor.

[0021] The person capturing unit 13 identifies the viewpoint position of the user 91 based on the image and / or depth of the user space acquired by the visual information acquiring unit 12. Here, the viewpoint refers to the position of the eyeball. The viewpoint position is a position in three-dimensional space and can be expressed, for example, by coordinates in a three-axis Cartesian coordinate system. For example, the person capturing unit 13 detects a person in the user space in real time based on the image and / or depth information, and further identifies the head of the detected person and sets the center of the head as the viewpoint position of the user 91. Note that the viewpoint position may be any position as long as it is the viewpoint position of the user 91 or a position that can be considered to be the viewpoint of the user 91, and is not limited to the example of this embodiment. The viewpoint position of the user 91 may also be the center of a face area recognized by face recognition processing or person recognition processing, or the center of both recognized eyes.

[0022] The information transmission unit 14 transmits image transmission information based on the image and depth of the user space acquired by the visual information acquisition unit 12, and audio transmission information based on the audio acquired by the audio acquisition unit 11 to other devices.

[0023] The image transmission information is information that enables the image display unit 15 of the other device to reproduce an image. The image transmission information may include, for example, image and depth information as acquired by the visual information acquisition unit 12, compressed information thereof, partially deleted information thereof, or image and depth information of only the person's feature points. The feature points are feature points of the user 91's face and body, such as the positions of the arm joints, eyes, and mouth. The person's posture, movements, and facial expressions can be reproduced from the information on these feature points.

[0024] The audio transmission information is information that enables audio output by the audio output unit 16 of the other device. The audio transmission information may include, for example, audio information directly acquired by the audio acquisition unit 11, compressed information thereof, information with a portion of the audio information reduced, or processed information thereof.

[0025] The image display unit 15 forms a shared virtual space in which a first virtual space in which the user space of the own device is fixedly arranged and a second virtual space in which the user space of the other device is fixedly arranged are arranged at predetermined relative positions, and displays an image of the shared virtual space as seen from the viewpoint of the own user on a display surface whose relative position is fixed with respect to the virtual space, based on image transmission information transmitted from the other device. Note that the location of the physical entity that performs the process of generating the image to be displayed on the display surface of the other device based on visual information acquired by the own device is not particularly limited. The process may be performed on the own device, on the other device, or by cloud computing between the own device and the other device.

[0026] The user's viewpoint position is updated in real time by the person capture unit 13, and the image display unit 15 uses the information on the user's viewpoint position updated in real time to display an image of the shared virtual space seen from the real-time viewpoint position on the display surface. As a result, an image with motion parallax is displayed on the display surface.

[0027] The display surface on which an image is displayed is not particularly limited and includes a display surface that displays an image from a display device and a display surface onto which an image from a projector is projected. The display surface may be composed of one or more flat surfaces. If the display surface is rectangular and flat, the display surface can be represented by the position (e.g., the center position), attitude (e.g., roll angle, pitch angle, and yaw angle), and size (e.g., vertical and horizontal lengths) of the entire display surface, rather than by the position of each point included in the display surface in three-dimensional space. The display surface may also be composed of one or more curved surfaces. For example, a curved surface that spans a predetermined angular range in the circumferential direction of the inner surface of a cylinder is uniquely determined by the radius of the cross section of the cylinder, the height of the cylinder, and the angular range of the curved surface as viewed from the center of the circle. As another example, a curved surface that spans a predetermined angular range of the inner surface of a sphere is uniquely determined by the radius of the sphere and the angular range (roll angle, pitch angle, and yaw angle) as viewed from the center of the sphere. A curved display surface can generally be represented by the position in three-dimensional space of each point included in the display surface. Furthermore, the display surface may be configured to include one or more curved surfaces and one or more flat surfaces.

[0028] In this embodiment, as a process for creating a realistic and natural-looking image that creates the illusion that other users in the shared virtual space are actually present in the shared virtual space, data on three-dimensional objects in the shared virtual space (hereinafter, "three-dimensional data") is generated based on image and depth information transmitted from other devices. When generating an image to be displayed on the display screen, a projective transformation is performed to project the three-dimensional objects in the shared virtual space defined in the three-dimensional data onto the display screen, i.e., a two-dimensional surface. However, other processing methods that achieve a similar effect may also be used. An example of another processing method is to use the shared virtual space as a reference coordinate space and project each point of the three-dimensional data in that reference coordinate space onto the point where a line connecting each point to the viewpoint intersects with the display screen. Another processing method for generating an image to be displayed on the display screen may involve performing arithmetic operations on specific matrices or numbers based on empirical rules on the image and its three-dimensional parameters.

[0029] Fig. 3 is a conceptual diagram for explaining the process of displaying an image. Fig. 3 shows a plan view from above of first virtual space 21, second virtual space 31, and shared virtual space 41. For convenience of illustration, Fig. 3 shows the spaces in a plane, but they also extend in the vertical direction.

[0030] The image display unit 15 defines a first virtual space 21 in which the user space 22 of the own device is fixedly arranged. In the example of FIG. 3, the visual information acquisition unit 12 has an image capture device 40 that integrates an image sensor for acquiring images and a depth sensor for acquiring depth, and the user space 22 is a space that extends within the angle of view range of the image capture device 40. However, the size and shape of the user space 22 are not particularly limited. The size and shape of the first virtual space 21 are not particularly limited. The size of the first virtual space 21 may be infinite. In the user space 22 of the own device, there is a viewpoint position 24 of the own user 23 identified by the person capture unit 13.

[0031] Furthermore, the image display unit 15 defines a second virtual space 31 in which a user space 32 of the other device is fixedly arranged. The size and shape of the second virtual space 31 are not particularly limited. The size of the second virtual space 31 may be infinite. In the example of FIG. 3, the second virtual space 31 and the first virtual space 21 are shown to be equal in size and shape. However, they do not necessarily have to be equal. Another user 33 exists in the user space 32 of the other device, and the image and depth of the other user 33 are acquired by the other device, and image transmission information based on the image and depth information is transmitted to the own device.

[0032] Image display unit 15 defines shared virtual space 41 in which first virtual space 21 and second virtual space 31 are arranged at a predetermined relative position. Here, shared virtual space 41 is formed by overlapping first virtual space 21 and second virtual space 31. In shared virtual space 41, viewpoint position 24 of user 23 and other users 33 exist.

[0033] Furthermore, the image display unit 15 displays an image of the shared virtual space as seen from the viewpoint position 24 of the user 23 on a display surface 42 whose relative position with respect to the shared virtual space 41 is fixed, based on image transmission information transmitted from another device.

[0034] FIG. 4 is a diagram showing an image of a shared virtual space displayed on a display surface.

[0035] An image of the shared virtual space 41 seen from the viewpoint position 24 of the user 23, which is identified based on the image and / or depth acquired by the image capture device 40, is displayed on the display surface 42. From the user 23's perspective, it appears as if the other user 33 is in front of them.

[0036] The image display unit 15 may be capable of constructing not only objects that actually exist in the user spaces of the mutually connected communication terminal devices 10 but also virtual objects that are common to the multiple communication terminal devices 10 in the shared virtual space 41 and displaying the images on the display surface. For example, when a user 91 of a certain communication terminal device 10 holds a certain real object over the image sensor of the visual information acquisition unit 12, the visual information acquisition unit 12 may acquire three-dimensional data of the object, and the image display units 15 of the multiple communication terminal devices 10 may be capable of constructing virtual objects based on the three-dimensional data in the shared virtual space 41.

[0037] The audio output unit 16 outputs audio based on audio transmission information transmitted from another device. The audio output unit 16 may include, for example, a speaker (not shown) and output audio through the speaker. In this case, the audio output unit 16 may reproduce audio based on the audio transmission information transmitted from another device as audio heard from the direction of the other user's fixed position in the shared virtual space 41 using a stereo system, a surround system, a three-dimensional sound system, or the like. Multiple people in remote locations can communicate with each other in the virtual space using real-life images and stereophonic sound.

[0038] FIG. 5 is a diagram showing how the image displayed on the display surface changes.

[0039] As described above, the person capturing unit 13 identifies the viewpoint position 24 of the user 23, and the image display unit 15 displays an image of the shared virtual space seen from the viewpoint position 24 of the user 23 on the display surface 42 based on the image transmission information transmitted from the other device. Therefore, as shown in Fig. 5, even if the other user 33 does not move, if the viewpoint position 24 of the user 23 changes, the image displayed on the display surface 42 will change due to motion parallax. For example, if the user 23 shifts the viewpoint position 24 from directly in front to the right, the user 23 can peer into the profile of the other user 33 diagonally to the left.

[0040] As described above, according to this embodiment, the communication terminal device 10 shares a virtual space with other communication terminal devices 10, and displays live-action images of other users 33 sharing the virtual space on the display surface 42, which has a fixed relative position in the virtual space, so that they can be seen from the viewpoint position 24 of the user 23, thereby realizing good communication in a virtual space shared by multiple people in remote locations while viewing each other's live-action images.

[0041] In addition, the processes performed by the voice acquisition unit 11, visual information acquisition unit 12, information transmission unit 14, person capture unit 13, voice output unit 16, and image display unit 15 shown in Figure 2 may be defined by a software program, and a computer having a memory and a processor may execute the software program.

[0042] A more specific example will be described below. [Example]

[0043] In Example 1, a remote conference system that simulates a face-to-face conference held by multiple users in the same conference room is illustrated. The remote conference system of Example 1 has the same basic configuration as the communication system 1 according to the embodiment shown in FIGS.

[0044] FIG. 6 is a block diagram of a communication terminal device according to a first embodiment. The communication terminal device 10 according to this embodiment includes a voice acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, a voice output unit 16, and an arrangement designation receiving unit 17. The voice acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, and the voice output unit 16 are basically the same as those in the embodiment shown in FIG. 2. The arrangement designation receiving unit 17 receives a designation from a user 91 regarding the arrangement of the user space 22 of the own device and the user space 32 of the other device relative to the shared virtual space 41, and shares the designation with the other device. In this embodiment, the visual information acquisition unit 12 includes three image capture devices 40A, 40B, and 40C, and the image display unit 15 includes three displays 42A, 42B, and 42C. In this embodiment, the visual information acquisition unit 12 and the image display unit 15 perform the operations described below based on the layout accepted by the layout designation accepting unit 17 .

[0045] Fig. 7 is a schematic plan view of the communication terminal device according to the first embodiment. Fig. 8 is a schematic perspective view of the communication terminal device according to the first embodiment.

[0046] 7, in the first embodiment, a predetermined position in the user space 22 is defined as a user home position 51, and a predetermined direction from the user home position 51 is defined as a user home direction 52. For example, the user home position 51 is a position where a user 91 using the communication terminal device 10 is likely to be mainly present. For example, the user home direction 52 is a direction in which the user 91 using the communication terminal device 10 is likely to be mainly facing.

[0047] As shown in FIG. 8, displays 42A to 42C are all display devices that have a rectangular, flat display surface and are arranged with their display surfaces facing user fixed position 51. Display 42B is arranged in user fixed direction 52 as viewed from user fixed position 51. Display 42C is arranged on the right side of user fixed direction 52 as viewed from user fixed position 51. Display 42A is arranged on the left side of user fixed direction 52 as viewed from user fixed position 51. All of displays 42A to 42C are arranged with their display surfaces facing user fixed position 51. It is preferable that the angle between the display surfaces of display 42C and display 42B is 90° or more and less than 180°; in this embodiment, the angle is 90° and display 42C is close to the right side of display 42B, but display 42C may be arranged at an angle less than 90° as long as the user can use the device comfortably. The display surfaces of display 42A and display 42B are preferably at an angle of 90° or more and less than 180°, and in this embodiment are at 90° and are adjacent to the left side of display 42B, but may be disposed at an angle less than 90° as long as the user can use the device comfortably. Also, although the example shown shows all of displays 42A to 42C standing at an angle of 90° relative to the horizontal plane, they may be angled at an angle between 0° and 180° relative to the horizontal plane.

[0048] The imagers 40A to 40C are arranged such that their relative positions with respect to the user fixed position 51 and their relative directions with respect to the user fixed direction 52 are fixed, and they are spaced apart from one another, with their imaging directions directed toward the user fixed position 51 and capturing images of at least a portion of the user space 22. Preferably, the imager 40B is arranged near the center of the upper edge of the display surface of the display 42B. The imager 40A is arranged on the upper edge of the display surface of the display 42A or on the left edge of the display surface of the display 42A. The imager 40C is arranged on the upper edge of the display surface of the display 42C or on the right edge of the display surface of the display 42C. In this embodiment, as shown in FIG. 8, the imager 40A is arranged on the left edge of the display surface of the display 42A, and the imager 40C is arranged on the right edge of the display surface of the display 42C.

[0049] 9 is a flowchart of the image display process in Example 1. The image display process is a process in which the device itself realizes a remote conference through mutual communication with other devices, and is mainly executed by the information transmission unit 14, the image display unit 15, and the placement designation receiving unit 17.

[0050] First, in step S101, placement designation receiving unit 17 receives a designation for the placement of the user's device and other devices relative to shared virtual space 41. User space 22 and user home position 51 are fixedly defined for displays 42A-42C and image capture devices 40A-40C of the user's device, and user space 32 and user home positions of other devices are fixedly defined for displays 42A-42C and image capture devices 40A-40C of the other devices. Therefore, by receiving the placement of the user's device and other devices relative to shared virtual space 41, the placement of user spaces 22, 32, user home position 51, and user home positions of other devices relative to shared virtual space 41 can be determined.

[0051] At this time, the placement specification receiving unit 17 may, for example, display a placement specification screen on the display surface of the display 42B, which is a plan view of a shared space in which objects corresponding to the own device and objects corresponding to other devices can be movably placed, as seen from above, and may receive specifications for the placement of the user spaces 22, 32 relative to the shared virtual space 41 based on the positions of the objects specified on the plan view.

[0052] FIG. 10 is a diagram illustrating an example of an arrangement designation screen in the first embodiment. The arrangement designation screen 60 includes a conference room area 61 and a confirm button 63. The conference room area 61 is a plan view simulating the shared virtual space 41, which corresponds to a virtual conference room. In the conference room area 61, desk icons 62, which are objects corresponding to the communication terminal devices 10 used by each member participating in the conference, are arranged. The desk icons 62 can be moved or fixed, for example, by mouse operation, touch operation on the display 42B, or gesture operation (described later). The shape and size of the conference room area 61 may be changeable. Here, as an example, the conference room area 61 is assumed to resemble the shared virtual space 41. Therefore, the shape and size of the shared virtual space 41 may be determined according to the shape and size of the conference room area 61. When the confirm button 63 is pressed, the arrangement of the user spaces 22 and 32 of each communication terminal device 10, the user fixed position 51, and the user fixed positions of other devices in the shared virtual space 41 is determined based on the arrangement of the desk icons 62 relative to the conference room area 61 at that time.

[0053] There are situations where multiple people meet face to face. A typical meeting format is one where multiple people meet face to face, such as in negotiations. Various layouts are possible, such as one-to-one or one-to-multiple, but the example in Figure 10 illustrates a layout where six users are arranged in a three-to-three arrangement. Also, here, Persons A, B, and C who belong to Company X are arranged facing each other, and Persons D, E, and F who belong to Company Y, but this relationship between affiliation and layout is merely an example.

[0054] Although the example shown here is one in which the user 91 freely moves the desk icon 62 to specify the layout, other examples are also possible. For example, the layout designation receiving unit 17 may assist in the designation of the layout, or may automatically set or suggest the layout.

[0055] When the placement designation receiving unit 17 assists in the designation of the placement, a plurality of typical placement configuration patterns may be prepared in advance, and the user 91 may select one of them. For example, when the number of users participating in the conference is first input, the placement designation receiving unit 17 may present the user with patterns corresponding to that number of users, and allow the user to select one of them.

[0056] Furthermore, when the placement designation receiving unit 17 automatically sets or proposes placement, for example, if the conference is a venue for some kind of presentation, the main presenter may be placed in a position that is easily visible to all conference participants (in the example of FIG. 10, the positions of Mr. B from Company X and Mr. E from Company Y). Furthermore, for example, the main presenter may be identified based on the amount of speech in the conference, and the placement may be set.

[0057] In this embodiment, this can be achieved by arranging the user home positions 51 of multiple communication terminal devices 10 on each of two parallel lines, and setting the user home directions 52 of the communication terminal devices 10 on each line perpendicular to the other line. Note that arranging the user home positions 51 on a line does not mean strict positioning, and directing the user home directions 52 perpendicular to the line does not mean strict orientation. In either case, it is sufficient to be able to simulate a situation where multiple communication terminal devices 10 are seated side by side at desks in a meeting.

[0058] In step S102, the information transmission unit 14 selects an imager based on the imaging direction of the imagers 40A to 40C of the device itself in the shared virtual space and the relative angle between the user fixed position of the other device and the straight line connecting the user fixed position of the device itself, so that an image of the visible portion of an object within a specified range of the user fixed position of the device itself (e.g., user space) is acquired when viewed from within a specified range of the user fixed position of the other device (e.g., user space), and transmits image transmission information including the image acquired by the selected imager and depth information to the other device.

[0059] When the user home position of the other device is located to the right of the user home position 51 of the user's own device in the user home direction 52 in the shared virtual space 41, the information transmission unit 14 may transmit to the other device image transmission information that includes information from the imager 40C on the right side and does not include information from the imager 40A on the left side. When the user home position of the other device is located to the left of the user home position 51 of the user's own device in the user home direction 52 in the shared virtual space 41, the information transmission unit 14 may transmit to the other device image transmission information that includes information from the imager 40A on the left side and does not include information from the imager 40C on the right side.

[0060] For example, in a case where the user 23 is a person A of company X and the other user 33 is a person F of company Y, an image of the person A of company X within a range visible from the viewpoint of the person F of company Y can be generated on the other device, even if the image and depth data of the person A of company X acquired by the imager 40A on the left side of the user's device is not available. Therefore, the information transmission unit 14 of the user's device used by the person A of company X transmits image transmission information based on the image and depth data acquired by the imager 14B and the imager 14C to the other device used by the person F of company Y. In addition, in a case where the user 23 is a person A of company X and the other user 33 is a person B of company X, an image of the person A of company X within a range visible from the viewpoint of the person B of company X can be generated on the other device, even if the image and depth data of the person A of company X acquired by the imager 40A on the left side of the user's device is not available. The information transmission unit 14 of the device used by person A of company X transmits image transmission information based on the image and depth data acquired by image capture device 14B and image capture device 14C to another device used by person B of company X.

[0061] Here, the information transmission unit 14 selects one of the multiple imagers 40A-40C based on the imaging direction of that imager and the relative angle between the imaging direction and the line connecting the user's fixed position of the other device and the user's fixed position of the user's own device. However, other examples are possible. For example, the information transmission unit 14 may select one of the multiple imagers 40A-40C based on the relative angle between the imaging direction of that imager and the line connecting the user's fixed position of the other device and the user's fixed position of the user's own device, as well as the position, angle, and size of the display surface of the other device. In this case, the information transmission unit 14 may select an imager so that an image of a portion of an object within a predetermined range of the user's fixed position (e.g., user space) of the other device is acquired when viewed from within a predetermined range of the user's fixed position of the other device (e.g., user space) through the display surface of the other device. Here, the information transmission unit 14 of the user selects image transmission information to be sent to the communication network 90 in advance and then sends it. However, this method is not limited to this. The information transmission unit 14 may transmit the image transmission information obtained from all the image capture devices to a server on the communication network 90, after which the server performs a process of selecting and discarding the image transmission information before sending it to other users. Also, the selection of transmission data in step S102 is not always necessary, and this step may be skipped in an environment where a sufficient data communication bandwidth is available and there is no need to reduce the amount of data on the network.

[0062] Next, in step S103, the image display unit 15 displays an image of the shared virtual space 41 visible from the viewpoint position 24 of the user 23 on the display surfaces of the displays 42A, 42B, 42C based on the image transmission information transmitted from the other devices. Since the user spaces 22, 32 of each user are fixedly arranged in the shared virtual space 41, from the viewpoint position 24 of the user 23, other users who are located in positions corresponding to that arrangement can be seen.

[0063] Fig. 11 is a conceptual diagram showing a simulated remote conference in which multiple users meet in person in the same conference room. Fig. 11 shows an example in which the arrangement shown in Fig. 10 is simulated. The image display unit 15 may form a shared virtual space by arranging the user home positions 51 of at least one of the local device and the other devices on a first line and a second line that are parallel to each other, and by setting the user home direction 52 of the device whose user home position is located on the first line to a direction perpendicular to the first line toward the second line, and by setting the user home direction 52 of the device whose user home position 51 is located on the second line to a direction perpendicular to the second line toward the first line.

[0064] In the foreground of FIG. 11, A, B, and C from company X are lined up side by side in a straight line, facing the distance. In the background, D, E, and F from company Y are lined up side by side in a straight line, facing the distance. In reality, A, B, C, D, E, and F are using communication terminal devices 10 in different locations. Images of shared virtual space 41 seen from each user's viewpoint are displayed on displays 42A to 42C of each user's communication terminal device 10 in the arrangement shown in FIG. 11. This allows all users to see from their respective seats as if six people were having a meeting in a conference room, with their desks lined up in the arrangement shown in FIG. 11.

[0065] Next, in step S104, placement designation receiving unit 17 determines whether or not there is a request to change the placement of the own device and other devices in shared virtual space 41. If there is a request to change the placement, placement designation receiving unit 17 returns to step S101 and receives a placement designation.

[0066] By operating the layout specification screen 60 illustrated in Figure 10, the user or other users can change the shape and size of the shared virtual space 41, which corresponds to the conference room, and the layout of the user space relative to the shared virtual space 41, which corresponds to the layout of each user in the conference room.

[0067] 12 is a conceptual diagram showing a remote conference using another configuration. In the example of FIG. 12, Persons A, B, and C from Company X, who are actually using communication terminal devices 10 in different locations, are seated face-to-face around a round table in shared virtual space 41. Image display unit 15 can form shared virtual space 41 by arranging user designated positions 51 of the user's device and other devices on a predetermined circle or polygon, and aligning the user designated directions of the user's device and other devices toward the center of the circle or polygon.

[0068] FIG. 13 is a conceptual diagram illustrating a remote conference using yet another configuration. In the example of FIG. 13, Persons A, B, and C from Company X, who are actually using communication terminal devices 10 in different locations, are side-by-side in a shared virtual space 41, facing an image screen 65 on which images can be shared. The image display unit 15 arranges the user fixed positions 51 of the user's own device and the other devices on a predetermined straight line, and forms the shared virtual space 41 so that the user fixed directions of the user's own device and the other devices are perpendicular to the straight line. Multiple users can sit side-by-side and view images projected on a giant screen in the virtual space together. For example, images on a personal computer connected to the communication terminal device 10 or real-time streaming playback of video captured on a filming location can be displayed. Alternatively, the shared virtual space can be defined as a sphere large enough to encompass all users, and streaming or recorded 360-degree video can be attached to the inside of the sphere, allowing users to view the entire space together. 13 is configured in the shared virtual space 41 as a virtual object common to a plurality of communication terminal devices 10, and the image is displayed on the display screen. In the example of FIG. 13, the image displayed on person C's personal computer 68 is displayed on the image screen 65.

[0069] If there is no request to change the layout in step S104, then in step S105 the image display unit 15 determines whether there is a request to end the remote conference. If there is no request to end the remote conference, the image display unit 15 returns to step S103. If there is a request to end the remote conference, the communication terminal device 10 ends the remote conference.

[0070] According to this embodiment, the virtual spaces of the user's own device and other devices are overlapped and shared, and the live-action images of other users in the virtual space are displayed on a display surface whose relative position is fixed in the virtual space so that they can be seen from the user's viewpoint, thereby realizing good communication between multiple people in remote locations while viewing each other's live-action images in a virtual space shared by them.

[0071] Furthermore, according to this embodiment, it is possible to transmit image transmission information based on visual information acquired by only one of the three image sensors 40A to 40C selected based on the imaging direction and the relative angle between the imaging direction and the straight line connecting the user's fixed position of the other device and the user's fixed position 51 of the own device to the other device, thereby reducing the image transmission information transmitted from the own device to the other device and also reducing the amount of data on the network.

[0072] Furthermore, according to this embodiment, as shown in Figures 7 and 8, the display surface is arranged to surround the user's fixed position 51, so that a configuration that displays the shared virtual space 41 over a wide angle range can be realized in a compact occupied space.

[0073] Furthermore, according to this embodiment, the positional relationship between the own device and other devices can be easily set on the screen in the same way as moving a desk in a conference room on a floor plan.

[0074] In this embodiment, the image display unit 15 may generate three-dimensional data of the shared virtual space 41, representing objects in the second virtual space as a point cloud or polygon mesh, based on image and depth information transmitted from another device, and display an image of the shared virtual space 41 as seen from the viewpoint position 24 of the user 23 on the display surface based on the generated three-dimensional data. In this case, the image display unit 15 may be able to process the points of the point cloud. For example, the image display unit 15 may reduce the display size of points representing three-dimensional objects in front of the user 23 in the three-dimensional data, thereby displaying the three-dimensional objects semi-transparently and allowing the three-dimensional objects behind them to be seen. By processing a person displayed as a point cloud semi-transparently, it is possible to convey information about the person's background while retaining non-verbal information from the person's face or body. For example, by making a hand writing text semi-transparent and revealing the text obscured by the hand while writing, it is possible to simultaneously convey non-verbal information from the hand's movements and information about the written text. Furthermore, for example, by making a person writing text semi-transparent and revealing the text hidden by the person's figure, non-verbal information from the person's movements and the information from the written text can be simultaneously conveyed. Figure 14 is a conceptual diagram showing a screen in which a processed point cloud is displayed on a display screen. Figure 14 shows the display screens of displays 42A, 42B, and 42C, in which text on a whiteboard 35 in a virtual space is visible in the background of another user 33, which is displayed semi-transparently after the point cloud has been processed. While the example shown here illustrates a process that changes the display size of the points in the point cloud, various other processes are also possible. For example, it is also possible to change the color of each point constituting the point cloud, blur the displayed points, or replace each point with a three-dimensional object with a specific shape before drawing it. Such processes, for example, can convert a person into a monochromatic figure that retains only the outer shell shape, enabling the representation of a person tailored to a female user without makeup or for purposes in which the facial expression and clothing are not specified.To reduce the amount of data transmitted and avoid the need for realistic representations using live-action images, it is possible to use virtual 3DCG objects to represent the posture and movement of a person, based on the skeletal information calculated from the acquired images and / or depth information. Alternatively, it is possible to selectively display the user's body parts based on the skeletal information, such as only drawing the face or the face and one hand. Additionally, it is possible to implement a process known as optical camouflage, where a certain user is highlighted by surrounding them with light to draw attention to them, or to blend in with the background by appropriately coloring each point with a color close to the background color according to the viewpoint to avoid attention to users other than the user.

[0075] In addition, in this embodiment, an example has been shown in which the person capturing unit 13 constantly and continuously identifies the viewpoint position 24 of the user 91, and the image display unit 15 tracks the viewpoint position 24 and displays an image of the shared virtual space 41 seen from the viewpoint position 24 on the display surfaces of the displays 42A to 42C. This configuration gives the user 91 a sense of visual presence through motion parallax when the user 91 moves. However, in situations such as meetings, the movement of the viewpoint position 24 of the user 91 may not be large. Considering such cases, a configuration that does not constantly track the viewpoint position 24 of the user 91 is also possible.

[0076] As another example, once the person capturing unit 13 has identified the viewpoint position 24 of the user 91, the image display unit 15 may fix the viewpoint position 24 and display an image of the shared virtual space 41 seen from that position on the display surfaces of the displays 42A to 42C. For example, in a preliminary sensing operation, the user 91 may be made to assume a natural posture, and during that time the person capturing unit 13 may identify the viewpoint position 24. Alternatively, the person capturing unit 13 may identify the viewpoint position 24 multiple times over a certain period of time, and the average value of the identified viewpoint positions may be used as an average viewpoint position, and the image display unit 15 may display an image of the shared virtual space 41 seen from the average viewpoint position on the display surfaces of the displays 42A to 42C. Alternatively, the person capture unit 13 may continuously calculate the viewpoint position 24 of the user 91 for a certain period of time, and the image display unit 15 may track the viewpoint position 24 and display an image of the shared virtual space 41 as seen from the viewpoint position 24 on the display surfaces of the displays 42A to 42C, during which time the person capture unit 13 may calculate an average viewpoint position, and the image display unit 15 may converge from the viewpoint position 24 identified by the person capture unit 13 to the average viewpoint position over a certain period of time.

[0077] As another example, the image display unit 15 may determine a reference viewpoint position in advance and display an image of the shared virtual space 41 seen from the reference designated position on the display surfaces of the displays 42A to 42C. In this case, the process of the person capture unit 13 identifying the viewpoint position 24 of the user 91 is unnecessary. For example, the viewpoint position when a person with a standard build uses the communication terminal device 10 may be used as the reference viewpoint position. Alternatively, the viewpoint positions of multiple users using the communication terminal device 10 may be measured, and the average value may be used as the reference designated position. The viewpoint position fixed by such calculation or definition may be determined when each user previously used the device by reading information stored in a server on the communication network 90 when connecting to the communication network 90, or by reading information stored in the user's own communication terminal device.

[0078] The communication terminal device 10 of this embodiment constitutes, for example, a one-person conference booth. Multiple such conference booths may be installed in the same room. Conference booths may also be installed in private homes for telecommuting. In such cases, it is preferable to take noise reduction measures to prevent the user 91's voice from leaking to the surroundings. Therefore, the audio output unit 16 may be capable of outputting audio via earphones, a neck speaker, or the like. The audio output unit 16 may also have a speaker with directivity toward the user's fixed position 51. Furthermore, the communication terminal device 10 may use a silent speaker to reduce the amount of the user 91's voice leaking to the surroundings. Furthermore, the output from the audio output unit 16 may be completely turned off, and instead of outputting the voices of other users from the audio output unit 16, the voices may be converted into text using voice recognition technology and output to the image display unit 15.

[0079] Although the present embodiment illustrates an example in which the image display unit 15 includes three displays 42A-42C, other configurations are also possible. For example, in addition to the displays 42A-42C, the image display unit 15 may further include a display located near the bottom of the displays 42A, 42B, and 42C and having a display surface on the top surface of a rectangle. Furthermore, the image display unit 15 may further include a display located near the ceiling-facing side of the displays 42A, 42B, and 42C and having a display surface on the bottom surface of a rectangle. In either case, the image display unit 15 uses information about the user's viewpoint position, which is updated in real time, to display images of the shared virtual space as seen from the real-time viewpoint position on the display surfaces of the four displays. In a usage scenario in which multiple users view screen images together by arranging the user's designated positions 51 and user designated directions 52 of each device as shown in FIG. 13, the images of the shared virtual space can be viewed with a field of view that extends to the bottom and top. Furthermore, the communication terminal device 10 has the functionality of a computer capable of executing application software, and may display the application software screen on the display surface of a display device having a display surface on its upper surface, for example. In this case, the display device may be a touch panel display, and may accept operations on the application software by the user 91. Note that the display device having a display surface on its upper surface and the display device having a display surface on its lower surface described here may be arranged at an angle between 0° and 180° with respect to the horizontal plane.

[0080] In addition, in this embodiment, an example has been shown in which imager 40A is disposed on the upper edge or left edge of the display surface of display 42A, and imager 40C is disposed on the upper edge or right edge of the display surface of display 42C. However, this configuration is not limited to this. As another example, imager 40A may be disposed at a position where the upper edge of the display surface of display 42A and the upper edge of the display surface of display 42B are adjacent to each other, and imager 40C may be disposed at a position where the upper edge of the display surface of display 42C and the upper edge of the display surface of display 42B are adjacent to each other, or they do not have to be located directly above or close to the edges of the displays. The purpose of arranging imagers 40A to 40C is to widely acquire image images and depth information of user 23 so as to obtain an overall image of the user, or to acquire specific parts of the user, such as facial expressions or palms, with higher resolution. Depending on the purpose, imagers 40A to 40C may be disposed farther or closer to user fixed position 51. Furthermore, even if each imaging device and each display device are not connected directly or indirectly via a connector or the like, as long as their relative positions and relative angles are fixed, each display device may be fixed to, for example, a fixture standing on the floor of a room where the image display unit 15 is fixedly arranged or a fixture hanging from the ceiling.

[0081] The communication terminal device 10 of this embodiment may also have an external connection terminal, to which a personal computer can be connected and controlled from the personal computer. The communication terminal device 10 outputs control data for receiving control of the voice acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, the voice output unit 16, and the placement designation receiving unit 17 from the external connection terminal, and receives corresponding control information from the external connection terminal. For example, the placement designation receiving unit 17 may output data for the placement designation screen 60 shown in FIG. 10 to the personal computer via the external connection terminal, obtain control information, such as for moving a desk icon 62, from the personal computer via the external connection terminal, and receive a designation for the placement of the user spaces 22 and 32 in the shared virtual space 41 based on the obtained control information. The image display unit 15 may also configure the image screen 65 illustrated in FIG. 13 in the shared virtual space 41 based on the control information obtained from the personal computer via the external connection terminal, and display the image on the display screen. The external connection terminal is, for example, a wired LAN terminal or a USB terminal. Control information is exchanged directly between the personal computer and the communication terminal device 10, or exchanged via a server on the communication network 90. ​​In this case, the external connection terminal does not have to be a physical connection terminal, and may be a wireless connection such as wireless LAN or Bluetooth. Another example is a terminal that complies with the HDMI (High-Definition Multimedia Interface) (HDMI is a registered trademark) communication standard. The control information is inserted into HDMI capture information and transmitted. In either case, control signals may be transmitted unilaterally from the personal computer to the communication terminal device 10, that is, control may be performed in a situation where no control data is transmitted from the communication terminal device 10 to the personal computer.

[0082] Furthermore, the communication terminal device 10 of this embodiment may be controlled by a user 91 using gestures. The communication terminal device 10 stores gesture setting information in which predetermined movements using hands (hereinafter referred to as control movements) are associated in advance with predetermined controls related to the voice acquisition unit 11, the visual information acquisition unit 12, the information transmission unit 14, the image display unit 15, the voice output unit 16, and the placement designation reception unit 17. The person capturing unit 13 monitors the movements of the captured person, and when it detects that the person has performed a control movement, it instructs the voice acquisition unit 11, the visual information acquisition unit 12, the information transmission unit 14, the image display unit 15, the voice output unit 16, or the placement designation reception unit 17 to perform the control associated with the control movement in the gesture setting information. As an example, an operation of moving an object, whose image is displayed on the display screen and configured in the shared virtual space 41 as a virtual object shared by multiple communication terminal devices 10, will be described. The gesture setting information associates a first control action of holding a hand over an object with a first control for selecting the object, and a second control action of moving a clenched hand toward the object with a second control for moving the object within the shared virtual space 41. The person capturing unit 13 detects the first control action based on visual information acquired by the visual information acquiring unit 12. When the person capturing unit 13 detects the first control action, the image display unit 15 sets the object to a selected state. At this time, the person capturing unit 13 determines that an object located on a line connecting the viewpoint position 24 and the center of the open hand in the shared virtual space 41, or on a line connecting the joint position of a specific body part, such as an elbow or shoulder joint, and the center of the hand, or in the vicinity of either of these lines, is the object selected by the user 41, and the image display unit 15 sets the object to a selected state. For example, the object may be highlighted with color, light, shadow, or the like to indicate that the object has been selected. The information transmission unit 14 may notify other devices that the first control action has been detected or that the object has entered a selected state. In this state, when the person capture unit 13 detects a second control action, the image display unit 15 moves the object in the shared virtual space 41. In this case, the image display unit 15 moves the object if the second control action is detected after the first control action, but does not move the object if the second control action is detected without the first control action. The movement of the object in the shared virtual space 41 is communicated to other devices via image transmission information, and the object also moves on the display of the other devices. While the movement of a virtual object is illustrated here, the object that moves due to a gesture is not particularly limited. For example, the image screen 65 shown in FIG. 13 may be moved, or its size, position, or angle may be changed.

[0083] In addition, in this embodiment, the communication terminal device 10 displays an image of the shared virtual space 41 viewed from one viewpoint position 24 on the display surface, but this is not limited to this example. As a modified example, the communication terminal device 10 may display an image consisting of two images of the shared virtual space 41 (hereinafter referred to as an image for the left eye and an image for the right eye, respectively) with parallax between the left and right eyes. The image display unit 15 includes a display device that displays an image on the display surface and a three-dimensional glasses device worn by the user 91. The image display unit 15 generates an image for the left eye and an image for the right eye and displays them on the display surface. The three-dimensional glasses device displays an image for the left eye to the left eye of the user 91 and an image for the right eye to the right eye of the user 91. By providing parallax between the left and right eyes, a three-dimensional effect is imparted to the image, providing the user 91 with a greater sense of realism and immersion. Note that the method for displaying images with parallax to the left and right eyes is not particularly limited. For example, an anaglyph type, a polarized type, or a liquid crystal shutter type may be used. In general, 3D glasses do not cause anxiety or discomfort like head-mounted displays.

[0084] Although the present embodiment illustrates an example in which the same shared virtual space 41 is shared between the user's own device and the other device, the present invention is not limited to this. As another example, the layout of the user space 22 of the user's own device and the user space 32 of the other device relative to the shared virtual space 41 may be different between the user's own device and the other device. For example, the layout may be adjusted so that the user can easily see the expressions and movements of the other user, while taking into account the layout and size of the display of each device, without causing any discomfort to the user 91. When the user's own device and the other device use shared virtual spaces 41 with different layouts, the layout information used by each device may be shared among all devices, and each device may select an imager from which to transmit visual information to each of the other devices. Alternatively, each device may specify the direction from which it wants visual information captured by each of the other devices, and each device may transmit visual information captured from the direction specified by that device to each of the other devices.

[0085] Although the present embodiment illustrates an example in which the communication terminal device 10 displays a real-time image of the face, the present invention is not limited to this example. As another example, the transmitting device may notify the receiving device of the user's psychological state, and the receiving device may process the user's facial expression based on the notified psychological state. The information transmission unit 14 of the device estimates the user's psychological state from the user's facial expression using the person capture unit 13 of the device. For example, the information transmission unit 14 estimates whether the user is happy, angry, sad, enjoying, agreeing with the conversation content, disagreeing with the conversation content, or not understanding the conversation content. The information transmission unit 14 of the device transmits the information on the psychological state estimated by the person capture unit 13 to the receiving device as image transmission information instead of an image of the face. The receiving device generates and displays an image of the transmitting user's facial region based on the notified psychological state information. This allows the visualization of qualitative information, such as the other person's psychology, which is difficult to visualize, and can help achieve more strategically effective communication. In addition, the amount of data transmitted between devices can be reduced.

[0086] As yet another example, the transmitting device may be able to process the facial expression in the visual information of the user 91 into an expression that expresses the psychological state desired by the user 91, at the request of the user 91. When the desired psychological state is specified by the user 91, the person capturing unit 13 of the transmitting device processes the facial expression of the user 91 into an expression that expresses the specified psychological state. The information transmitting unit 14 transmits the image transmission information in which the facial expression has been processed by the person capturing unit 13 to another device. Since the facial expression can be processed at the request of the user 91, the user 91 can intentionally use the information conveyed by the facial expression to achieve effective communication. [Example]

[0087] In the first embodiment, the communication terminal device 10 selects an image capturer that captures an image to be transmitted to another device, and transmits image transmission information including the image captured by the selected image capturer and depth information to the other device. This reduces the amount of data in the image transmission information transmitted from the device itself to the other device. In the second embodiment, an example is shown in which the amount of data in the image transmission information transmitted from the device itself to the other device is further reduced.

[0088] The configuration of the communication system 1 of the second embodiment is basically the same as that of the first embodiment shown in Fig. 1. The configuration of the communication terminal device 10 of the second embodiment is similar to that of the first embodiment shown in Figs. 6 to 8. The image display process executed by the communication terminal device 10 of the second embodiment is basically similar to that of the first embodiment shown in Fig. 9. However, in the second embodiment, the specific processes in steps S102 and S103 are different from those in the first embodiment.

[0089] In step S102, in the second embodiment, the person capturing unit 13 detects the user in the user space 22 based on the image and / or depth of the user space 22 acquired by the visual information acquiring unit 12. The information transmitting unit 14 generates information representing the face of the user detected from the image and / or depth information of the user space 22 acquired by the visual information acquiring unit 12 and information representing the user's skeleton, and transmits this information to another device as image transmission information. Here, the information representing the skeleton is information that enables the identification of the person's posture, and may be, for example, information indicating the positions of characteristic body parts such as the head, torso, buttocks, shoulders, elbows, hands, knees, and heels. The posture of the person here refers to a human body shape that allows humans to visually observe poses and movements and obtain non-verbal information useful for communication. There is no particular limitation on the level of detail and accuracy required for identifying the posture.

[0090] In step S103, in the second embodiment, the image display unit 15 converts a three-dimensional object in the shared virtual space, including a person, into a two-dimensional image according to the user's viewpoint position, and displays the two-dimensional image on the display surfaces of the displays 42A to 42C. The image of the person's posture based on the information representing the skeleton may be an image that has been processed to express the posture and movement of the person by adding flesh to the information representing the skeleton using a virtual 3DCG object, for example.

[0091] According to this embodiment, it is not necessary to send all captured information from the transmitting communication terminal device 10 to the receiving communication terminal device 10, thereby reducing the amount of data transmitted between the communication terminal devices 10. Furthermore, since parts of the human body other than the face are represented by virtual objects, it is possible to avoid displaying a realistic image of a real object. [Example]

[0092] In the first embodiment, as shown in Figures 6 to 8, the communication terminal device has three displays, each having a rectangular, flat display surface and arranged with the display surface facing the user's fixed position. In contrast, in the third embodiment, the communication terminal device 10 has one display having a curved display surface that is curved so as to surround the user's fixed position.

[0093] 15 is a block diagram of a communication terminal device according to a third embodiment. A communication terminal device 10 according to this embodiment includes a voice acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, a voice output unit 16, and an arrangement designation receiving unit 17. The voice acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, and the voice output unit 16 are basically the same as those in the embodiment shown in FIG. 2. The arrangement designation receiving unit 17 receives a user designation for the arrangement of a user space 22 of the user's own device and a user space 32 of another device relative to a shared virtual space 41.

[0094] In this embodiment, unlike the first embodiment, the image display unit 15 includes one display 42B.

[0095] Fig. 16 is a schematic plan view of a communication terminal device according to a third embodiment. Fig. 17 is a schematic perspective view of a communication terminal device according to the third embodiment.

[0096] 16, in the fourth embodiment, similarly to the first embodiment, a predetermined position in the user space 22 is defined as a user home position 51, and a predetermined direction from the user home position 51 is defined as a user home direction 52. For example, the user home position 51 is a position where the user 23 using the communication terminal device 10 is likely to be present. For example, the user home direction 52 is a direction in which the user 23 using the communication terminal device 10 is likely to be facing.

[0097] As shown in FIG. 17, display 42B is a display device having a curved display surface that curves to surround user fixed position 51. More specifically, the display surface of display 42B is a curved surface that extends over a predetermined angular range (180°, for example) in the circumferential direction of the inner surface of a cylinder. Display 42B may be configured to project images onto a curved screen using multiple projectors, or may be configured with a curved organic EL display. The center of the display surface of display 42B in the horizontal direction is located in user fixed direction 52 when viewed from user fixed position 51. [Example]

[0098] 6 to 8, the first embodiment exemplifies a configuration in which the communication terminal device has three display devices each having a rectangular, flat display surface and arranged with the display surface facing the user's fixed position, and three imagers arranged horizontally separated from each other and with the imaging direction facing the user's fixed position. In contrast, the fourth embodiment exemplifies a configuration in which the communication terminal device 10 has one display device having a rectangular, flat display surface and with the display surface facing the user's fixed position, and one imager arranged with the imaging direction facing the user's fixed position.

[0099] 18 is a block diagram of a communication terminal device according to a fourth embodiment. A communication terminal device 10 according to this embodiment includes a voice acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, a voice output unit 16, and an arrangement designation receiving unit 17. The voice acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, and the voice output unit 16 are basically the same as those in the embodiment shown in FIG. 2. The arrangement designation receiving unit 17 receives a user designation for the arrangement of a user space 22 of the user's own device and a user space 32 of another device relative to a shared virtual space 41.

[0100] In this embodiment, unlike the first embodiment, the visual information acquisition unit 12 includes one image capture device 40B, and the image display unit 15 includes one display device 42B.

[0101] Fig. 19 is a schematic plan view of a communication terminal device according to the fourth embodiment. Fig. 20 is a schematic perspective view of the communication terminal device according to the fourth embodiment.

[0102] 19, in the fourth embodiment, similarly to the first embodiment, a predetermined position in the user space 22 is defined as a user home position 51, and a predetermined direction from the user home position 51 is defined as a user home direction 52. For example, the user home position 51 is a position where the user 23 using the communication terminal device 10 is likely to be present. For example, the user home direction 52 is a direction in which the user 23 using the communication terminal device 10 is likely to be facing.

[0103] 20, display device 42B is a display device having a rectangular, flat display surface, and is disposed with the display surface facing user fixed position 51. Display device 42B is disposed in user fixed direction 52 when viewed from user fixed position 51. Imager 40B is disposed such that its position relative to user fixed position 51 is fixed, its direction relative to user fixed direction 52 is fixed, and its imaging direction faces user fixed position 51. Preferably, imager 40B is disposed near the center of the upper or lower edge of the display surface of display device 42B.

[0104] 21 is a flowchart of the image display process in Example 4. The image display process in Example 4 differs from that in Example 1 shown in FIG. 9 in that it does not include step S102 for selecting transmission data, but is otherwise basically the same as that in Example 1. As described above, in Example 4, the visual information acquisition unit 12 has one imager 40B, and the information transmission unit 14 transmits image transmission information including the image and depth information acquired by the imager 40B to another device.

[0105] The communication terminal device 10 of this embodiment can be configured as a device in which the voice acquisition unit 11, visual information acquisition unit 12, person capture unit 13, information transmission unit 14, image display unit 15, and voice output unit 16 are built into a housing, the image capture unit 40B of the visual information acquisition unit is disposed at a predetermined position on the housing, and an external output terminal is provided on the housing that is connected to a display having a flat display surface and outputs image data captured by the image display unit 15 to the display. In this configuration, the image display unit 15 receives specifications for the size of the display surface and the relative position and orientation of the display surface with respect to the housing. Based on these specifications, the image display unit 15 generates image data of a shared virtual space visible from the user's viewpoint on the display surface and outputs the image data from the external output terminal. By connecting a general-purpose flat display externally, a communication terminal device 10 that realizes communication through a shared space can be easily configured. While this embodiment shows a configuration with only one display 42B, a configuration with multiple external output terminals from the housing and multiple displays may also be used. In this case, the image display unit 15 needs to receive the same number of specifications of the size of the display screen and the relative position and orientation of the display screen with respect to the housing as the number of connected displays.

[0106] Furthermore, as described above, the communication terminal device 10 of this embodiment has only one imager 40B in the visual information acquisition unit 12, so the three-dimensional data that can be generated from the image transmission information that can be transmitted to other devices is limited. Even in the configurations of the first and third embodiments, as typified by the configurations shown in Figures 8 and 16, the three-dimensional data that can be generated may be limited and incomplete depending on the arrangement of the imagers in the installed visual information acquisition unit 12. Therefore, for example, the portion that is not included in the visual information or image transmission information of the user 91 of the communication terminal device 10 of this embodiment may be supplemented in the own device or another device. In the case of performing completion within the device itself, once the person capturing unit 13 captures a person from the visual information from the visual information acquisition unit 12, the information transmission unit 14 generates image transmission information by completing the portions not included in the visual information of the captured person. Specifically, the information transmission unit 14 may store complementary visual information of the side or back of the user 91 acquired by the visual information acquisition unit 12 in advance, and complete the side or back portions of the person not included in the visual information acquired by the visual information acquisition unit 12 using the complementary visual information. Alternatively, the person capturing unit 13 may complete the side or back portions of the person not included in the visual information acquired by the visual information acquisition unit 12 based on the information of the portions included in the visual information. For example, if the user 91 is wearing black clothing, the visual information of the user 91's back may be created in black. When complementing with another device, the information transmission unit 14 and / or a server on the communication network 90 complements the parts of the person included in the image transmission information transmitted from its own device (or the other device from the other device's perspective) that are not included in the image transmission information. Specifically, the information transmission unit 14 may acquire and store complementary visual information of the side or back of the user 91 from its own device (or the other device from the other device's perspective) in advance, and complement the side or back parts of the person that are not included in the image transmission information with the complementary visual information. Alternatively, the person capture unit 13 may complement the side or back parts of the person that are not included in the image transmission information based on the information of the parts included in the image transmission information. [Example]

[0107] In the first embodiment, a configuration in which the communication terminal device includes a display having a rectangular, flat display surface is illustrated as shown in Figures 7 and 8. In the fifth embodiment, a configuration in which the communication terminal device includes a display having a curved display surface is illustrated.

[0108] 22 is a block diagram of a communication terminal device according to Example 5. A communication terminal device 10 according to this example includes a voice acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, a voice output unit 16, and an arrangement designation reception unit 17. The voice acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, the voice output unit 16, and the arrangement designation reception unit 17 are basically the same as those in Example 1 shown in FIG.

[0109] In this embodiment, the visual information acquisition unit 12 includes three image capture devices 40A, 40B, and 40C, and the image display unit 15 includes one display device 42D.

[0110] Fig. 23 is a schematic plan view of a communication terminal device according to a fifth embodiment. Fig. 24 is a schematic perspective view of a communication terminal device according to the fifth embodiment.

[0111] 23 and 24, in the fifth embodiment, the display 42D of the image display unit 15 is configured such that the inner peripheral surface of a vertical cylinder is used as the display surface. The interior of the cylinder of the display 42D forms the user space 22. A predetermined position in the user space 22 is set as the user home position 51. For example, the user home position 51 is a position where the user 23 using the communication terminal device 10 is likely to be located. Since the display 42D is cylindrical, the user home position 51 may be set near its central axis.

[0112] The display 42D is a display device whose display surface is the inner peripheral surface of a cylinder surrounding the user fixed position 51.

[0113] Imagers 40A to 40C are each disposed close to the display surface of display 42D, with a fixed position relative to user fixed position 51, and with their image capturing direction facing user fixed position 51. Imagers 40A to 40C may be disposed, for example, such that imager 40A is oriented in a direction that forms an angle of 110° with imager 40B about the central axis of display 42D, and imager 40C is oriented in a direction that forms an angle of −110° with imager 40B about the central axis of display 42D. Furthermore, assuming that user 23 is of average height, imagers 40A to 40C may be disposed at the height of the head, feet, or face of the user, for example.

[0114] The flowchart of the image display process in the fifth embodiment is the same as that in the first embodiment shown in FIG.

[0115] FIG. 25 is a diagram illustrating an example of an arrangement designation screen in the fifth embodiment. The arrangement designation screen 60 includes a shared space area 64 and a confirm button 63. The shared space area 64 is a plan view simulating the shared virtual space 41, which corresponds to a virtual space shared between the user 23 and the other users 33. In the shared space area 64, booth icons 66, which are objects corresponding to the communication terminal devices 10 used by each member participating in the dialogue, are arranged. The booth icons 66 can be moved or fixed by, for example, a mouse operation, a touch operation on the display surface of the display 42D, or a gesture operation detected based on visual information acquired by the image capturers 40A to 40C. The shape and size of the shared space area 64 may be changeable. Here, as an example, the shared space area 64 is assumed to be simulating the shared virtual space 41, and therefore the shape and size of the shared virtual space 41 may be determined according to the shape and size of the shared space area 64. When the enter button 63 is pressed, the arrangement of the user space of each communication terminal device 10, the user fixed position 51, and the user fixed positions of other devices in the shared virtual space 41 is determined based on the arrangement of the booth icon 66 relative to the shared space area 64 at that time. In the example of Fig. 25, an arrangement in which three users form the vertices of a triangle is illustrated.

[0116] FIG. 26 is a conceptual diagram showing a situation in which a plurality of users are engaged in a simulated face-to-face conversation in a shared space. FIG. 26 shows an example in which the arrangement shown in FIG. 25 is simulated. In FIG. 26, Persons A, B, and C from Company X are standing at the vertices of a triangle and engaging in a conversation. An image of the shared virtual space 41 as seen from the viewpoint of each user is displayed on the display 42D of each user's communication terminal device 10 in the arrangement shown in FIG. 26. This allows all users to see the conversation as if they are engaged in a conversation in the arrangement shown in FIG. 26.

[0117] According to this embodiment, an image of the virtual space shared with other users is displayed in any direction of 360° as seen from the user 23, giving the user 23 a high sense of realism. For example, a virtual space created by CG or a three-dimensional space created by live-action video can be used as the shared virtual space, and a situation in which multiple users are together and interacting in the shared virtual space can be reproduced. The live-action video may be, for example, real-time video captured by a 360-degree camera or recorded video at a location different from the installation location of any of the communication terminal devices 10.

[0118] Although the present embodiment shows an example in which the visual information acquisition unit 12 includes three imagers 40A to 40C, the number of imagers included in the visual information acquisition unit 12 is not limited to three. As another example, the number may be two, or four or more. Furthermore, the positions at which the imagers are arranged are not limited to those described in the present embodiment. [Example]

[0119] In the fifth embodiment, a communication terminal device having a display surface on the inner peripheral surface of a cylindrical shape is exemplified, whereas in the sixth embodiment, a communication terminal device having a display surface on the outer peripheral surface of a columnar shape is exemplified.

[0120] 27 is a block diagram of a communication terminal device according to Example 6. A communication terminal device 10 according to this example includes a voice acquisition unit 11, a visual information acquisition unit 12, a person capture unit 13, an information transmission unit 14, an image display unit 15, a voice output unit 16, and an arrangement designation reception unit 17. The voice acquisition unit 11, the visual information acquisition unit 12, the person capture unit 13, the information transmission unit 14, the image display unit 15, the voice output unit 16, and the arrangement designation reception unit 17 are basically the same as those in Example 1 shown in FIG.

[0121] In this embodiment, the visual information acquisition unit 12 includes eight image capture devices 40A to 40H, and the image display unit 15 includes one display device 42E.

[0122] Fig. 28 is a schematic plan view of a communication terminal device according to a sixth embodiment. Fig. 29 is a schematic perspective view of a communication terminal device according to the sixth embodiment.

[0123] As shown in FIGS. 28 and 29, in the sixth embodiment, a display 42E of the image display unit 15 has a vertical cylindrical outer peripheral surface as a display surface.

[0124] Display 42E is a display device whose display surface is the outer peripheral surface of a cylindrical shape. The outside of the cylinder of display 42E is user space 22. Imagers 40A-40H are each disposed close to the display surface of display 42E, fixed facing the outside of the cylinder, and arranged with their imaging direction facing user fixed position 51. Imagers 40A-40H may be disposed, for example, in directions that form a 45° angle with each other about the central axis of display 42E. Furthermore, the height of imagers 40A-40H may be, for example, above, at the feet of, or at face height, assuming that user 23 is of average height.

[0125] The communication terminal device 10 of the sixth embodiment is suitable for conducting a dialogue in combination with the communication terminal device 10 of the fifth embodiment. Hereinafter, the user of the communication terminal device 10 of the sixth embodiment will be referred to as user 91(5), and the user of the communication terminal device 10 of the fifth embodiment will be referred to as user 91(4).

[0126] The size of the shared virtual space 41 is set based on the size of the real space in which the communication terminal device 10 of the sixth embodiment is installed. For example, the shared virtual space 41 in the communication terminal device 10 of the sixth embodiment may be infinite. The relative positions of the central axis of the columnar shape of the communication terminal device 10 of the sixth embodiment and the central axis of the cylindrical shape of the communication terminal device 10 of the fifth embodiment are made to coincide in the shared virtual space 41. Then, for example, a 360-degree video of the real space in which the user 91(5) in which the communication terminal device 10 is installed is created by synthesizing image videos obtained from the imagers 40A to 40H in real time, and projected onto the display of the communication terminal device 10 of the user 91(4). This allows the user 91(4) to see as if he or she were immersed in the real space in which the user 91(5) is located. The user 91(5) can see as if the user 91(4) appeared in the real space in which the user 91(5) is located. FIG. 30 is a conceptual diagram showing the field of view obtained by the users of the sixth embodiment. In the real space 71 in the room where the user 91(5) is actually present, a user 91(4) who is actually remotely present appears, simulating a situation in which the users 91(5) and 91(4) are face-to-face and conversing.

[0127] According to this embodiment, the user can simultaneously view the real space where the user is actually present and the shared virtual space displayed on the display screen, so that the user can communicate with other users as if the other users had appeared in the user's location. Also, the other users using the communication terminal device 10 of the fifth embodiment can communicate with the user as if they were visiting the location where the communication terminal device 10 of the sixth embodiment is installed. For example, a use case such as inspecting a remote location is envisioned.

[0128] In this embodiment, the outer peripheral surface of a cylindrical shape is used as the display surface, but the shape of the display surface is not limited to this. For example, the outer surface of a polygonal prism or sphere may be used as the display surface, or the outer surface of an egg-shaped shape may be used as the display surface. [Explanation of symbols]

[0129] 1...communication system, 10...communication terminal device, 11...audio acquisition unit, 12...visual information acquisition unit, 13...person capture unit, 14...information transmission unit, 14C...image capture device, 14B...image capture device, 15...image display unit, 16...audio output unit, 17...placement designation reception unit, 21...virtual space, 22...user space, 23...own user, 24...viewpoint position, 31...virtual space, 32...user space, 33...other users, 35...whiteboard, 40...image capture device, 41...shared virtual space, 42...display surface, 42A to 42E...display device, 51...user fixed position, 52...user fixed direction, 60...placement designation screen, 61...conference room area, 62...desk icon, 63...decision button, 64...shared space area, 65...image screen, 66...booth icon, 71...real space, 90...communication network, 91...user

Claims

1. A plurality of communication terminal devices interconnected via a communication line, a voice acquisition unit that acquires voice including the voice of the user using the device; a visual information acquisition unit that acquires visual information that enables the creation of three-dimensional data of a person in a predetermined user space; an information transmission unit that transmits image transmission information based on the visual information in the user space and audio transmission information based on the audio to another device; an image display unit that generates and displays, on a display surface whose relative position is fixed with respect to a shared virtual space in which the user space of the user's own device and the user space of the other device are arranged, a display image of the three-dimensional data of the person in the shared virtual space based on image transmission information transmitted from the other device, the three-dimensional data of the person appearing as if the three-dimensional data of the person in the shared virtual space were viewed from the viewpoint position of the user of the own device through the display surface; an audio output unit that outputs audio based on the audio transmission information transmitted from the other device; A communication terminal device having the above configuration.

2. a person capturing unit that identifies a viewpoint position of the user based on visual information of the user space; The communication terminal device according to claim 1 .

3. the person capturing unit detects the user based on visual information in the user space; When the user is detected, the information transmission unit transmits information representing a face part of the user to the other device as the image transmission information; the image display unit displays, on the display surface, an image including a face image based on the information representing a face part received from the other device. The communication terminal device according to claim 1 .

4. When the user is detected, the information transmission unit further transmits information representing the skeleton of the user to the other device as the image transmission information; the image display unit displays, on the display surface, an image of the shared virtual space including a face image based on information representing a face portion received from the other device and an image based on information representing a skeleton received from the other device. The communication terminal device according to claim 1 .

5. the image display unit displays the person in the shared virtual space using a point cloud and / or a polygon mesh that can be processed; The communication terminal device according to claim 1 .

6. the audio output unit reproduces audio that can be heard from a direction in which a user space of the other device is located in the shared virtual space, based on audio transmission information transmitted from the other device; The communication terminal device according to claim 1 .

7. a predetermined position in the user space is set as a user home position; the visual information acquisition unit has a plurality of image capture devices that acquire visual information in respective image capture directions, the plurality of image capture devices are arranged to be separated from one another, have fixed relative positions with respect to the user fixed position, and are arranged such that the image capture directions are oriented in a direction in which at least a part of the user space is captured; the information transmission unit acquires, from the other device, image transmission information based on visual information acquired by an image capture device selected from the plurality of image capture devices of the other device based on a relative angle between an image capture direction of the image capture device and a line connecting a user's fixed position of the other device and a user's fixed position of the own device; The communication terminal device according to claim 1 .

8. a predetermined position in the user space is set as a user home position; the visual information acquisition unit defines a predetermined direction from the user's fixed position as a user's fixed direction, and includes a right image capturer disposed on the right side of the user's fixed direction and a left image capturer disposed on the left side of the user's fixed direction; The information transmission unit acquires image transmission information from the other device, which includes information from a right image sensor of the other device and does not include information from a left image sensor of the other device, when the user position of the other device is located to the right of the user position of the user's own device in the user direction in the shared virtual space, and receives image transmission information from the other device, which includes information from a left image sensor of the other device and does not include information from a right image sensor of the other device, when the user position of the other device is located to the left of the user position of the user's own device in the user direction in the shared virtual space. The communication terminal device according to claim 1 .

9. a predetermined position in the user space is a user home position, and a predetermined direction from the user home position is a user home direction; the image display unit arranges a user home position of at least one of the own device and the other device on each of a first line and a second line that are aligned with each other, and forms the shared virtual space such that a user home direction of the device whose user home position is arranged on the first line is a direction toward the second line, and a user home direction of the device whose user home position is arranged on the second line is a direction toward the first line. The communication terminal device according to claim 1 .

10. a predetermined position in the user space is a user home position, and a predetermined direction from the user home position is a user home direction; the image display unit forms the shared virtual space such that user fixed positions of the own device and the other devices are arranged in a ring or polygonal shape, and user fixed directions of the own device and the other devices are directed toward the inside of the ring or polygon. The communication terminal device according to claim 1 .

11. a predetermined position in the user space is a user home position, and a predetermined direction from the user home position is a user home direction; the image display unit forms the shared virtual space by arranging user fixed positions of the own device and the other device on a predetermined line and aligning user fixed directions of the own device and the other device with the same direction relative to the line. The communication terminal device according to claim 1 .

12. A predetermined direction from the user's fixed position is defined as a user's fixed direction, The image display unit has a front display having a rectangular, flat display surface arranged in the user-specific direction as viewed from the user's fixed position, a right display having a rectangular, flat display surface arranged to the right of the user's fixed direction as viewed from the user's fixed position, and a left display having a rectangular, flat display surface arranged to the left of the user's fixed direction as viewed from the user's fixed position, the display surface of the right display being adjacent to the right side of the display surface of the front display, and the display surface of the left display being adjacent to the left side of the display surface of the front display. The communication terminal device according to claim 1 .

13. the image display unit further comprises a lower display disposed adjacent to the bottom sides of the three displays consisting of the front display, the left display, and the right display, and having a display surface on the upper surface of a rectangle, and / or an upper display disposed adjacent to the ceiling side of the three displays consisting of the front display, the left display, and the right display, and having a display surface on the lower surface of a rectangle, The communication terminal device according to claim 12.

14. a predetermined position in the user space is set as a user home position; The image display unit further includes a front display having a rectangular, flat display surface arranged in the user's direction when viewed from the user's position, and a lower display arranged close to the bottom side of the front display and having a display surface on the rectangular upper surface, and / or an upper display arranged close to the top side of the front display and having a display surface on the rectangular lower surface, The communication terminal device according to claim 1 .

15. a predetermined position in the user space is set as a user home position; the display surface of the image display unit is an inner surface that surrounds the user position in the horizontal direction; The communication terminal device according to claim 1 .

16. The image display unit further includes a display having a display surface that is an inner surface that surrounds the user position in the horizontal direction, and a lower display disposed adjacent to the bottom side of the display and having a display surface on a rectangular upper surface, and / or an upper display disposed adjacent to the top side of the front display and having a display surface on a rectangular lower surface, 16. The communication terminal device according to claim 15.

17. a predetermined position in the user space is set as a user home position; the image display unit has at least a portion of the display surface near an outer edge curved toward the user's fixed position; The communication terminal device according to claim 1 .

18. The image display unit further includes a display having a display surface in which at least a portion of the vicinity of an outer edge of the display surface is curved toward the user's fixed position, and a lower display disposed close to the bottom side of the display and having a display surface on a rectangular upper surface, and / or an upper display disposed close to the top side of the front display and having a display surface on a rectangular lower surface.

18. The communication terminal device according to claim 17.

19. The image display unit has a display surface that is a columnar outer peripheral surface. The communication terminal device according to claim 1 .

20. a layout specification receiving unit that displays a screen on which an object corresponding to the own device and an object corresponding to the other device can be movably arranged, and receives a specification of the layout of the user spaces of the own device and the other device relative to the shared virtual space by arranging the objects on the screen; The communication terminal device according to claim 1 .

21. the audio acquisition unit, the visual information acquisition unit, the information transmission unit, the image display unit, and the audio output unit are built into a housing, an image pickup device of the visual information acquisition unit is disposed at a predetermined position of the housing, and at least one or more external output terminals are provided on the housing, which are connected to at least one or more display units having a flat display surface and output image data of the image display unit to the display units; The image display unit receives designation of the size of the display surface and the relative position and orientation of the display surface with respect to the housing, and based on the designation, generates a display image of the three-dimensional data of the person in the shared space that looks like the three-dimensional data of the person in the shared virtual space when viewed from the viewpoint position of the user via the display surface, and outputs the display image from the external output terminal. The communication terminal device according to claim 1 .

22. the information transmission unit complements information on a part of the person included in the image transmission information that is not included in the image transmission information; The communication terminal device according to claim 1 .

23. Further having an external connection terminal, control information for receiving control of at least one of the voice acquisition unit, the visual information acquisition unit, the person capture unit, the information transmission unit, the image display unit, and the voice output unit is input from the external connection terminal; The communication terminal device according to claim 1 .

24. the person capturing unit detects that the user has performed a predetermined action; At least one of the voice acquisition unit, the visual information acquisition unit, the information transmission unit, the image display unit, and the voice output unit executes a process corresponding to the operation. The communication terminal device according to claim 1 .

25. the person capturing unit detects a first action of the user holding a hand over an object in the shared virtual space and a second action of the user fisting and moving the hand toward the object; the image display unit moves the object when the second motion is detected after the first motion is detected, and does not move the object when the second motion is detected without the first motion being detected.

22. The communication terminal device according to claim 21.

26. the image display unit includes a display device that displays the image on the display surface, and a three-dimensional eyeglass device worn by the user; the display device generates a left-eye image and a right-eye image having parallax in accordance with the viewpoint position, and displays the images on the display surface; the 3D eyeglass device displays the left-eye image to the left eye of the user, and the right-eye image to the right eye of the user; The communication terminal device according to claim 1 .

27. When the person capturing unit receives a designation of a desired psychological state, the person capturing unit processes a facial portion of the user in the visual information into an expression that expresses the psychological state, The information transmission unit transmits image transmission information based on the visual information in which the face portion of the user has been processed to another device. The communication terminal device according to claim 1 .

28. A communication method using a plurality of communication terminal devices interconnected via a communication line, comprising: The communication terminal device Acquires audio including the voice of the user using the device; acquiring visual information that enables the formation of three-dimensional data of a person in a predetermined user space; transmitting image transmission information based on the visual information in the user space and audio transmission information based on the audio to another device; a display screen having a fixed relative position with respect to a shared virtual space in which the user space of the user's own device and the user space of the other device are arranged, based on image transmission information transmitted from the other device, generating and displaying a display image of the three-dimensional data of the person in the shared space in a manner that makes the three-dimensional data appear as if the three-dimensional data of the person in the shared virtual space were viewed from the viewpoint position of the user's own device through the display screen; outputting a voice based on the voice transmission information transmitted from the other device; Communication method.

29. A software program for causing a computer to operate as a control device of a plurality of communication terminal devices interconnected via a communication line, Acquires audio including the voice of the user using the device; acquiring visual information that enables the formation of three-dimensional data of a person in a predetermined user space; transmitting image transmission information based on the visual information in the user space and audio transmission information based on the audio to another device; a display screen having a fixed relative position with respect to a shared virtual space in which the user space of the user's own device and the user space of the other device are arranged, based on image transmission information transmitted from the other device, generating and displaying a display image of the three-dimensional data of the person in the shared space in a manner that makes the three-dimensional data appear as if the three-dimensional data of the person in the shared virtual space were viewed from the viewpoint position of the user's own device through the display screen; outputting a voice based on the voice transmission information transmitted from the other device; A software program that causes a computer to carry out instructions.

Images