Terminal equipment

Abstract

To enable a user to more easily recognize the body movement of another user in another space.SOLUTION: A terminal apparatus is provided with a control unit for: generating a shadow image of a user object as a second image 12 based on a positional relationship between a virtual light source arranged in a virtual three-dimensional space, a display object corresponding to a first display 36, and a user object corresponding to a first user 61 operating a first image 11 displayed on the first display 36; displaying the first image 11 on a second display 46 directed to a second user 62 while superimposing the second image 12 on the first image 11 on the second display 46; when a change in a relative position of the first user 61 is detected, adjusting the relative position of the user object in accordance with the detected change; updating the second image 12 based on a change in the positional relationship resulting from the adjusting; and reflecting the updating of the second image 12 on the second display 46.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to a terminal device, an image display method, and a program. [Background technology]

[0002] Patent Document 1 discloses a program for causing another avatar object to share the viewpoint of an avatar object placed in a virtual space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-192177 A Summary of the Invention [Problem to be solved by the invention]

[0004] It is conceivable to use a remote dialogue system to realize a service in which a student receives lessons involving physical movements, such as calligraphy lessons, from a teacher in a remote location. However, in conventional systems, it is difficult for the student to recognize the teacher's physical movements.

[0005] An object of the present disclosure is to make it easier for users to recognize the body movements of other users in another space. [Means for solving the problem]

[0006] A terminal device according to an embodiment of the present disclosure includes: The device further comprises a control unit that generates an image of a shadow of the user object cast on the display object as a second image based on a positional relationship between a virtual light source, a display object corresponding to a first display, and a user object corresponding to a first user manipulating a first image displayed on the first display, which are arranged in a virtual three-dimensional space, and displays the first image on a second display for a second user in a second space separate from a first space in which the first user is located, and superimposes the second image on the first image on the second display, and upon detecting a change in a relative position of the first user with respect to the first display, adjusts a relative position of the user object with respect to the display object in accordance with the detected change, updates the second image based on the change in the positional relationship resulting from the adjustment, and reflects the update of the second image on the second display.

[0007] A terminal device according to another aspect of the present disclosure includes: The device includes a control unit that generates an image of the user object captured by the virtual camera as a second image based on a positional relationship between a virtual camera arranged in a virtual three-dimensional space and a user object corresponding to a first user manipulating a first image displayed on a first display, and a direction of the virtual camera, displays the first image on a second display toward a second user in a second space separate from a first space in which the first user is located, and superimposes the second image on the first image on the second display, and upon detecting a change in the line of sight of the second user, adjusts the direction of the virtual camera in accordance with the detected change, updates the second image based on the adjusted direction of the virtual camera, and reflects the update of the second image on the second display.

[0008] The image display method according to the present disclosure includes: generating, as a second image, an image of a shadow of the user object cast on the display object or an image of the user object captured by the virtual camera based on a positional relationship between a virtual light source arranged in a virtual three-dimensional space, a display object corresponding to a first display, and a user object corresponding to a first user who operates a first image displayed on the first display, or based on a positional relationship between a virtual camera arranged in the virtual three-dimensional space and the user object, and a direction of the virtual camera; displaying the first image on a second display for a second user in a second space separate from the first space in which the first user is located; superimposing the second image on the first image on the second display; when detecting a change in a relative position of the first user with respect to the first display or a change in a line of sight of the second user, adjusting a relative position of the user object with respect to the display object or a direction of the virtual camera in response to the detected change; updating the second image based on a change in the positional relationship resulting from the adjustment or a direction of the virtual camera after the adjustment; reflecting the update of the second image on the second display; and Includes. Effect of the Invention

[0009] According to the present disclosure, it becomes easier for a user to recognize the body movements of other users in a different space. [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram showing a configuration of a communication system according to a first embodiment and a second embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram showing a first space and a second space according to the first embodiment of the present disclosure. [Diagram 3] FIG. 2 is a diagram showing a virtual three-dimensional space according to the first embodiment of the present disclosure. [Figure 4]6 is a flowchart showing an operation of a second terminal device according to the first embodiment of the present disclosure. [Diagram 5] FIG. 11 is a diagram showing a first space and a second space according to a second embodiment of the present disclosure. [Figure 6] FIG. 11 is a diagram showing a virtual three-dimensional space according to a second embodiment of the present disclosure. [Figure 7] 10 is a flowchart showing an operation of a second terminal device according to a second embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, several embodiments of the present disclosure will be described with reference to the drawings.

[0012] In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of each embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate.

[0013] The configuration of a communication system 10 according to a first and second embodiment of the present disclosure will be described with reference to FIG.

[0014] The communication system 10 includes a server device 20, a first terminal device 30, and a second terminal device 40. The server device 20 is capable of communicating with the first terminal device 30 and the second terminal device 40 via a network 50. The first terminal device 30 may be capable of communicating with the second terminal device 40 via the network 50.

[0015] The server device 20 is installed in a facility such as a data center and operated by a service provider. The server device 20 is a server computer that belongs to a cloud computing system or other computing system.

[0016] The first terminal device 30 is installed in a first space 60 as shown in Fig. 2 or Fig. 5, and is used by a first user 61 in the first space 60. The first terminal device 30 is, for example, a general-purpose computer such as a PC, or a dedicated computer specialized for a specific calculation. "PC" is an abbreviation for personal computer.

[0017] The second terminal device 40 is installed in a second space 70 separate from the first space 60 as shown in Fig. 2 or 5, and is used by a second user 71 in the second space 70. The second terminal device 40 is, for example, a general-purpose computer such as a PC, or a dedicated computer specialized for a specific calculation.

[0018] Network 50 includes the Internet, at least one WAN, at least one MAN, or any combination thereof. "WAN" is an abbreviation for wide area network. "MAN" is an abbreviation for metropolitan area network. Network 50 may include at least one wireless network, at least one optical network, or any combination thereof. A wireless network may be, for example, an ad-hoc network, a cellular network, a wireless LAN, a satellite communication network, or a terrestrial microwave network. "LAN" is an abbreviation for local area network.

[0019] The configuration of a first terminal device 30 according to the first and second embodiments will be described with reference to FIG.

[0020] The first terminal device 30 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit , and an output unit .

[0021] The control unit 31 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or GPU, or a dedicated processor specialized for a specific process. "CPU" is an abbreviation for central processing unit. "GPU" is an abbreviation for graphics processing unit. The programmable circuit is, for example, an FPGA. "FPGA" is an abbreviation for field-programmable gate array. The dedicated circuit is, for example, an ASIC. "ASIC" is an abbreviation for application specific integrated circuit. The control unit 31 executes processes related to the operation of the first terminal device 30 while controlling each part of the first terminal device 30.

[0022] The storage unit 32 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. "RAM" is an abbreviation for random access memory. "ROM" is an abbreviation for read only memory. The RAM is, for example, an SRAM or a DRAM. "SRAM" is an abbreviation for static random access memory. "DRAM" is an abbreviation for dynamic random access memory. The ROM is, for example, an EEPROM. "EEPROM" is an abbreviation for electrically erasable programmable read only memory. The flash memory is, for example, an SSD. "SSD" is an abbreviation for solid-state drive. The magnetic memory is, for example, an HDD. "HDD" is an abbreviation for hard disk drive. The storage unit 32 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 32 stores data used in the operation of the first terminal device 30 and data obtained by the operation of the first terminal device 30.

[0023] The communication unit 33 includes at least one communication interface. The communication interface is, for example, an interface compatible with a wired LAN communication standard such as Ethernet (registered trademark), a wireless LAN communication standard such as IEEE802.11, or a mobile communication standard such as LTE, 4G standard, or 5G standard. "IEEE" is an abbreviation for Institute of Electrical and Electronics Engineers. "LTE" is an abbreviation for Long Term Evolution. "4G" is an abbreviation for 4th generation. "5G" is an abbreviation for 5th generation. The communication unit 33 communicates with the server device 20. The communication unit 33 may also communicate with the second terminal device 40. The communication unit 33 receives data used in the operation of the first terminal device 30, and transmits data obtained by the operation of the first terminal device 30.

[0024] The input unit 34 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen integrally provided with a display such as the first display 36 as shown in FIG. 2 or FIG. 5, a visible light camera, a depth camera, LiDAR, or a microphone. "LiDAR" is an abbreviation for light detection and ranging. The input unit 34 accepts an operation to input data used for the operation of the first terminal device 30. The input unit 34 may be connected to the first terminal device 30 as an external input device instead of being provided in the first terminal device 30. As the connection interface, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used. "USB" is an abbreviation for Universal Serial Bus. "HDMI (registered trademark)" is an abbreviation for High-Definition Multimedia Interface.

[0025] The output unit 35 includes at least one output interface. The output interface is, for example, a display such as the first display 36, or a speaker. The display is, for example, an LCD or an organic EL display. "LCD" is an abbreviation for liquid crystal display. "EL" is an abbreviation for electro luminescent. The first display 36 may be a display of any shape, but in the first and second embodiments, it is an L-shaped display in a side view having a vertical display surface extending in the vertical direction and a horizontal display surface 38 adjacent to the lower end of the vertical display surface and extending in the horizontal direction. The output unit 35 outputs data obtained by the operation of the first terminal device 30. The output unit 35 may be connected to the first terminal device 30 as an external output device instead of being provided in the first terminal device 30. As the connection interface, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0026] The functions of the first terminal device 30 are realized by executing a program according to the first or second embodiment on a processor serving as the control unit 31. That is, the functions of the first terminal device 30 are realized by software. The program causes a computer to execute the operations of the first terminal device 30, thereby causing the computer to function as the first terminal device 30. That is, the computer functions as the first terminal device 30 by executing the operations of the first terminal device 30 in accordance with the program.

[0027] The program may be stored in a non-transitory computer-readable medium. Examples of the non-transitory computer-readable medium include a flash memory, a magnetic recording device, an optical disk, a magneto-optical recording medium, or a ROM. The program may be distributed, for example, by selling, transferring, or lending a portable medium such as an SD card, DVD, or CD-ROM on which the program is stored. "SD" is an abbreviation for Secure Digital. "DVD" is an abbreviation for digital versatile disc. "CD-ROM" is an abbreviation for compact disc read only memory. The program may be distributed by storing the program in the storage of a server and transferring the program from the server to another computer. The program may be provided as a program product.

[0028] A computer temporarily stores in a main storage device, for example, a program stored in a portable medium or a program transferred from a server. Then, the computer reads the program stored in the main storage device with a processor, and executes processing according to the read program with the processor. The computer may read the program directly from a portable medium and execute processing according to the program. The computer may execute processing according to the received program each time a program is transferred from the server to the computer. Processing may be executed by a so-called ASP-type service that realizes functions only by execution instructions and result acquisition without transferring a program from the server to the computer. "ASP" is an abbreviation for application service provider. A program is information used for processing by a computer, and includes those equivalent to a program. For example, data that is not a direct command to a computer but has the property of defining computer processing falls under "those equivalent to a program."

[0029] A part or all of the functions of the first terminal device 30 may be realized by a programmable circuit or a dedicated circuit as the control unit 31. In other words, a part or all of the functions of the first terminal device 30 may be realized by hardware.

[0030] The configuration of a second terminal device 40 according to the first and second embodiments will be described with reference to FIG.

[0031] The second terminal device 40 includes a control unit 41, a storage unit 42, a communication unit 43, an input unit 44, and an output unit 45.

[0032] The control unit 41 includes at least one processor, at least one programmable circuit, at least one dedicated circuit, or any combination thereof. The processor is a general-purpose processor such as a CPU or a GPU, or a dedicated processor specialized for a specific process. The programmable circuit is, for example, an FPGA. The dedicated circuit is, for example, an ASIC. The control unit 41 executes processes related to the operation of the second terminal device 40 while controlling each part of the second terminal device 40.

[0033] The storage unit 42 includes at least one semiconductor memory, at least one magnetic memory, at least one optical memory, or any combination thereof. The semiconductor memory is, for example, a RAM, a ROM, or a flash memory. The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The flash memory is, for example, an SSD. The magnetic memory is, for example, an HDD. The storage unit 42 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 42 stores data used in the operation of the second terminal device 40 and data obtained by the operation of the second terminal device 40.

[0034] The communication unit 43 includes at least one communication interface. The communication interface is, for example, an interface compatible with a wired LAN communication standard such as Ethernet (registered trademark), a wireless LAN communication standard such as IEEE802.11, or a mobile communication standard such as LTE, 4G standard, or 5G standard. The communication unit 43 communicates with the server device 20. The communication unit 43 may also communicate with the first terminal device 30. The communication unit 43 receives data used in the operation of the second terminal device 40, and transmits data obtained by the operation of the second terminal device 40.

[0035] The input unit 44 includes at least one input interface. The input interface is, for example, a physical key, a capacitive key, a pointing device, a touch screen integrally provided with a display such as the second display 46 as shown in FIG. 2 or FIG. 5, a visible light camera, a depth camera, a LiDAR, or a microphone. The input unit 44 accepts an operation to input data used for the operation of the second terminal device 40. The input unit 44 may be connected to the second terminal device 40 as an external input device instead of being provided in the second terminal device 40. As the connection interface, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0036] The output unit 45 includes at least one output interface. The output interface is, for example, a display such as the second display 46, or a speaker. The display is, for example, an LCD or an organic EL. The second display 46 may be a display of any shape, but in the first and second embodiments, it is an L-shaped display in a side view having a vertical display surface 47 extending in the vertical direction and a horizontal display surface 48 adjacent to the lower end of the vertical display surface 47 and extending in the horizontal direction. The output unit 45 outputs data obtained by the operation of the second terminal device 40. The output unit 45 may be connected to the second terminal device 40 as an external output device instead of being provided in the second terminal device 40. As the connection interface, an interface compatible with a standard such as USB, HDMI (registered trademark), or Bluetooth (registered trademark) can be used.

[0037] The functions of the second terminal device 40 are realized by executing another program according to the first or second embodiment in a processor serving as the control unit 41. That is, the functions of the second terminal device 40 are realized by software. The program causes a computer to execute the operations of the second terminal device 40, thereby causing the computer to function as the second terminal device 40. That is, the computer functions as the second terminal device 40 by executing the operations of the second terminal device 40 in accordance with the program.

[0038] A part or all of the functions of the second terminal device 40 may be realized by a programmable circuit or a dedicated circuit as the control unit 41. In other words, a part or all of the functions of the second terminal device 40 may be realized by hardware.

[0039] The first embodiment will be outlined with reference to FIG. 2 and FIG. 3 in addition to FIG.

[0040] In FIG. 2, for convenience of explanation, the first space 60 and the second space 70 are adjacent to each other, but in reality, the first space 60 and the second space 70 are located at a distance from each other. The first user 61 operates the first image 11 displayed on the first display 36. In this embodiment, the first image 11 includes an image of a character handwritten by the first user 61, but may include other images, such as an image of a picture drawn by the first user 61, instead of or in addition to the image of the character. The first user 61 is, for example, a teacher in a remote location, and can take a lesson involving physical movements, such as a calligraphy lesson, by operating the first image 11 with a touch screen. The second user 62 visually checks the operation of the first image 11 by the first user 61. The second user 62 is, for example, a student, and can take a lesson involving physical movements, such as a calligraphy lesson, by visually checking the operation of the first image 11.

[0041] The second terminal device 40 places a virtual light source 81, a display object 82 corresponding to the first display 36, and a user object 83 corresponding to the first user 61 in a virtual three-dimensional space 80 as shown in FIG. 3. In this embodiment, the user object 83 includes an object representing the arm of the first user 61, but may include other objects such as an object representing the head of the first user 61 instead of or in addition to the object representing the arm. The second terminal device 40 generates an image of a shadow 84 of the user object 83 cast on the display object 82 as the second image 12 based on the positional relationship between the virtual light source 81, the display object 82, and the user object 83. In this embodiment, the second image 12 includes an image of a shadow of an object representing the arm of the first user 61, but may include an image of a shadow of another object such as an image of a shadow of an object representing the head of the first user 61 instead of or in addition to the image of the shadow of the object representing the arm. The second terminal device 40 displays the first image 11 on the second display 46 toward the second user 62 and also superimposes the second image 12 on the first image 11 on the second display 46 .

[0042] When the second terminal device 40 detects a change in the relative position of the first user 61 with respect to the first display 36, the second terminal device 40 adjusts the relative position of the user object 83 with respect to the display object 82 in accordance with the detected change. The second terminal device 40 updates the second image 12 based on the change in positional relationship resulting from the adjustment. The second terminal device 40 reflects the update of the second image 12 on the second display 46.

[0043] In this embodiment, a shadow 84 of a user object 83 corresponding to the first user 61 is displayed on the second display 46, so that the second user 62 can visually recognize the operation of the first image 11 as if he / she were the first user 61. That is, according to this embodiment, it becomes easier for the second user 62 to recognize the body movement of the first user 61. For example, by displaying the shadow of a body part such as the arm of a teacher in a remote location toward the students and linking this shadow to the actual body movement of the teacher, it becomes easier for the students to recognize the body movement of the teacher during a lesson involving body movement such as a calligraphy lesson.

[0044] This embodiment can be applied to a remote dialogue system such as a video conference where a user can have a dialogue while viewing the image of the dialogue partner. For example, when using the remote dialogue system to realize a service in which a student receives a lesson involving arm movements, such as a calligraphy lesson, from a teacher in a remote location, two remote dialogue terminal devices each having a desk-type display are connected online. By displaying an image simulating the shadow of the teacher's arm on the teacher's display on the student's display, the student can indirectly visually recognize the difference between his or her own arm movements and the teacher's arm movements.

[0045] The first terminal device 30 corresponds to one of the two remote dialogue terminal devices. The first terminal device 30 displays a first image 11 on a horizontal display surface 38 of a first display 36. The first terminal device 30 generates a visible light image of the first user 61 using a visible light camera. The first terminal device 30 generates a depth image of the first user 61 using a depth camera.

[0046] The second terminal device 40 corresponds to the other of the two remote dialogue terminal devices. The second terminal device 40 receives the visible light image and the depth image from the first terminal device 30. The second terminal device 40 places the 3D object of the dialogue partner in the virtual three-dimensional space 80 based on the received visible light image and the depth image. The second terminal device 40 generates, as the second image 12, an image of the shadow of the 3D object that can be made on a plane in the virtual three-dimensional space 80 corresponding to the horizontal display surface 38 of the first display 36 based on the position of the virtual light source 81 in the virtual three-dimensional space 80. The second terminal device 40 displays the first image 11 displayed on the first display 36 on the horizontal display surface 48 of the second display 46, and displays the second image 12 on the horizontal display surface 48 of the second display 46 in a form in which the second image 12 is superimposed on the first image 11. This makes it possible to give the second user 62 the illusion that he or she has become a dialogue partner. For example, when a student tries to write simultaneously with a teacher with whom he or she is interacting, the student can indirectly see the difference in the movement of the two students' arms by comparing the actual shadow of the student's arm with the shadow of the teacher's arm displayed on the top of a desk-type display.

[0047] The second terminal device 40 may also display an image on the vertical display surface 47 of the second display 46. For example, the second terminal device 40 may display, on the vertical display surface 47, an image of the user object 83 captured by a virtual camera disposed at a position that captures the user object 83 from the front in the virtual three-dimensional space 80.

[0048] In this embodiment, a light source 72 capable of irradiating light onto the horizontal display surface 48 of the second display 46 is installed in the second space 70. The second terminal device 40 places a virtual light source 81 at a position in the virtual three-dimensional space 80 corresponding to the position of the light source 72 in the second space 70. This makes it easier for the second user 62 to compare his or her own body movement with that of the first user 61. For example, when there is no difference in arm movement between the student and the teacher, the actual shadow of the student's arm matches the shadow of the teacher's arm displayed on the top surface of the desk-type display, so the student can easily determine whether or not his or her arm movement is appropriate.

[0049] The operation of the second terminal device 40 according to this embodiment will be described with reference to Fig. 4. This operation corresponds to the image display method according to this embodiment.

[0050] The flow shown in FIG. 4 is repeatedly executed while the first user 61 and the second user 71 are performing video communication such as a web conference using the first terminal device 30 and the second terminal device 40, respectively.

[0051] In step S101, the control unit 41 of the second terminal device 40 receives the first image 11 and the visible light image and depth image of the first user 61 from the server device 20 via the communication unit 43, and transmits the visible light image and depth image of the second user 62 to the server device 20 via the communication unit 43. The first image 11 includes an image of characters handwritten by the first user 61. The first image 11 specifically includes an image of paper and an image of black characters handwritten on the paper image by the first user 61 using a touch screen. The control unit 41 may further receive the voice of the first user 61 from the server device 20 via the communication unit 43. The control unit 41 may further transmit the voice of the second user 71 to the server device 20 via the communication unit 43. The process of step S101 is specifically executed in the following procedure.

[0052] The control unit 31 of the first terminal device 30 displays the first image 11 on the horizontal display surface 38 of the first display 36 serving as the output unit 35. The control unit 31 captures a visible light image and a depth image of the first user 61 using a visible light camera and a depth camera serving as the input unit 34, respectively. The control unit 31 transmits the displayed first image 11 and the captured visible light image and depth image to the server device 20 via the communication unit 33. The control unit 31 may acquire the voice of the first user 61 via a microphone serving as the input unit 34, and further transmit the acquired voice to the server device 20 via the communication unit 33. The server device 20 receives the first image 11 and the visible light image and depth image of the first user 61 from the first terminal device 30 via the network 50. The server device 20 transmits the received first image 11 and the visible light image and depth image of the first user 61 to the second terminal device 40 via the network 50. When the server device 20 receives the voice of the first user 61 from the first terminal device 30, it transmits the received voice to the second terminal device 40 via the network 50. The control unit 41 of the second terminal device 40 receives the first image 11 and the visible light image and depth image of the first user 61 from the server device 20 via the communication unit 43. The control unit 41 may further receive the voice of the first user 61 from the server device 20 via the communication unit 43.

[0053] The control unit 41 of the second terminal device 40 captures a visible light image and a depth image of the second user 71 using a visible light camera and a depth camera as the input unit 44, respectively. The control unit 41 transmits the captured visible light image and depth image to the server device 20 via the communication unit 43. The control unit 41 may acquire the voice of the second user 71 via a microphone as the input unit 44, and further transmit the acquired voice to the server device 20 via the communication unit 43. The server device 20 receives the visible light image and the depth image of the second user 71 from the second terminal device 40 via the network 50. The server device 20 transmits the received visible light image and the depth image to the first terminal device 30 via the network 50. When the server device 20 receives the voice of the second user 71 from the second terminal device 40, it transmits the received voice to the first terminal device 30 via the network 50. The control unit 31 of the first terminal device 30 receives the visible light image and the depth image of the second user 71 from the server device 20 via the communication unit 33. The control unit 31 may further receive the voice of the second user 71 from the server device 20 via the communication unit 33.

[0054] In step S102, the control unit 41 of the second terminal device 40 places a virtual light source 81, a display object 82 corresponding to the first display 36, and a user object 83 corresponding to the first user 61 in the virtual three-dimensional space 80. The user object 83 includes an object representing the arm of the first user 61. The user object 83 specifically includes objects representing each body part of the upper body of the first user 61, such as both arms and the head of the first user 61. The control unit 41 generates an image of a shadow 84 of the user object 83 cast on the display object 82 as the second image 12 based on the positional relationship between the virtual light source 81, the display object 82, and the user object 83. The second image 12 includes an image of a shadow of an object representing the arm of the first user 61. The second image 12 specifically includes an image of a shadow of an object representing each body part of the upper body of the first user 61. The processing of step S102 is specifically executed in the following procedure.

[0055] The control unit 41 of the second terminal device 40 refers to the depth image of the first user 61 received in step S101 to specify the relative position of the first user 61 with respect to the first display 36. The control unit 41 may specify the relative position of the first user 61 with reference to information on the distance between the first display 36 and a depth camera used to capture the depth image of the first user 61, in addition to the depth image of the first user 61. The information on the distance may be stored in advance in the storage unit 42 of the second terminal device 40, or may be appropriately notified from the first terminal device 30 to the second terminal device 40. If the depth camera is integrated with the first display 36 or installed in the vicinity of the first display 36, the control unit 41 may specify the relative position of the first user 61 with reference only to the depth image of the first user 61. The control unit 41 sets the relative position of the user object 83 with respect to the display object 82 according to the specified relative position. More specifically, the control unit 41 places the display object 82 and the user object 83 in the virtual three-dimensional space 80 so that the relative positions of the objects representing each body part of the upper body of the first user 61 included in the user object 83 with respect to the display object 82 match the relative positions of each body part of the upper body of the first user 61 with respect to the first display 36.

[0056] Control unit 41 of second terminal device 40 sets a relative position of virtual light source 81 with respect to display object 82, depending on a relative position of light source 72 installed in second space 70 with respect to second display 46. More specifically, control unit 41 arranges virtual light source 81 and display object 82 in virtual three-dimensional space 80 such that the relative position of virtual light source 81 with respect to display object 82 matches the relative position of light source 72 with respect to second display 46. Alternatively, control unit 41 may arrange virtual light source 81 and display object 82 in virtual three-dimensional space 80 such that the relative position of virtual light source 81 with respect to display object 82 becomes a relative position set in advance.

[0057] The control unit 41 of the second terminal device 40 calculates the shadows of objects representing each body part of the upper body of the first user 61, which are included in the user object 83 and cast on the surface of the display object 82 corresponding to the horizontal display surface 38 of the first display 36, based on the position of the virtual light source 81 in the virtual three-dimensional space 80. The method of calculating the shadows can be the same as that used in known 3D computer graphics. The control unit 41 generates an image of the calculated shadow as the second image 12.

[0058] In step S103, the control unit 41 of the second terminal device 40 displays the first image 11 received in step S101 on the second display 46 serving as the output unit 45 toward the second user 62, and also superimposes the second image 12 generated in step S102 on the first image 11 on the second display 46. The process of step S103 is specifically executed in the following procedure.

[0059] The control unit 41 of the second terminal device 40 displays the first image 11 on the horizontal display surface 48 of the second display 46, and also displays the second image 12 on the horizontal display surface 48 of the second display 46 in a form in which the second image 12 is superimposed on the first image 11. The control unit 41 may render the user object 83 by referring to the visible light image of the first user 61 received in step S101, and then display, on the vertical display surface 47 of the second display 46, an image of the user object 83 captured by a virtual camera arranged at a position capturing the user object 83 from the front in the virtual three-dimensional space 80. When the control unit 41 receives the voice of the first user 61 in step S101, the control unit 41 may output the received voice from a speaker serving as the output unit 45.

[0060] When displaying the first image 11 on the second display 46, the control unit 41 of the second terminal device 40 may display the image of the characters included in the first image 11 as is, but in this embodiment, the image of the characters is displayed in a color other than black, such as vermilion. More specifically, the control unit 41 identifies an unchanging portion on the image of paper included in the first image 11 as a character portion based on the change over time of the first image 11. The control unit 41 converts the color of the identified character portion to a color other than black, such as vermilion. When displaying the first image 11 on the horizontal display surface 48 of the second display 46, the control unit 41 displays an image after converting the color of the character portion. According to this embodiment, by displaying the character portion in a color other than black, such as vermilion, it is possible to avoid erroneous recognition of characters due to the shadow of a hand falling on the character portion. The control unit 31 of the first terminal device 30 may identify the character portion, or identify the character portion and convert the color of the character portion, instead of the control unit 41 of the second terminal device 40.

[0061] In step S104, the control unit 41 of the second terminal device 40 receives the latest first image 11 and the latest visible light image and depth image of the first user 61 from the server device 20 via the communication unit 43, and transmits the latest visible light image and depth image of the second user 62 to the server device 20 via the communication unit 43. The control unit 41 may further receive the latest voice of the first user 61 from the server device 20 via the communication unit 43. The control unit 41 may further transmit the latest voice of the second user 71 to the server device 20 via the communication unit 43. The specific procedure of the process of step S104 is similar to the process of step S101, and therefore will not be described.

[0062] The process of step S104 is executed every time the first image 11 is transmitted. In this embodiment, the first image 11 is transmitted from the first terminal device 30 when it is automatically changed or when it is changed by some operation performed by the first user 61, such as writing characters, but it may be transmitted from the first terminal device 30 periodically regardless of whether it has been changed.

[0063] In step S105, the control unit 41 of the second terminal device 40 refers to the latest depth image of the first user 61 received in step S104 to determine whether the relative position of the first user 61 with respect to the first display 36 has changed. The control unit 41 may determine whether the relative position of the first user 61 has changed by referring to information on the distance between the first display 36 and a depth camera used to capture the depth image of the first user 61 in addition to the depth image of the first user 61. If the depth camera is integrated with the first display 36 or installed near the first display 36, the control unit 41 may determine whether the relative position of the first user 61 has changed by referring only to the depth image of the first user 61.

[0064] If it is determined in step S105 that the relative position of the first user 61 has not changed, i.e., if a change in the relative position of the first user 61 has not been detected, the process of step S106 is executed. On the other hand, if it is determined in step S105 that the relative position of the first user 61 has changed, i.e., if a change in the relative position of the first user 61 has been detected, the process of step S107 is executed.

[0065] In step S106, the control unit 41 of the second terminal device 40 reflects only the update of the first image 11 on the second display 46 serving as the output unit 45. Specifically, the control unit 41 replaces the first image 11 displayed on the horizontal display surface 48 of the second display 46 with the latest first image 11 received in step S104, and superimposes the second image 12 displayed on the horizontal display surface 48 of the second display 46 on the latest first image 11. When replacing the first image 11 displayed on the horizontal display surface 48 of the second display 46 with the latest one, the control unit 41 displays the image of characters in a color other than black, such as vermilion, in the same manner as in step S103.

[0066] After step S106, the processes from step S104 onwards are executed again.

[0067] When the first image 11 is periodically transmitted from the first terminal device 30 regardless of whether it has been changed or not, the latest first image 11 received in step S104 may be the same as the previously received first image 11. In such a case, the process of step S106 may be omitted.

[0068] In step S107, the control unit 41 of the second terminal device 40 adjusts the relative position of the user object 83 with respect to the display object 82 in accordance with the change detected in step S105. Specifically, the control unit 41 updates the setting of the relative position of the user object 83 with respect to the display object 82 in accordance with the relative position after the change detected in step S105. More specifically, the control unit 41 moves, for a body part of the upper body of the first user 61 whose relative position with respect to the first display 36 has changed, an object representing the body part included in the user object 83 in the virtual three-dimensional space 80 such that the relative position of the object representing the body part with respect to the display object 82 matches the relative position of the body part with respect to the first display 36 after the change.

[0069] In step S108, control unit 41 of second terminal device 40 updates second image 12 based on a change in the positional relationship among virtual light source 81, display object 82, and user object 83 resulting from the adjustment in step S107. Specifically, control unit 41 calculates, similarly to step S102, shadows of objects representing each body part of the upper body of first user 61, which are included in user object 83 and cast on the surface of display object 82 corresponding to horizontal display surface 38 of first display 36, based on the position of virtual light source 81 in virtual three-dimensional space 80. Control unit 41 updates second image 12 with images of the calculated shadows.

[0070] In step S109, the control unit 41 of the second terminal device 40 reflects the update of the first image 11 and the update of the second image 12 in step S108 on the second display 46 serving as the output unit 45. Specifically, the control unit 41 replaces the first image 11 displayed on the horizontal display surface 48 of the second display 46 with the latest first image 11 received in step S104, and replaces the second image 12 displayed on the horizontal display surface 48 of the second display 46 with the latest shadow image calculated in step S108 by superimposing the latest first image 11 on the latest first image 11. When replacing the first image 11 displayed on the horizontal display surface 48 of the second display 46 with the latest one, the control unit 41 displays the character image in a color other than black, such as vermilion, in the same manner as in step S103. Control unit 41 may render user object 83 by referring to the latest visible light image of first user 61 received in step S104, and then display, on vertical display surface 47 of second display 46, an image of user object 83 captured by a virtual camera positioned to capture user object 83 from the front in virtual three-dimensional space 80. When control unit 41 receives the latest audio of first user 61 in step S104, it may output the received audio from a speaker serving as output unit 45.

[0071] After step S109, the processes from step S104 onwards are executed again.

[0072] When the first image 11 is periodically transmitted from the first terminal device 30 regardless of whether it has been changed, the latest first image 11 received in step S104 may be the same as the previously received first image 11. In such a case, reflection of the update of the first image 11 in step S109 may be omitted.

[0073] In this embodiment, the above-mentioned operation enables the second user 62 to visually recognize the operation of the first image 11 as if he / she were the first user 61. That is, according to this embodiment, the second user 62 can easily recognize the body movements of the first user 61. For example, by displaying the shadow of a body part such as the arm of a teacher in a remote location toward the students and linking this shadow to the actual body movements of the teacher, it becomes easier for the students to recognize the body movements of the teacher during a lesson involving body movements such as a calligraphy lesson.

[0074] The second embodiment will be outlined with reference to FIG. 5 and FIG. 6 in addition to FIG.

[0075] In FIG. 5, as in FIG. 3, the first space 60 and the second space 70 are adjacent to each other for convenience of explanation, but in reality, the first space 60 and the second space 70 are located at a distance from each other. The first user 61 operates the first image 11 displayed on the first display 36. In this embodiment, the first image 11 includes an image of a character handwritten by the first user 61, but may include other images, such as an image of a picture drawn by the first user 61, instead of or in addition to the image of the character. The first user 61 is, for example, a teacher in a remote location, and can take a lesson involving physical movements, such as a calligraphy lesson, by operating the first image 11 with a touch screen. The second user 62 visually checks the operation of the first image 11 by the first user 61. The second user 62 is, for example, a student, and can take a lesson involving physical movements, such as a calligraphy lesson, by visually checking the operation of the first image 11.

[0076] The second terminal device 40 places a first virtual camera 85 and a user object 83 corresponding to the first user 61 in a virtual three-dimensional space 80 as shown in FIG. 6. In this embodiment, the user object 83 includes an object representing the arm of the first user 61, but may include other objects. In this embodiment, the first virtual camera 85 is placed at a position where the object representing the arm of the first user 61 is captured from above, but may be placed at other positions. The second terminal device 40 generates an image of the user object 83 captured by the first virtual camera 85 as a second image 13 based on the positional relationship between the first virtual camera 85 and the user object 83 and the direction of the first virtual camera 85. In this embodiment, the second image 13 includes an image of the shadow of the object representing the arm of the first user 61, but may include an image of the shadow of another object. The second terminal device 40 displays the first image 11 on the second display 46 toward the second user 62 and superimposes the second image 13 on the first image 11 on the second display 46.

[0077] When the second terminal device 40 detects a change in the line of sight of the second user 62, it adjusts the direction of the first virtual camera 85 in response to the detected change. The second terminal device 40 updates the second image 13 based on the adjusted direction of the first virtual camera 85. The second terminal device 40 reflects the update of the second image 13 on the second display 46.

[0078] In this embodiment, since an image of a user object 83 corresponding to the first user 61 is displayed on the second display 46, the second user 62 can visually recognize the operation of the first image 11 as if he / she were the first user 61. That is, according to this embodiment, it becomes easier for the second user 62 to recognize the body movement of the first user 61. For example, by displaying a 3D image of a body part such as an arm of a teacher in a remote location to the students and linking this 3D image to the actual body movement of the teacher, it becomes easier for the students to recognize the body movement of the teacher during a lesson involving body movement such as a calligraphy lesson.

[0079] In this embodiment, the second terminal device 40 further places a display object 82 corresponding to the first display 36 in the virtual three-dimensional space 80. When the second terminal device 40 detects a change in the relative position of the first user 61 with respect to the first display 36, the second terminal device 40 adjusts the relative position of the user object 83 with respect to the display object 82 in accordance with the detected change. The second terminal device 40 updates the second image 12 based on the change in positional relationship resulting from the adjustment. The second terminal device 40 reflects the update of the second image 12 on the second display 46.

[0080] Like the first embodiment, this embodiment can be applied to a remote dialogue system such as a video conference where a user can have a dialogue while viewing a video of the dialogue partner. For example, when using a remote dialogue system to realize a service in which a student receives a lesson involving arm movements, such as a calligraphy lesson, from a teacher in a remote location, two remote dialogue terminal devices each having a desk-type display are connected online. A 3D image such as a hologram of the teacher's arm on the teacher's display is rendered from a pseudo first-person perspective, and the 3D image is displayed on the student's display, allowing the student to visually recognize the teacher's arm movements as if they were his or her own arm movements.

[0081] The first terminal device 30 corresponds to one of the two remote dialogue terminal devices. The first terminal device 30 displays a first image 11 on a horizontal display surface 38 of a first display 36. The first terminal device 30 generates a visible light image of the first user 61 using a visible light camera. The first terminal device 30 generates a depth image of the first user 61 using a depth camera.

[0082] The second terminal device 40 corresponds to the other of the two remote dialogue terminal devices. The second terminal device 40 receives the visible light image and the depth image from the first terminal device 30. The second terminal device 40 places the 3D object of the dialogue partner in the virtual three-dimensional space 80 based on the received visible light image and depth image. The second terminal device 40 generates a visible light image of the second user 62 using a visible light camera. The second terminal device 40 generates a depth image of the second user 62 using a depth camera. The second terminal device 40 detects the line of sight of the second user 62 based on the visible light image, the depth image, or both. The second terminal device 40 links the direction of the first virtual camera 85 in the virtual three-dimensional space 80 to the line of sight of the second user 62. The second terminal device 40 generates a rendering image of the 3D object as the second image 13 using the first virtual camera 85. The second terminal device 40 displays the first image 11 displayed on the first display 36 on the horizontal display surface 48 of the second display 46, and also displays the second image 13 on the horizontal display surface 48 of the second display 46 in a form in which the second image 13 is superimposed on the first image 11. This allows the second user 62 to have the illusion that he or she is the conversation partner. For example, a rendering image is displayed on the top surface of a desk-type display as if the student were looking at a 3D model of the teacher, who is the conversation partner, from a first-person perspective, so that the student can visually recognize the movement of the teacher's arms when he or she is actually writing letters as if he or she were the teacher himself or herself.

[0083] In this embodiment, the second terminal device 40 also displays an image on the vertical display surface 47 of the second display 46. Specifically, the second terminal device 40 displays, as the third image 14, an image of the user object 83 captured by the second virtual camera 86 disposed at a position capturing the user object 83 from the front in the virtual three-dimensional space 80 on the vertical display surface 47. Therefore, the second user 62 can check the behavior of the first user 61 on the horizontal display surface 48 while checking the state of the first user 61 on the vertical display surface 47.

[0084] The operation of the second terminal device 40 according to the present embodiment will be described with reference to Fig. 7. This operation corresponds to the image display method according to the present embodiment.

[0085] The flow shown in FIG. 7 is repeatedly executed while the first user 61 and the second user 71 are performing video communication such as a web conference using the first terminal device 30 and the second terminal device 40, respectively.

[0086] The process of step S201 is similar to the process of step S101 in the first embodiment, and therefore a description thereof will be omitted.

[0087] In step S202, the control unit 41 of the second terminal device 40 places a first virtual camera 85, a second virtual camera 86, a display object 82 corresponding to the first display 36, and a user object 83 corresponding to the first user 61 in the virtual three-dimensional space 80. The user object 83 includes an object representing the arm of the first user 61. Specifically, the user object 83 includes objects representing each body part of the upper body of the first user 61, such as both arms and the head of the first user 61. The first virtual camera 85 is placed at a position to capture the object representing the arm of the first user 61 from above. Specifically, the first virtual camera 85 is placed at a position to capture the object representing the arm of the first user 61 from a pseudo first-person perspective. The second virtual camera 86 is placed at a position to capture the user object 83 from the front in the virtual three-dimensional space 80. The control unit 41 generates an image of the user object 83 captured by the first virtual camera 85 as the second image 13 based on the positional relationship between the first virtual camera 85 and the user object 83 and the direction of the first virtual camera 85. The second image 13 includes an image of an object representing the arm of the first user 61 viewed from above. Specifically, the second image 13 includes an image of the arm of the first user 61 as if viewed from a first-person perspective. The control unit 41 further generates an image of the user object 83 captured by the second virtual camera 86 as the third image 14 based on the positional relationship between the second virtual camera 86 and the user object 83 and the direction of the second virtual camera 86. The third image 14 includes an image of an object representing each body part of the upper body of the first user 61 viewed from the front. Specifically, the third image 14 includes an image of the entire upper body of the first user 61 viewed from the front. The process of step S202 is specifically executed in the following procedure.

[0088] The control unit 41 of the second terminal device 40 refers to the depth image of the first user 61 received in step S201 to specify the relative position of the first user 61 with respect to the first display 36. The control unit 41 may specify the relative position of the first user 61 with reference to information on the distance between the first display 36 and a depth camera used to capture the depth image of the first user 61, in addition to the depth image of the first user 61. The information on the distance may be stored in advance in the storage unit 42 of the second terminal device 40, or may be appropriately notified from the first terminal device 30 to the second terminal device 40. If the depth camera is integrated with the first display 36 or installed in the vicinity of the first display 36, the control unit 41 may specify the relative position of the first user 61 with reference only to the depth image of the first user 61. The control unit 41 sets the relative position of the user object 83 with respect to the display object 82 according to the specified relative position. More specifically, the control unit 41 places the display object 82 and the user object 83 in the virtual three-dimensional space 80 so that the relative positions of the objects representing each body part of the upper body of the first user 61 included in the user object 83 with respect to the display object 82 match the relative positions of each body part of the upper body of the first user 61 with respect to the first display 36.

[0089] The control unit 41 of the second terminal device 40 places the first virtual camera 85 in the vicinity of an object representing the head, face, or eyes of the first user 61, which is included in the user object 83. The control unit 41 sets the direction of the first virtual camera 85 so that the object representing the arm of the first user 61, which is included in the user object 83, and the display object 82 are within the shooting range of the first virtual camera 85.

[0090] The control unit 41 of the second terminal device 40 places the second virtual camera 86 in front of the user object 83. The control unit 41 sets the direction of the second virtual camera 86 so that objects representing each body part of the upper body of the first user 61, which are included in the user object 83, are within the shooting range of the second virtual camera 86.

[0091] The control unit 41 of the second terminal device 40 generates, as the second image 13, a 3D image of an object representing the arm of the first user 61, which is included in the user object 83 and is captured by the first virtual camera 85 in the virtual three-dimensional space 80. The method of generating the 3D image may be the same as that of known 3D computer graphics. When generating the second image 13, the control unit 41 may render the object representing the arm of the first user 61 by referring to the visible light image of the first user 61 received in step S201.

[0092] The control unit 41 of the second terminal device 40 generates, as the third image 14, 3D images of objects representing each body part of the upper body of the first user 61, which are included in the user object 83 and are captured by the second virtual camera 86 in the virtual three-dimensional space 80. The same method as that used for known 3D computer graphics can be used as a method for generating the 3D image. When generating the third image 14, the control unit 41 may render the objects representing each body part of the upper body of the first user 61 by referring to the visible light image of the first user 61 received in step S201.

[0093] In step S203, the control unit 41 of the second terminal device 40 displays the first image 11 received in step S201 on the second display 46 serving as the output unit 45 toward the second user 62, and also superimposes the second image 13 generated in step S202 on the first image 11 on the second display 46. The control unit 41 also displays the third image 14 generated in step S202 on the second display 46 toward the second user 62. The process of step S203 is specifically executed in the following procedure.

[0094] The control unit 41 of the second terminal device 40 displays the first image 11 on the horizontal display surface 48 of the second display 46, and also displays the second image 13 on the horizontal display surface 48 of the second display 46 in a form in which the second image 13 is superimposed on the first image 11. The control unit 41 displays the third image 14 on the vertical display surface 47 of the second display 46. When the control unit 41 receives the voice of the first user 61 in step S201, the control unit 41 may output the received voice from a speaker serving as the output unit 45.

[0095] When displaying the first image 11 on the second display 46, the control unit 41 of the second terminal device 40 may display the image of the characters included in the first image 11 as is, but in this embodiment, as in the first embodiment, the image of the characters is displayed in a color other than black, such as vermilion. More specifically, the control unit 41 identifies an unchanging part on the image of paper included in the first image 11 as a character part based on the change over time of the first image 11. The control unit 41 converts the color of the identified character part to a color other than black, such as vermilion. When displaying the first image 11 on the horizontal display surface 48 of the second display 46, the control unit 41 displays an image after converting the color of the character part. According to this embodiment, by displaying the character part in a color other than black, such as vermilion, it is possible to avoid erroneous recognition of characters due to the shadow of a hand falling on the character part. The identification of the character part, or the identification of the character part and the color conversion of the character part may be performed by the control unit 31 of the first terminal device 30 instead of the control unit 41 of the second terminal device 40.

[0096] The processes in steps S204 and S205 are similar to those in steps S104 and S105 in the first embodiment, respectively, and therefore will not be described.

[0097] If it is determined in step S205 that the relative position of the first user 61 has not changed, i.e., if no change in the relative position of the first user 61 has been detected, the process of step S206 is executed. On the other hand, if it is determined in step S205 that the relative position of the first user 61 has changed, i.e., if a change in the relative position of the first user 61 has been detected, the process of step S208 is executed.

[0098] In step S206, the control unit 41 of the second terminal device 40 captures a visible light image and a depth image of the second user 71 using the visible light camera and the depth camera as the input unit 44, respectively. The control unit 41 determines whether the line of sight of the second user 62 has changed by referring to the captured visible light image, the depth image, or both. A known method can be used as a method for detecting the line of sight by image recognition. Machine learning such as deep learning may also be used.

[0099] If it is determined in step S206 that the line of sight of the second user 62 has not changed, i.e., if a change in the line of sight of the second user 62 has not been detected, the process of step S207 is executed. On the other hand, if it is determined in step S206 that the line of sight of the second user 62 has changed, i.e., if a change in the line of sight of the second user 62 has been detected, the process of step S210 is executed.

[0100] In step S207, the control unit 41 of the second terminal device 40 reflects only the update of the first image 11 on the second display 46 serving as the output unit 45. Specifically, the control unit 41 replaces the first image 11 displayed on the horizontal display surface 48 of the second display 46 with the latest first image 11 received in step S204, and superimposes the second image 13 displayed on the horizontal display surface 48 of the second display 46 on the latest first image 11. When replacing the first image 11 displayed on the horizontal display surface 48 of the second display 46 with the latest one, the control unit 41 displays the image of characters in a color other than black, such as vermilion, in the same manner as in step S203.

[0101] After step S207, the processes from step S204 onwards are executed again.

[0102] When the first image 11 is periodically transmitted from the first terminal device 30 regardless of whether it has been changed, the latest first image 11 received in step S204 may be the same as the previously received first image 11. In such a case, the process of step S207 may be omitted.

[0103] The process of step S208 is similar to the process of step S107 in the first embodiment, and therefore a description thereof will be omitted.

[0104] After step S208, the process of step S209 is executed. The process of step S209 is similar to the process of step S206, and therefore the description thereof will be omitted.

[0105] If it is determined in step S209 that the line of sight of the second user 62 has not changed, that is, if a change in the line of sight of the second user 62 has not been detected, the process of step S211 is executed. On the other hand, if it is determined in step S209 that the line of sight of the second user 62 has changed, that is, if a change in the line of sight of the second user 62 has been detected, the process of step S210 is executed.

[0106] In step S210, control unit 41 of second terminal device 40 adjusts the direction of first virtual camera 85 in response to the change detected in step S206 or step S209. Specifically, control unit 41 changes the direction of first virtual camera 85 in the same direction and by the same amount as the change detected in step S206 or step S209.

[0107] In step S211, when the process of step S208 has been executed, the control unit 41 of the second terminal device 40 updates the second image 13 based on the change in the positional relationship between the first virtual camera 85 and the user object 83 resulting from the adjustment of step S208. When the process of step S210 has been executed, the control unit 41 updates the second image 13 based on the direction of the first virtual camera 85 after the adjustment of step S210. When the processes of both steps S208 and S210 have been executed, the control unit 41 updates the second image 13 based on the change in the positional relationship between the first virtual camera 85 and the user object 83 and the direction of the first virtual camera 85 after the adjustment. Specifically, the control unit 41 updates the second image 13 with a 3D image of an object representing the arm of the first user 61, which is included in the user object 83 and is photographed by the first virtual camera 85 in the virtual three-dimensional space 80, in the same manner as in step S202. When updating second image 13, control unit 41 may render an object representing the arm of first user 61 by referring to the latest visible light image of first user 61 received in step S204.

[0108] When the process of step S208 has been executed, the control unit 41 of the second terminal device 40 also updates the third image 14 based on a change in the positional relationship between the second virtual camera 86 and the user object 83 resulting from the adjustment of step S208. Specifically, the control unit 41 updates the third image 14 with 3D images of objects representing each body part of the upper body of the first user 61 and included in the user object 83, photographed by the second virtual camera 86 in the virtual three-dimensional space 80, in the same manner as in step S202. When updating the third image 14, the control unit 41 may render the objects representing each body part of the upper body of the first user 61 by referring to the latest visible light image of the first user 61 received in step S204.

[0109] In step S212, the control unit 41 of the second terminal device 40 reflects the update of the first image 11 and the update of the second image 13 in step S211 on the second display 46 as the output unit 45. Specifically, the control unit 41 replaces the first image 11 displayed on the horizontal display surface 48 of the second display 46 with the latest first image 11 received in step S204, and replaces the second image 13 displayed on the horizontal display surface 48 of the second display 46 with the 3D image obtained in step S211 and captured by the first virtual camera 85 in such a manner that the 3D image is superimposed on the latest first image 11. When replacing the first image 11 displayed on the horizontal display surface 48 of the second display 46 with the latest image, the control unit 41 displays the image of the characters in a color other than black, such as vermilion, in the same manner as in step S203. When control unit 41 receives the latest voice of first user 61 in step S204, it may output the received voice from a speaker serving as output unit 45.

[0110] If the third image 14 is also updated in step S211, the control unit 41 of the second terminal device 40 also reflects the update of the third image 14 in step S211 on the second display 46. Specifically, the control unit 41 replaces the third image 14 displayed on the vertical display surface 47 of the second display 46 with the 3D image captured by the second virtual camera 86 obtained in step S211.

[0111] After step S212, the processes from step S204 onwards are executed again.

[0112] When the first image 11 is periodically transmitted from the first terminal device 30 regardless of whether it has been changed, the latest first image 11 received in step S204 may be the same as the previously received first image 11. In such a case, reflection of the update of the first image 11 in step S212 may be omitted.

[0113] In this embodiment, the above-mentioned operation enables the second user 62 to visually recognize the operation of the first image 11 as if he / she were the first user 61. That is, according to this embodiment, it becomes easier for the second user 62 to recognize the body movements of the first user 61. For example, by displaying a 3D image of a body part such as the arm of a teacher in a remote location to the students and linking this 3D image to the actual body movements of the teacher, it becomes easier for the students to recognize the body movements of the teacher during a lesson involving body movements such as a calligraphy lesson.

[0114] Some of the embodiments of the present disclosure will be described below as examples. However, it should be noted that the embodiments of the present disclosure are not limited to these examples. [Appendix 1] a control unit that generates, based on a positional relationship between a virtual light source arranged in a virtual three-dimensional space, a display object corresponding to a first display, and a user object corresponding to a first user operating a first image displayed on the first display, an image of a shadow of the user object cast on the display object as a second image, displays the first image on a second display for a second user in a second space separate from a first space in which the first user is located, and superimposes the second image on the first image on the second display, and upon detecting a change in a relative position of the first user with respect to the first display, adjusts a relative position of the user object with respect to the display object in accordance with the detected change, updates the second image based on the change in the positional relationship resulting from the adjustment, and reflects the update of the second image on the second display. [Appendix 2] A communication unit that communicates with another terminal device used by the first user, The control unit receives a depth image of the first user via the communication unit, and identifies the relative position of the first user with respect to the first display by referring to the received depth image. [Appendix 3] The terminal device according to claim 1 or 2, wherein the control unit sets a relative position of the virtual light source with respect to the display object in accordance with a relative position of a light source installed in the second space with respect to the second display. [Appendix 4] The terminal device according to any one of Supplementary Note 1 to Supplementary Note 3, wherein the user object includes an object representing an arm of the first user. [Appendix 5] a display having an L-shape in a side view and corresponding to the second display, the display having a vertical display surface extending in a vertical direction and a horizontal display surface adjacent to a lower end of the vertical display surface and extending in a horizontal direction; 5. The terminal device according to claim 1, wherein the control unit displays the first image and the second image on the horizontal display surface. [Appendix 6] The terminal device according to any one of Supplementary Note 1 to Supplementary Note 5, wherein the first image includes an image of characters handwritten by the first user. [Appendix 7] The terminal device according to claim 6, wherein the control unit displays the image of the text in a color other than black when displaying the first image on the second display. [Appendix 8] The terminal device according to claim 7, wherein the control unit displays the image of the character in vermilion. [Appendix 9] A terminal device comprising: a control unit that generates an image of a user object captured by a virtual camera as a second image based on a positional relationship between a virtual camera arranged in a virtual three-dimensional space and a user object corresponding to a first user manipulating a first image displayed on a first display and a direction of the virtual camera, displays the first image on a second display toward a second user in a second space separate from a first space in which the first user is located, and superimposes the second image on the first image on the second display, and upon detecting a change in the line of sight of the second user, adjusts the direction of the virtual camera in accordance with the detected change, updates the second image based on the adjusted direction of the virtual camera, and reflects the update of the second image on the second display. [Appendix 10] The terminal device described in Appendix 9, wherein the control unit adjusts a relative position of the user object with respect to a display object corresponding to the first display arranged in the virtual three-dimensional space in response to a change in the relative position of the first user with respect to the first display, updates the second image based on the change in the positional relationship resulting from the adjustment, and reflects the update of the second image on the second display. [Appendix 11] A communication unit that communicates with another terminal device used by the first user, The control unit receives a depth image of the first user via the communication unit, and identifies the relative position of the first user with respect to the first display by referring to the received depth image. [Appendix 12] the user object includes an object representing an arm of the first user, 12. The terminal device according to any one of Supplementary Note 9 to Supplementary Note 11, wherein the virtual camera is positioned at a position to capture an object representing the arm of the first user from above. [Appendix 13] a display having an L-shape in a side view and corresponding to the second display, the display having a vertical display surface extending in a vertical direction and a horizontal display surface adjacent to a lower end of the vertical display surface and extending in a horizontal direction; 12. The terminal device according to claim 9, wherein the control unit displays the first image and the second image on the horizontal display surface. [Appendix 14] The terminal device described in Appendix 13, wherein when the virtual camera is a first virtual camera and another virtual camera positioned to capture the user object from the front in the virtual three-dimensional space is a second virtual camera, the control unit generates an image of the user object captured by the second virtual camera as a third image based on a positional relationship between the second virtual camera and the user object and a direction of the second virtual camera, and displays the third image on the vertical display screen. [Appendix 15] The terminal device according to any one of Supplementary Note 9 to Supplementary Note 14, wherein the first image includes an image of characters handwritten by the first user. [Appendix 16] The terminal device according to claim 15, wherein the control unit displays the image of the text in a color other than black when displaying the first image on the second display. [Appendix 17] The terminal device according to claim 16, wherein the control unit displays the image of the character in vermilion. [Appendix 18] generating, as a second image, an image of a shadow of the user object cast on the display object or an image of the user object captured by the virtual camera based on a positional relationship between a virtual light source arranged in a virtual three-dimensional space, a display object corresponding to a first display, and a user object corresponding to a first user who operates a first image displayed on the first display, or based on a positional relationship between a virtual camera arranged in the virtual three-dimensional space and the user object, and a direction of the virtual camera; displaying the first image on a second display for a second user in a second space separate from the first space in which the first user is located; superimposing the second image on the first image on the second display; when detecting a change in a relative position of the first user with respect to the first display or a change in a line of sight of the second user, adjusting a relative position of the user object with respect to the display object or a direction of the virtual camera in response to the detected change; updating the second image based on a change in the positional relationship resulting from the adjustment or a direction of the virtual camera after the adjustment; reflecting the update of the second image on the second display; and An image display method comprising: [Appendix 19] A program that causes a computer to execute an operation corresponding to the image display method described in appendix 18.

[0115] The present disclosure is not limited to the above-mentioned embodiment. For example, two or more blocks shown in the block diagram may be integrated, or one block may be divided. Instead of executing two or more steps shown in the flowchart in chronological order as described, they may be executed in parallel or in a different order depending on the processing capacity of the device executing each step, or as necessary. Other modifications are possible without departing from the spirit of the present disclosure. [Explanation of symbols]

[0116] 10. Communication Systems 11 First image 12,13 2nd image 14 3rd image 20 Server device 30 First terminal device 31 Control Unit 32 Storage section 33 Communications Department 34 Input section 35 Output section 36 1st Display 38 Horizontal display surface 40 Second terminal device 41 Control section 42 Storage section 43 Communications Department 44 Input section 45 Output section 46 Second Display 47 Vertical display surface 48 Horizontal display surface 50 Network 60 1st space 61 First User 70 Second space 71 Second User 72 Light source 80 Virtual 3D Space 81 Virtual Light Source 82 Display Objects 83 User Object 84 Shadow 85 1st Virtual Camera 86 Second Virtual Camera

Claims

1. A terminal device comprising: a control unit that generates, based on a positional relationship between a virtual light source arranged in a virtual three-dimensional space, a display object corresponding to a first display, and a user object corresponding to a first user operating a first image displayed on the first display, an image of a shadow of the user object that can be cast on the display object as a second image; displays the first image on a second display for a second user in a second space separate from a first space in which the first user is located, and superimposes the second image on the first image on the second display; and, upon detecting a change in a relative position of the first user with respect to the first display, adjusts a relative position of the user object with respect to the display object in accordance with the detected change, updates the second image based on the change in the positional relationship resulting from the adjustment, and reflects the update of the second image on the second display.

2. A communication unit that communicates with another terminal device used by the first user, The terminal device according to claim 1 , wherein the control unit receives a depth image of the first user via the communication unit, and identifies a relative position of the first user with respect to the first display by referring to the received depth image.

3. The terminal device according to claim 1 , wherein the control unit sets a relative position of the virtual light source with respect to the display object in accordance with a relative position of a light source installed in the second space with respect to the second display.

4. The terminal device according to claim 1 , wherein the user object includes an object representing an arm of the first user.

5. a display having an L-shape in a side view and corresponding to the second display, the display having a vertical display surface extending in a vertical direction and a horizontal display surface adjacent to a lower end of the vertical display surface and extending in a horizontal direction; The terminal device according to claim 1 , wherein the control unit displays the first image and the second image on the horizontal display surface.

6. The terminal device according to claim 1 , wherein the first image includes an image of characters handwritten by the first user.

7. The terminal device according to claim 6 , wherein the control unit displays the image of the characters in a color other than black when displaying the first image on the second display.

8. The terminal device according to claim 7 , wherein the control unit displays the image of the characters in vermilion.

Images