Image generation device, image generation method, and program

Abstract

To realize better communication with a remote person.SOLUTION: An image generation device comprises: an acquisition section for acquiring a user image including a two-dimensional image of the contour of a user in a first space and terminal information that is information showing the position and direction of a terminal device in a second space that differs from the first space; a position setting section for setting a viewpoint position in the virtual space on the basis of the terminal information; an image synthesis section for synthesizing a synthesis image that is a two-dimensional image projecting the user image from the viewpoint position in a state in which the user image is located in the virtual space; and an output control section for outputting the synthesis image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image generation device, an image generation method, and a program. [Background technology]

[0002] Many methods for communicating with people in remote locations have been studied. Patent Document 1 describes a communication system that allows a user to communicate with an agent in a remote location while virtually sharing a space, by acquiring an image of an agent in a second space that is different from a first space in which the user exists, and position information of the agent in the second space using a sensor installed in the second space, and projecting the image of the agent at a predetermined position in the first space based on the position information of the agent in the second space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-157959 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, the acquired image of the agent is projected directly into the first space, but there is room for improvement in the way the image is displayed in order to better communicate with people in remote locations.

[0005] The present disclosure has been made in consideration of the above, and aims to provide an image generation device, an image generation method, and a program that can realize better communication with people in remote locations. [Means for solving the problem]

[0006] The image generating device according to the present disclosure includes an acquisition unit that acquires a user image including a two-dimensional image of the user's outline in a first space and terminal information that is information indicating the position and orientation of a terminal device in a second space different from the first space, a position setting unit that sets a viewpoint position in a virtual space based on the terminal information, an image synthesis unit that generates a composite image that is a two-dimensional image obtained by projecting the user image from the viewpoint position while the user image is located in the virtual space, and an output control unit that outputs the composite image.

[0007] The image generation method of the present disclosure includes the steps of acquiring a user image including a two-dimensional image of the contour of a first space and terminal information that is information indicating the position and orientation of a terminal device in a second space different from the first space, setting a viewpoint position in a virtual space based on the terminal information, generating a composite image that is a two-dimensional image in which the user image is projected from the viewpoint position while the user image is located in the virtual space, and outputting the composite image.

[0008] The program of the present disclosure causes a computer to execute the steps of acquiring a user image including a two-dimensional image of the contour of a first space and terminal information that is information indicating the position and orientation of a terminal device in a second space different from the first space, setting a viewpoint position in a virtual space based on the terminal information, generating a composite image that is a two-dimensional image in which the user image is projected from the viewpoint position while the user image is located in the virtual space, and outputting the composite image. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to provide an image generation device, an image generation method, and a program that can realize better communication with people in remote locations. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram of an image generation system according to the first embodiment. [Figure 2] FIG. 2 is a schematic block diagram of the imaging device according to the first embodiment. [Figure 3] FIG. 3 is a schematic block diagram of the image generating device according to the first embodiment. [Figure 4] FIG. 4 is a schematic diagram showing example 1 of the virtual space according to the first embodiment. [Figure 5] FIG. 5 is a schematic diagram illustrating Example 1 of a composite image according to the first embodiment. [Figure 6] FIG. 6 is a schematic diagram illustrating a second example of the virtual space according to the first embodiment. [Figure 7] FIG. 7 is a schematic diagram illustrating Example 2 of a composite image according to the first embodiment. [Figure 8] FIG. 8 is a flowchart illustrating the processing flow of the image generating device according to the first embodiment. [Figure 9] FIG. 9 is a schematic diagram of an image generation system according to the second embodiment. [Figure 10] FIG. 10 is a schematic block diagram of a terminal device according to the second embodiment. [Figure 11] FIG. 11 is a schematic diagram showing example 1 of a virtual space according to the second embodiment. [Figure 12] FIG. 12 is a schematic diagram illustrating the field of view according to the second embodiment. [Figure 13] FIG. 13 is a schematic diagram illustrating Example 1 of a composite image according to the second embodiment. [Figure 14] FIG. 14 is a schematic diagram illustrating Example 2 of a composite image according to the second embodiment. [Figure 15] FIG. 15 is a flowchart illustrating the processing flow of the image generating device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that the present disclosure is not limited to these embodiments, and when there are multiple embodiments, the present disclosure also includes configurations in which the respective embodiments are combined.

[0012] (First embodiment) The configuration of the image generation system will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of the image generation system according to the first embodiment.

[0013] (Image Generation System Overview) 1, the image generation system 1 according to the first embodiment is a system that outputs a composite image CI (described later) generated based on a user image YI, which is an image of a first user U1 present in a first space R1, to a second space R2 different from the first space. The image generation system 1 according to this embodiment is a system that realizes better communication with people in remote locations by outputting a composite image CI generated based on the user image YI of the first user U1 present in the first space R1 to the second space R2.

[0014] (Image generation system configuration) As shown in FIG. 1, the image generation system 1 includes an image generation device 100, an image capture device 200, and an output device .

[0015] (imaging device) The image capturing device 200 is a device that captures an image of the first user U1 present in the first space R1, converts the captured image into a user image YI that is a two-dimensional image showing the outline of the first user U1, and transmits it to the image generating device 100. The processing content of the image capturing device 200 will be described later.

[0016] (Image Generation Device) The image generating device 100 is a device that generates a composite image CI based on the acquired user image YI. The processing details of the image generating device 100 will be described later.

[0017] (output device) The output device 10 is a device that outputs the composite image CI generated by the image generation device 100 in the second space R2. The output device 10 may be any device, such as a projector or a display. The output device 10 according to this embodiment is a projector.

[0018] When the output device 10 is a projector, the output device 10 projects light for displaying the composite image CI onto a projection member in the second space R2. The composite image CI is displayed on the projection member. The projection member may be any member, and in this embodiment, it is a wall or floor of the second space R2. However, it is not limited to this and may be, for example, a screen provided in the second space R2. When the output device 10 is a display, the output device 10 displays the composite image CI on the display. The display may be provided on the entire wall surface of the second space R2, or on multiple surfaces.

[0019] (Configuration of imaging device) Next, the configuration of the imaging device 200 will be described.

[0020] 2 is a schematic block diagram of an image capturing device according to this embodiment. The image capturing device 200 captures an image of a first user U1 present in a first space R1. Furthermore, in this embodiment, the image capturing device 200 generates a user image YI, which is a two-dimensional image showing the outline of the first user U1, from the captured image of the first user U1.

[0021] 2, the image capturing device 200 includes a camera 210, a distance sensor 220, a storage unit 230, a control unit 240, and a communication unit 250. The camera 210 is a device that captures an image of the first space R1, and may be, for example, a device that includes an optical element such as a lens, and an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). The distance sensor 220 is a sensor that measures the distance between the image capturing device 200 and an object, and may be, for example, an optical sensor.

[0022] The storage unit 230 is a memory that stores the calculation contents and programs of the control unit 240, and includes, for example, at least one of a main storage device such as RAM (Random Access Memory) and ROM (Read Only Memory), and a non-volatile storage device such as a flash memory or SSD (Solid State Disk Drive).

[0023] The control unit 240 is a calculation device and includes a calculation circuit such as a CPU (Central Processing Unit). The control unit 240 includes an acquisition unit 241, a conversion unit 242, and an output control unit 243. The control unit 240 realizes the acquisition unit 241, the conversion unit 242, and the output control unit 243 by reading and executing a program (software) from the storage unit 230. The control unit 240 executes these processes using one CPU, or may be provided with multiple CPUs and execute the processes using the multiple CPUs. At least a portion of the acquisition unit 241, the conversion unit 242, and the output control unit 243 may be implemented by hardware.

[0024] The communication unit 250 is responsible for transmitting and receiving various types of data to and from external devices via wireless communication or wired communication. In the case of wireless communication, it may be equipped with a communication antenna, a radio frequency (RF) circuit, a communication processing circuit such as a wireless local area network (LAN) card, etc. In the case of wired communication, it may be equipped with, for example, a network interface card (NIC) equipped with a wired LAN terminal, a transmission circuit, other communication processing circuits, etc. The communication format according to this embodiment is wireless communication.

[0025] (Configuration of image generating device) Next, the image generating device 100 will be described.

[0026] 3 is a schematic block diagram of an image generating device according to this embodiment. The image generating device 100 according to this embodiment is a computer, and as shown in FIG. 3, has an input unit 110, a storage unit 130, a control unit 140, and a communication unit 150.

[0027] The input unit 110 is a device that inputs information to the image generating device 100 from an external device or from the user U, and is, for example, an input terminal, a keyboard, or a touch panel.

[0028] The storage unit 130 is a memory that stores the calculation contents and programs of the control unit 140, and includes at least one of a RAM, a main storage device such as a ROM, and a non-volatile storage device such as a flash memory or an SSD, for example.

[0029] The control unit 140 is a calculation device and includes a calculation circuit such as a CPU. The control unit 140 includes an acquisition unit 141, a position setting unit 142, an image composition unit 143, and an output control unit 144. The control unit 140 reads and executes a program (software) from the storage unit 130, thereby realizing the acquisition unit 141, the position setting unit 142, the image composition unit 143, and the output control unit 144 and performing their processing. The control unit 140 may perform these processes using a single CPU, or may be provided with multiple CPUs and perform the processes using the multiple CPUs. At least a portion of the acquisition unit 141, the position setting unit 142, the image composition unit 143, and the output control unit 144 may be realized by hardware.

[0030] The communication unit 150 is responsible for transmitting and receiving various types of data to and from external devices via wireless communication or wired communication. In the case of wireless communication, the communication unit 150 may be provided with a communication antenna, an RF circuit, a communication processing circuit such as a wireless LAN card, etc. In the case of wired communication, the communication unit 150 may be provided with, for example, a NIC with a wired LAN terminal, a transmission circuit, other communication processing circuits, etc. The communication format in this embodiment is wireless communication.

[0031] In the present embodiment, the image generation device 100, the photographing device 200, and the output device 10 are separate devices, but this is not limiting. For example, the image generation device 100 and the photographing device 200 may be a single device, in which case the image generation device 100 is disposed in the first space R1 and has both the functions of the image generation device 100 and the photographing device 200. Also, for example, the image generation device 100 and the output device 10 may be a single device, in which case the image generation device 100 is disposed in the second space R2 and has both the functions of the image generation device 100 and the output device 10.

[0032] (Image Generation System Processing) Next, the processing contents of the image generation system 1 will be described.

[0033] (Processing of imaging device) First, the processing performed by the image capturing device 200 will be described.

[0034] (Get user image) The acquisition unit 241 of the imaging device 200 controls the camera 210 to capture an image of the first space R1. The acquisition unit 241 acquires a captured image, which is an image captured by the camera 210. The captured image shows a first user U1 (user) who is present in the first space R1.

[0035] (Getting user location information) The acquisition unit 241 also controls the distance sensor 220 to detect the position of the first user U1 in the first space R1. The acquisition unit 241 acquires position information of the first user U1 based on the detection result of the position of the first user U1 in the first space R1. In this embodiment, the position information of the first user U1 is the position of the first user U1 in the first space R1 (the position of the first user U1 in the coordinate system of the first space R1). For example, the acquisition unit 241 acquires the position of the first user U1 relative to the distance sensor 220 as the detection result of the distance sensor 220. The acquisition unit 241 calculates the position of the first user U1 in the first space R1 as the position information of the first user U1 based on the position of the first user U1 relative to the distance sensor 220 and the position of the distance sensor 220 (the image capture device 200) in the first space R1. The position of the distance sensor 220 (image capturing device 200) in the first space R1 may be set in advance or may be detected by the distance sensor 220 (image capturing device 200). Furthermore, the user's position information is not limited to being acquired by a combination of the camera 210 and the distance sensor 220, but may also be acquired by a ToF (Time of Flight) sensor or LiDAR (Light Detection and Ranging).

[0036] In addition, the acquisition unit 241 is not limited to acquiring the position of the first user U1 within the first space R1 as the position information of the first user U1, but may also acquire the position of the first user U1 relative to the distance sensor 220 as the position information of the first user U1.

[0037] (Extraction of user contour image) The conversion unit 242 of the image capturing device 200 extracts a user contour image, which is an image including the contour of the first user U1, from the captured image. To explain in more detail, the captured image is an image of objects within the camera 210's angle of view, and includes not only the first user U1 but also objects placed in the first space R1 and the background of the first user U1. The conversion unit 242 according to this embodiment extracts an area in which the first user U1 appears from the captured image and acquires it as a user contour image. That is, the periphery of the user contour image is the contour of the first user U1, or in other words, the boundary line between the first user U1 and objects other than the first user U1.

[0038] Any method for acquiring the user contour image may be used, and the conversion unit 242 according to this embodiment extracts the user contour image based on the captured image and the position information of the first user U1. For example, the conversion unit 242 performs image analysis on the captured image to extract objects appearing in the captured image. Then, the conversion unit 242 identifies an area in the captured image in which the first user U1 appears based on the position information of the first user U1, and extracts an image of the object in the identified area as the user contour image. However, the method for acquiring the user contour image is not limited to using the position information of the first user U1. For example, the conversion unit 242 may extract an object corresponding to the first user U1 from multiple objects obtained by image analysis using any method, and extract the image of the object as the user contour image.

[0039] The conversion unit 242 may also include in the user contour image objects other than the first user U1 that are in contact with the first user U1 (including objects held by the first user U1). In other words, in this case, the user contour image is an image that includes the first user U1 and objects that are in contact with the first user U1. In this case, the periphery of the user contour image is expanded to include objects other than the first user U1 as part of the first user U1, forming a boundary line between the first user U1 (including objects that are in contact with the first user U1) and objects other than the first user U1 (including objects that are in contact with the first user U1). In this case, the conversion unit 242 extracts, as the user contour image, the area in which the first user U1 is captured and the area in which objects that are in contact with the first user U1 are captured. In this case, the method for extracting the user contour image may be any method, but for example, the conversion unit 242 may extract the first user U1 and an object in contact with the first user U1 as a single object through image analysis of the captured image, and use the image of the extracted object as the user contour image.

[0040] (User image generation) The conversion unit 242 generates a user image YI based on the user contour image. The user image YI is a two-dimensional image including the contour of the first user U1. Any method may be used to generate the user image YI, but in this embodiment, the conversion unit 242 sets the gradation value of each pixel included in the acquired user contour image (each pixel within the area surrounded by the periphery of the user contour image) to a predetermined value to generate the user image YI. While the predetermined value may be any value, the conversion unit 242 sets the gradation value of each pixel to the same, thereby generating the user image YI as a single-color image. Furthermore, the conversion unit 242 generates the user image YI as a black image. As a result, the user image YI becomes a two-dimensional image showing the shadow of the first user U1.

[0041] (User image output) The output control unit 243 of the photographing device 200 outputs the user image YI (image data of the user image YI) generated by the conversion unit 242 to the image generating device 100. Furthermore, the output control unit 243 outputs the user image YI and position information of the first user U1 when the captured image used to generate the user image YI was captured in association with each other to the image generating device 100.

[0042] The above-described process of generating the user image YI may be performed by a device other than the photographing device 200, or may be performed by the image generating device 100. In this case, for example, the image generating device 100 may acquire a captured image from the photographing device 200, extract a user contour image from the captured image, and generate the user image YI. Alternatively, the image generating device 100 may acquire a user contour image from the photographing device 200, and generate the user image YI from the user contour image.

[0043] (Image Generation Device Processing) Next, the processing contents of the image generating device 100 will be described.

[0044] (Get user image) The acquisition unit 141 of the image generating device 100 acquires the user image YI (image data of the user image YI) and the position information of the first user U1 from the photographing device 200.

[0045] (Virtual space) FIG. 4 is a schematic diagram showing Example 1 of a virtual space according to this embodiment. As shown in FIG. 4, the virtual space VR is a virtual three-dimensional space set by the image generating device 100. The virtual space VR is used to generate a composite image CI. The image generating device 100 may also set the virtual space VR to have a shape that resembles the second space R2. By setting the virtual space VR to have a shape that resembles the second space R2, it is possible to provide an effect in which a second user U2 present in the second space R2 feels as if a first user U1, who is not actually present in the second space R2, is present in the second space R2. Note that the virtual space VR having a shape that resembles the second space R2 may refer to, for example, setting a correspondence between the coordinate system of the second space R2 and the coordinate system of the virtual space VR in advance. Hereinafter, one direction in the virtual space VR is referred to as direction Dx, a direction perpendicular to direction Dx is referred to as direction Dy, and a direction perpendicular to both direction Dx and direction Dy is referred to as direction Dz. In the following description, the direction Dy is assumed to be the depth direction in the virtual space VR (the direction in which each image, which will be described later, is superimposed).

[0046] A method for generating the composite image CI will be specifically described below.

[0047] (Setting the position of the user image) The position setting unit 142 sets depth information of the user image YI based on the position information of the first user U1. The depth information of the user image YI refers to the position of the user image YI in the depth direction (direction Dy) in the virtual space VR, in other words, the position of the user image YI in the direction Dy (Dy coordinate value) in the coordinate system of the virtual space VR. In this embodiment, the position setting unit 142 also sets the position of the user image YI in the direction Dx in the virtual space VR (Dx coordinate value in the coordinate system of the virtual space VR) and the position of the user image YI in the direction Dz in the virtual space VR (Dz coordinate value in the coordinate system of the virtual space VR) based on the position information of the first user U1.

[0048] The position setting unit 142 may set the Dy coordinate value (depth information), Dx coordinate value, and Dz coordinate value of the user image YI using any method based on the position information of the first user U1. For example, in this embodiment, a correspondence relationship between the coordinate system of the first space R1 and the coordinate system of the virtual space VR is set in advance, and the position setting unit 142 calculates the Dy coordinate value (depth information), Dx coordinate value, and Dz coordinate value of the user image YI based on the position information of the first user U1 in the first space R1 and the correspondence relationship between the first space R1 and the virtual space VR. Specifically, the position setting unit 142 calculates the position of the first user U1 in the virtual space VR by coordinate conversion of the position information of the first user U1 in the first space R1 using the correspondence relationship. The position setting unit 142 sets the calculated Dy coordinate value, Dx coordinate value, and Dz coordinate value of the position of the first user U1 in the virtual space VR as the Dy coordinate value, Dx coordinate value, and Dz coordinate value of the user image YI, respectively.

[0049] FIG. 4 shows an example in which the Dy coordinate value of the user image YI is y2.

[0050] (Object image) In this embodiment, the image generating device 100 generates a composite image CI so as to include an object image OI, which is an image other than the user image YI. The object image OI is a two-dimensional image. The position setting unit 142 acquires the object image OI (image data of the object image OI). The position setting unit 142 may acquire the object image OI by any method. For example, multiple types of object images OI may be set in advance (for example, stored in the storage unit 130), and the position setting unit 142 may acquire an object image OI selected from the multiple types of object images OI. The object image OI may be selected by the user U or may be automatically selected. Furthermore, the position setting unit 142 is not limited to acquiring a preset object image OI, and may automatically generate the object image OI or acquire an object image OI input by the user U.

[0051] Note that, like the user image YI, the object image OI may also be a single-color image in which the gradation value of each pixel is set to the same. The object image OI may be a black image. This makes the object image OI a two-dimensional image showing the shadow of the object.

[0052] The position setting unit 142 sets depth information of the object image OI. The depth information of the object image OI refers to the position (Dy coordinate value) of the object image OI in the direction Dy in the coordinate system of the virtual space VR. In this embodiment, the position setting unit 142 also sets the position of the object image OI in the direction Dx in the virtual space VR (Dx coordinate value in the coordinate system of the virtual space VR) and the position of the object image OI in the direction Dz in the virtual space VR (Dz coordinate value in the coordinate system of the virtual space VR). The position setting unit 142 may set the Dy coordinate value, Dx coordinate value, and Dz coordinate value of the object image OI using any method. For example, the Dy coordinate value, Dx coordinate value, and Dz coordinate value of the object image OI may be set in advance, and the position setting unit 142 may acquire the set Dy coordinate value, Dx coordinate value, and Dz coordinate value of the object image OI. For example, the position setting unit 142 may automatically generate the Dy coordinate values, Dx coordinate values, and Dz coordinate values ​​of the object image OI, or may acquire the Dy coordinate values, Dx coordinate values, and Dz coordinate values ​​input by the user U.

[0053] In the example of Fig. 4, a first object image BI1 which is an image of a bush and a second object image BI2 which is an image of a tree are set as object images OI. In the example of Fig. 4, the Dy coordinate value of the first object image BI1 is y1, and the Dy coordinate value of the second object image BI2 is y3.

[0054] (composite image) The image synthesis unit 143 generates a synthesized image CI, which is a two-dimensional image obtained by projecting the user image YI in the virtual space VR, with the user image YI being at a position indicated by the depth information in the virtual space VR. This will be explained in detail below.

[0055] The image composition unit 143 places the user image YI at a position indicated by the set depth information in the virtual space VR. More specifically, the image composition unit 143 places the user image YI at a position in the virtual space VR that corresponds to the set Dy, Dx, and Dz coordinate values. The image composition unit 143 also places the object image OI at a position in the virtual space VR that corresponds to the set Dy, Dx, and Dz coordinate values. In the example of FIG. 4, the user image YI is placed at a position y2 in the Dy direction, the first object image BI1 is placed at a position y1 in the Dy direction, and the second object image BI2 is placed at a position y3 in the Dy direction. In FIG. 4, y1, y2, and y3 are values ​​that are arranged in order in the Dy direction, so that in the virtual space VR, the first object image BI1, the user image YI, and the second object image BI2 are arranged in this order in the Dy direction.

[0056] The image composition unit 143 generates a two-dimensional image as a composite image CI by projecting the user image YI and the object image OI arranged at set positions in the virtual space VR in the Dy direction. Specifically, the image composition unit 143 projects the user image YI and the object image OI in the Dy direction onto a projection surface PL in the virtual space VR, and generates the image projected onto the projection surface PL as a composite image CI. The projection surface PL may be set at any position within the virtual space VR, but in this embodiment, it is preferably set at a position further back (in the Dy direction) than all images (user image YI and object image OI).

[0057] Furthermore, it is preferable that the image synthesis unit 143 projects the user image YI and the object image OI onto the projection surface PL from a predetermined viewpoint position P1 in the virtual space VR, and generates the image projected onto the projection surface PL as a synthesized image CI. The viewpoint position P1 here may be set to any position, and in this embodiment, it is preferable that it is located closer to the viewer (opposite the Dy direction) than all images (the user image YI and the object image OI). In this embodiment, the viewpoint position P1 is fixed.

[0058] In this way, by projecting the user image YI and the object image OI onto the projection surface PL, the composite image CI becomes a two-dimensional image in which the user image YI and the object image OI are displayed on the same plane.

[0059] Fig. 5 is a schematic diagram illustrating Example 1 of a composite image according to this embodiment. Fig. 5 shows an example of a composite image CI when the images located at the positions shown in Fig. 4 are projected onto a projection surface PL. As shown in Fig. 5, in this example, the composite image CI is a two-dimensional image in which the user image YI, the first object image BI1, and the second object image BI2 are projected onto the same surface.

[0060] The image composition unit 143 according to this embodiment updates and changes the position of the user image YI in the virtual space VR as needed, depending on the position of the first user U1 in the first space R1. The image composition unit 143 also changes the shape of the user image YI depending on the position and movement of the first user U1 in the first space R1. As a result, the position of the user image YI in the composite image CI changes depending on the position of the first user U1 in the first space R1, and the shape of the user image YI changes depending on the position and movement of the first user U1 in the first space R1. Note that the image composition unit 143 may fix the Dy-direction position of the user image YI in the virtual space VR, among the positions of the user image YI in the Dx, Dy, and Dz directions, and move only the Dx and Dz directions of the user image YI in the virtual space VR depending on the position of the first user U1 in the first space R1. This allows the user image YI to be changed while keeping the depth direction position of the user image YI fixed in the virtual space VR.

[0061] In Figure 4, the first object image BI1 showing an image of a patch of grass is positioned so that it is closer to reference point O1 (in the -Dy direction) than the user image YI. Therefore, even if the first user U1 moves within the first space R1 and the user image YI showing an image of a shadow moves along the direction Dx in the virtual space VR, and the image of the patch of grass and the image of the shadow of the first user U1 overlap, the composite image CI is such that the image of the patch of grass is always closer to reference point O1 (in the -Dy direction) than the image of the shadow of the first user U1. Also, in the relationship between the user image YI showing the image of the shadow of the first user U1 and the second object image BI2 showing the background of trees, in Figure 4, the user image YI is superimposed so that it is closer to the reference point O1 (-Dy direction) than the second object image BI2.Therefore, even if the first user U1 moves and the image of the tree and the image of the shadow of the first user U1 overlap, the composite image CI is always such that the image of the shadow of the first user U1 is closer to the reference point O1 (-Dy direction) than the background image of the trees.

[0062] Furthermore, the projection surface PL onto which the user image YI and object image OI arranged at set positions in the virtual space VR are projected may be multiple surfaces. For example, the projection surface PL in the virtual space VR may be set to two surfaces corresponding to the wall and floor surfaces of the second space R2. In this case, for example, the image synthesis unit 143 may project the user image YI and the object image OI onto each projection surface PL from a predetermined viewpoint position in the virtual space VR and generate each of the images projected onto the projection surfaces PL as a composite image CI. FIG. 6 is a schematic diagram illustrating Example 2 of the virtual space according to this embodiment. In the example of FIG. 6, the image synthesis unit 143 projects the user image YI and the object image OI onto a first projection surface PL1 and a second projection surface PL2 from a predetermined viewpoint position P2 in the virtual space VR and generates the image projected onto the first projection surface PL1 as a first composite image CI1 and the image projected onto the second projection surface PL2 as a second composite image CI2. The viewpoint position P2 here may be set at any position, and in this example, it is located closer to the user image YI (opposite the Dy direction) and further back (in the Dy direction) than the first object image BI1. In this example, the viewpoint position P2 is fixed. The first projection plane PL1 may be set at any position within the virtual space VR, but in this example, it is set at a position further back (in the Dy direction) than all images (user image YI and object image OI). The second projection plane PL2 may also be set at any position within the virtual space VR, but in this example, it is set on the Dx-Dy plane orthogonal to the first projection plane PL1.

[0063] FIG. 7 is a schematic diagram illustrating Example 2 of a composite image according to this embodiment. FIG. 7 shows a first composite image CI1 and a second composite image CI2 obtained when the images positioned as shown in FIG. 6 are projected onto the first projection plane PL1 and the second projection plane PL2. As shown in FIG. 7, in this example, the first composite image CI1 and the second composite image CI2 are two-dimensional images in which the user image YI and the second object image BI2 are projected onto the same plane. Here, the second composite image CI2 is an image of the shadow of the first user U1 or the tree, starting from the position of the first user U1's feet or the root of the tree in the first composite image CI1 and extending to their respective end positions on the floor. The end positions of the second composite image CI2 (the end positions of the second composite image CI2 on the floor) coincide with the depth positions of the user image YI and the second object image BI2 in the virtual space VR. In this example, the Dy coordinate value of the end point of the image of the shadow of the first user U1 in the second composite image CI2 is y2, which matches the depth position of the user image YI in the virtual space VR. Also, in this example, the Dy coordinate value of the end point of the image of the shadow of the tree in the second composite image CI2 is y3, which matches the depth position of the second object image BI2 in the virtual space VR. As in this example, by projecting the shadows extending from the first user image YI and the second object image BI2, which are at different depth positions, as the second composite image, a parallax effect is imparted to the second user U2 present in the second space R2, making it appear as if the first user U1, who is not actually present in the second space R2, is present in the second space R2, and this effect can be more strongly imparted compared to a case where there is no parallax effect.

[0064] (Output of composite image) The output control unit 144 outputs the composite image CI generated by the image synthesis unit 143. Specifically, the output control unit 144 outputs the composite image CI (image data of the composite image CI) to the output device 10 arranged in the second space R2 via the communication unit 150. The output device 10 displays the composite image CI acquired from the image generation device 100. As described above, if the output device 10 is a projector, it projects light for displaying the composite image CI onto a projection member in the second space R2. If the output device 10 is a display, it displays the composite image CI on a screen.

[0065] (Display mode of composite image) Furthermore, the image composition unit 143 according to this embodiment may change the display mode of the generated composite image CI. The image composition unit 143 changes the display mode of the user image YI included in the composite image CI to change the display mode of the composite image CI. Specifically, the image composition unit 143 according to this embodiment changes the display mode of the user image YI based on at least one of the behavioral state and biometric information of the user U. Note that the image composition unit 143 may change the display mode of the object image OI included in the composite image CI.

[0066] (Display style based on user behavior) An example will be described in which the image composition unit 143 changes the display mode of the user image YI based on the behavior of the user U. Here, the behavior of the user U may refer to, for example, the number of utterances made by the first user U1 or the distance between the first user U1 and the image capture device 200. Specifically, the image composition unit 143 acquires audio information in the first space R1 from a microphone (not shown) or the like of the image capture device 200, calculates the number of utterances made by the first user U1 in a predetermined time period by performing frequency analysis on the acquired audio information, and changes the display mode of the user image YI showing the shadow of the first user U1 based on the calculated number of utterances. The display mode here may be at least one of the size and color of the user image YI. For example, the image composition unit 143 may change the color of the user image YI to a lighter color as the number of utterances made by the first user U1 in a predetermined time period decreases. Furthermore, the image composition unit 143 may change the color of the user image YI to a darker color as the number of utterances made by the first user U1 in a predetermined time period increases.

[0067] Furthermore, the image composition unit 143 may change the display mode of the user image YI based on the distance between the image capturing device 200 and the first user U1. In this case, the image composition unit 143 may not fix the position of the user image YI in the Dy direction in the virtual space VR, but may move the user image YI in the Dx direction, Dy direction, and Dz direction in the virtual space VR according to the position of the first user U1 in the first space R1. Specifically, the image composition unit 143 may change the size of the user image YI so that the closer the distance between the image capturing device 200 and the first user U1 is, the larger the size of the user image YI becomes.

[0068] (Display mode based on user's biometric information) An example will be described in which the image composition unit 143 changes the display mode of the user image YI based on the biometric information of the user U. Here, the biometric information of the user U is preferably not unchanging information such as a fingerprint, but information whose value changes depending on the user U's psychological state. Furthermore, the biometric information is preferably information related to the user U's autonomic nervous system, i.e., information whose value changes regardless of the user U's will. Examples of the biometric information of the user U include body temperature, pulse wave, and brain wave. Specifically, the image composition unit 143 acquires the biometric information of the first user U1 from a biometric sensor (such as a thermometer, pulse wave sensor, or brain wave sensor) (not shown) attached to the first user U1, compares the acquired biometric information with a threshold, calculates the difference, and changes the display mode of the user image YI based on the difference. For example, the image composition unit 143 according to this embodiment may calculate the difference between the biometric information of the first user U1 and a threshold to predict the mental state of the first user U1 and change the color of the user image YI to a color corresponding to the predicted mental state.

[0069] Furthermore, the change in the display mode of the user image YI is not limited to a change in color or size. For example, the image composition unit 143 may change the user image YI to a two-dimensional image (such as an avatar) that is unrelated to the contour of the first user UI, based on the behavioral state or biological information of the user U.

[0070] (Processing flow) Next, the processing flow of this embodiment will be described. FIG. 8 is a flowchart illustrating the processing flow of the image generating device according to the first embodiment. As shown in FIG. 8, the acquisition unit 141 of the image generating device 100 acquires a user image YI, which is a two-dimensional image including the outline of the first user U1, and position information of the first user U1 in the first space R1 (step S1). The position setting unit 142 of the image generating device 100 sets the position of the user image YI in the virtual space VR based on the position information of the first user U1 (step S3). The position setting unit 142 of the image generating device 100 acquires an object image OI and the position of the object image OI in the virtual space VR (step S5). The image synthesis unit 143 of the image generating device 100 generates a synthesized image CI by projecting the user image YI and the object image OI (step S7). The output control unit 144 of the image generating device 100 outputs the synthesized image CI to the output device 10 arranged in the second space R2 (step S9), and this processing ends.

[0071] (effect) As described above, the image generating device 100 according to this embodiment sets the depth position (Dy coordinate value) in the virtual space VR of the user image YI, which includes the outline of the first user U1, based on the position of the first user U1 in the first space R1. Then, the image generating device 100 generates a two-dimensional image, as a composite image CI, by projecting the user image YI in the virtual space VR, with the user image YI at the set position in the virtual space VR. According to the present disclosure, by setting the depth position of the user image YI in the virtual space VR, the user image YI in the composite image CI can be made to have a warm (human) feel, in accordance with the first user U1 in the first space R1, rather than an impersonal image. This allows for better communication with people in remote locations.

[0072] Furthermore, according to this embodiment, the positions of the user image YI and the object image OI in the virtual space VR are set, and an image obtained by projecting the user image YI and the object image OI arranged at those positions is taken as a composite image CI. This allows the positional relationship between the user U and the object to be reflected in the two-dimensional composite image CI, resulting in a warmer (more human-like) image and enabling better communication.

[0073] (Second embodiment) Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that a composite image CI is generated based on the position and orientation of the terminal device 400 in the second space R2. In the second embodiment, a description of parts that are common to the first embodiment will be omitted.

[0074] Fig. 9 is a schematic diagram of an image generation system according to this embodiment. As shown in Fig. 9, the image generation system 1A according to this embodiment generates a composite image CI based on the position and orientation of a terminal device 400 in the second space R2 and a user image YI of a first user U1 present in the first space R1. In other words, the image generation system 1A according to this embodiment is a system in which the composite image CI output to the second space R2 changes depending on the position and orientation of a terminal device 400 held by a second user U2 present in the second space R2. The configuration of the image generation system 1A will be described below.

[0075] (Image generation system configuration) As shown in FIG. 9, the image generation system 1A includes an image generation device 100, a photographing device 200, an output device 10, and a terminal device 400.

[0076] (Terminal Device) The terminal device 400 is a device placed in the second space R2. The terminal device 400 is placed in the second space R2 such that its position and orientation in the second space R2 are changeable. The terminal device 400 is a device that detects the position and orientation of the terminal device 400 itself in the second space R2. The terminal device 400 may be any device, but is preferably a device that can be carried by the second user U2.

[0077] In this embodiment, the terminal device 400 is stored in a housing 401. The shape of the housing 401 may be any shape, but in this embodiment, it is flashlight-shaped. The shape of the housing 401 may be set so that the terminal device 400 faces in a predetermined vertical and horizontal direction that is set in advance within the second space R2. When the second user U2 holds the housing 401 in which the terminal device 400 is stored and changes the orientation of the housing 401, the orientation of the terminal device 400 in the second space R2 is calculated based on the predetermined vertical and horizontal orientations.

[0078] It should be noted that terminal device 400 is detachable from housing 401, and terminal device 400 is separate from housing 401. However, the present invention is not limited to this, and terminal device 400 may be non-detachable from housing 401, or terminal device 400 itself may be shaped like a flashlight.

[0079] FIG. 10 is a schematic block diagram of a terminal device according to this embodiment. As shown in FIG. 10, the terminal device 400 according to this embodiment includes a sensor unit 410, an input unit 420, a communication unit 430, a storage unit 440, and a control unit 450. The sensor unit 410 is a sensor that acquires terminal information, which is information indicating the position and orientation of the terminal device 400 in the second space R2, and is, for example, a GPS sensor or a gyro sensor. The input unit 420 is a device through which the user U inputs information to the terminal device, and may be, for example, a touch panel. The communication unit 430 is a communication module that enables the terminal device 400 to communicate with external devices. The communication unit 430 is responsible for transmitting and receiving various data between the terminal device 400 and external devices via wireless communication or wired communication. In the case of wireless communication, the communication unit 430 may include communication processing circuits such as a communication antenna, an RF circuit, and a wireless LAN card. In the case of wired communication, the communication unit 430 may include, for example, a NIC with a wired LAN terminal, a transmission circuit, and other communication processing circuits. The communication format according to this embodiment is wireless communication.

[0080] The storage unit 440 is a memory that stores the calculation contents and programs of the control unit 450, and includes at least one of a RAM, a main storage device such as a ROM, and a non-volatile storage device such as a flash memory or an SSD, for example.

[0081] The control unit 450 is a calculation device and includes a calculation circuit such as a CPU. The control unit 450 includes an acquisition unit 451 and an output control unit 452. The control unit 450 realizes the acquisition unit 451 and the output control unit 452 by reading and executing a program (software) from the storage unit 440. The control unit 450 executes these processes by using one CPU, or may be provided with multiple CPUs and execute the processes by using the multiple CPUs. At least a part of the acquisition unit 451 and the output control unit 452 may be implemented by hardware.

[0082] (Terminal device processing) The processing performed by the terminal device 400 will be described below.

[0083] (Getting device information) The acquisition unit 451 of the terminal device 400 controls the sensor unit 410 to detect the position and orientation of the terminal device 400 in the second space R2. The acquisition unit 451 acquires the detection result of the position and orientation of the terminal device 400 in the second space R2 as terminal information indicating the position and orientation of the terminal device 400 in the second space R2.

[0084] (Output of terminal information) The output control unit 452 of the terminal device 400 outputs the terminal information acquired by the acquisition unit 451 to the image generating device 100 via the communication unit 430.

[0085] (Image Generation Device Processing) Next, the processing contents of the image generating device 100 will be described.

[0086] (user images and object images) The image generating device 100 acquires a user image YI and sets the position (Dy coordinate value, Dx coordinate value, and Dz coordinate value) of the user image YI in the virtual space VR in the same manner as in the first embodiment. Also, the image generating device 100 acquires an object image OI and sets the position (Dy coordinate value, Dx coordinate value, and Dz coordinate value) of the object image OI in the virtual space VR in the same manner as in the first embodiment.

[0087] (viewpoint position) Furthermore, the position setting unit 142 of the image generating device 100 sets a viewpoint position P' in the virtual space VR based on the acquired terminal information. Based on the position of the terminal device 400 in the second space R2 indicated in the terminal information, the position setting unit 142 calculates the position of the viewpoint position P' in the direction Dy in the virtual space VR (Dy coordinate value in the coordinate system of the virtual space VR), the position of the direction Dx in the virtual space VR (Dx coordinate value in the coordinate system of the virtual space VR), and the position of the direction Dz in the virtual space VR (Dz coordinate value in the coordinate system of the virtual space VR). Based on the orientation of the terminal device 400 in the second space R2 indicated in the terminal information, the position setting unit 142 calculates the direction (vector) of the viewpoint position P' in the virtual space VR.

[0088] The position setting unit 142 may set the Dy coordinate value (depth information), Dx coordinate value, Dz coordinate value, and orientation of the viewpoint position P' using any method based on the terminal information. For example, in this embodiment, a correspondence relationship between the coordinate system of the second space R2 and the coordinate system of the virtual space VR is set in advance, and the position setting unit 142 calculates the Dy coordinate value (depth information), Dx coordinate value, Dz coordinate value, and orientation of the viewpoint position P' based on the terminal information and the correspondence relationship between the second space R2 and the virtual space VR. Here, the correspondence relationship between the second space R2 and the virtual space VR may be set arbitrarily. For example, the correspondence relationship may be set so that the shape of the virtual space VR is a shape that resembles the second space R2 onto which the composite image CI is projected. The position setting unit 142 calculates the position and orientation of the terminal device 400 in the virtual space VR by coordinate conversion of the position and orientation of the terminal device 400 in the second space R2 using the correspondence relationship. The position setting unit 142 sets the calculated Dy coordinate value, Dx coordinate value, Dz coordinate value and orientation of the terminal device 400 in the virtual space VR as the Dy coordinate value, Dx coordinate value, Dz coordinate value and orientation of the viewpoint position P', respectively.

[0089] (composite image) The image synthesis unit 143 generates a synthesized image CI, which is a two-dimensional image obtained by projecting the user image YI from the viewpoint position P', in a state in which the user image YI is located in the virtual space VR.

[0090] More specifically, the image composition unit 143 according to this embodiment places the user image YI and the object image OI at set positions in the virtual space VR using the same method as in the first embodiment.

[0091] Then, the image composition unit 143 generates a two-dimensional image as a composite image CI by projecting the user image YI and the object image OI arranged at the set positions in the virtual space VR from the set viewpoint position P' in the direction of the set viewpoint position P'. FIG. 11 is a schematic diagram illustrating Example 1 of the virtual space according to this embodiment. As shown in FIG. 11, it is preferable that the image composition unit 143 projects the user image YI and the object image OI in the virtual space VR from the set viewpoint position P' in the direction of the set viewpoint position P' onto the projection surface PL, and generates the image projected on the projection surface PL as the composite image CI. That is, while the viewpoint position P1 or P2 was a fixed point in the first embodiment, the viewpoint position P' according to this embodiment changes depending on the position and orientation (terminal information) of the terminal device 400 in the second space R2. Therefore, the composite image CI changes depending on the viewpoint position P', in other words, depending on the position and orientation of the terminal device 400 in the second space R2.

[0092] More specifically, in this embodiment, the position setting unit 142 sets the field of view range SA within the virtual space VR based on the terminal information, and the image synthesis unit 143 generates a two-dimensional image within the field of view range SA as a synthesized image CI by projecting the user image YI (and the object image OI) from the viewpoint position P'.

[0093] The field of view range SA refers to the area on the projection surface PL where the user image YI and the object image OI are displayed. The position setting unit 142 may set the field of view range SA using any method based on the terminal information. For example, the position setting unit 142 may set the field of view range SA as an area of ​​a predetermined size centered on the intersection of a line extending from the viewpoint position P' in a direction along the direction of the viewpoint position P' (the vector of the viewpoint position P') and the projection surface PL. The size of the field of view range SA here (predetermined size) may be constant or may vary depending on the terminal information. For example, the field of view range SA may be increased as the distance between the terminal device 400 and the area in the second space R2 where the composite image CI is displayed (e.g., the projection component or the screen of the output device 10) decreases. The shape of the field of view range SA is arbitrary, and may be, for example, circular.

[0094] The image synthesis unit 143 generates a synthesized image CI from the user image YI and the object image OI within the field of view SA among the two-dimensional images on the projection surface PL in which the user image YI and the object image OI are projected in a set direction from the viewpoint position P'. In other words, the user image YI and the object image OI are not displayed outside the field of view SA. FIG. 12 is a schematic diagram illustrating the field of view according to this embodiment. As shown in FIG. 12, the image synthesis unit 354 sets the gradation value of each pixel in the area occupied by the user image YI and the object image OI within the field of view SA to a first predetermined value (e.g., a value corresponding to black). Furthermore, the image synthesis unit 143 sets the gradation value of each pixel in the area not occupied by the user image YI and the object image OI within the field of view SA to a second predetermined value (e.g., a value corresponding to white) different from the first predetermined value. Furthermore, the image synthesis unit 143 sets the gradation value of each pixel in the area outside the field of view SA to a first predetermined value (e.g., a value corresponding to black). This makes it possible to display the user image YI and the object image OI only within the field of view SA. However, by setting the gradation value of each pixel in the area outside the field of view SA to a third predetermined value different from the first predetermined value and the second predetermined value, the area outside the field of view SA may be given a color different from that of the area within the field of view SA.

[0095] However, the image synthesis unit 143 may generate an image that does not include an image of an area outside the field of view SA, but includes only an area inside the field of view SA, as the synthetic image CI. In this case, the output device 10 displays the synthetic image CI that includes only the area inside the field of view SA.

[0096] (Output of composite image) The output control unit 355 outputs the composite image CI generated by the image synthesis unit 354 to the output device 10. The output device 10 displays the composite image CI acquired from the image generation device 100 in the second space R2.

[0097] 13 is a schematic diagram showing Example 1 of a composite image displayed in the second space. As described above, the position and orientation of the viewpoint position P' change depending on the position and orientation of the terminal device 400 in the second space R2, and therefore the composite image CI displayed by the output device 10 also changes depending on the position and orientation of the terminal device 400 in the second space R2. Therefore, according to this embodiment, the composite image CI changes depending on, for example, the operation of the terminal device 400 by the second user U2, and therefore the composite image CI can be displayed interactively, enabling better communication with people in remote locations.

[0098] Furthermore, the composite image CI displays the user image YI and the object image OI only in the field of view SA. That is, the user image YI and the object image OI are displayed in a spot-like manner within the field of view SA. For example, if pseudo light is irradiated into the second space R2 from the housing 401 simulating a flashlight, the second user U2 will visually recognize that the user image YI and the object image OI are displayed in an area that is expected to be bright (illuminated by the light). Therefore, the positions at which the user image YI and the object image OI are displayed change in response to operations on the terminal device 400, allowing the composite image CI to be displayed more interactively.

[0099] FIG. 14 is a schematic diagram showing Example 2 of a composite image displayed in the second space. As the position and orientation of the viewpoint position (the position and orientation of the terminal device 400 in the second space R2) change, the position of the field of view SA also changes. FIG. 14 shows a composite image CI including the field of view SA at the position and orientation of the viewpoint position P" in FIG. 12. As shown in FIG. 14, as the position of the field of view SA changes, the displayed positions of the user image YI and object image OI also change.

[0100] (Processing flow) Next, the processing flow of this embodiment will be described. FIG. 15 is a flowchart illustrating the processing flow of the image generating device according to the second embodiment. As shown in FIG. 15, the acquisition unit 141 of the image generating device 100 acquires a user image YI, which is a two-dimensional image including the contour of the first user U1, and position information of the first user U1 in the first space R1 (step S21). The position setting unit 142 of the image generating device 100 sets the position of the user image YI in the virtual space VR based on the position information of the first user U1 (step S23). The position setting unit 142 of the image generating device 100 acquires an object image OI and the position of the object image OI in the virtual space VR (step S25). The position setting unit 142 sets the position and orientation of the viewpoint position P′ and the field of view range SA in the virtual space VR based on the terminal information of the terminal device 400 (step S27). The image synthesis unit 143 generates a two-dimensional image within the field of view SA by projecting the user image YI and the object image OI from the viewpoint position P' as a synthetic image CI (step S29). The output control unit 144 outputs the synthetic image CI to the output device 10 arranged in the second space R2 via the communication unit 150 (step S31), and this process ends.

[0101] (Modification of the second embodiment) Next, a modified example of the second embodiment will be described. The modified example of the second embodiment differs from the second embodiment in that the user image YI and the object image OI are three-dimensional images with depth. In this modified example, the description of the parts that are common to the second embodiment will be omitted.

[0102] (Image Generation Device Processing) The processing contents of the image generating device 100 according to this modified example will be explained.

[0103] (3D user image and 3D object image) The image generating device 100 acquires a 3D user image YIA, which is a three-dimensional image. The 3D user image YIA may be acquired by any method. For example, a plurality of user images YI, which are two-dimensional images, may be acquired from a plurality of image capturing devices 200, and the image generating device 100 may generate the 3D user image YIA from the acquired plurality of user images YI. The image generating device 100 then sets the position (Dy coordinate value, Dx coordinate value, and Dz coordinate value) of the 3D user image YIA in the virtual space VR using a method similar to that of the second embodiment. The image generating device 100 also acquires a 3D object image OIA, which is a three-dimensional image. The 3D object image OIA may be acquired by any method. For example, a plurality of types of 3D object images OIA may be set in advance (for example, stored in the storage unit 130), and the image generating device 100 may acquire a 3D object image OIA selected from the plurality of types of 3D object images OIA. The 3D object image OIA may be selected by the user U or automatically selected. Furthermore, the image generating device 100 is not limited to acquiring a preset 3D object image OIA, but may automatically generate a 3D object image OIA, or may acquire a 3D object image OIA input by the user U. Then, the image generating device 100 sets the position (Dy coordinate value, Dx coordinate value, and Dz coordinate value) of the 3D object image OIA in the virtual space VR in the same manner as in the second embodiment.

[0104] (composite image) The image synthesis unit 143 generates a synthesized image CI, which is a two-dimensional image obtained by projecting the 3D user image YI from the viewpoint position P', in a state in which the 3D user image YIA is located within the virtual space VR.

[0105] More specifically, the image synthesis unit 143 according to this embodiment places the 3D user image YIA and the 3D object image OIA at set positions in the virtual space VR using the same method as in the second embodiment.

[0106] The image synthesis unit 143 then generates a two-dimensional image as a synthetic image CI by projecting the 3D user image YIA and the 3D object image OIA, which are 3D images arranged at set positions in the virtual space VR, from the set viewpoint position P' in the direction of the set viewpoint position P'. That is, the synthetic image CI generated in this modification is a two-dimensional image in which the shapes of the projected 3D user image YIA and the 3D object image OIA change depending on the viewpoint position. In other words, the synthetic image CI according to this modification is an image of the shadows of the 3D user image YIA and the 3D object image OIA from any direction determined by the viewpoint position.

[0107] Also in this modified example, the position setting unit 142 sets the field of view range SA in the virtual space VR in the same manner as in the second embodiment, and the image synthesis unit 143 generates a two-dimensional image within the field of view range SA as a synthesized image CI by projecting the 3D user image YIA (and the 3D object image OIA) from the viewpoint position P'.

[0108] In the second embodiment, only the positions at which the user image YI and the object image OI are displayed are changed by changing the field of view SA in response to the operation of the terminal device 400. However, in this modified example, not only the positions at which the 3D user image YIA and the object image OIA are displayed but also the shapes of the 3D user image YIA and the object image OIA themselves can be changed by changing the field of view SA. This makes it possible to give a stronger effect than in the second embodiment in that it makes the second user U2 present in the second space R2 feel as if the first user U1, who is not actually present in the second space R2, is present in the second space R2.

[0109] (Effects of the present disclosure) As described above, the image generating device 100 of the present disclosure includes an acquisition unit 141 that acquires a user image YI including a two-dimensional image of the outline of a first user U1 in a first space R1 and terminal information that is information indicating the position and orientation of a terminal device 400 in a second space R2 different from the first space R1, a position setting unit 142 that sets a viewpoint position in a virtual space VR based on the terminal information, an image synthesis unit 143 that generates a composite image CI that is a two-dimensional image obtained by projecting the user image YI from the viewpoint position while the user image YI is located in the virtual space VR, and an output control unit 144 that outputs the composite image CI. According to the present disclosure, better communication with people in remote locations can be realized.

[0110] According to the present disclosure, the position setting unit 142 also sets the field of view SA in the virtual space VR based on the terminal information, and the image synthesis unit 143 generates a two-dimensional image within the field of view SA by projecting the user image YI from the viewpoint position as a synthetic image CI. According to the present disclosure, it is possible to generate a synthetic image CI in which the field of view SA changes based on the terminal information.

[0111] Furthermore, according to the present disclosure, the image composition unit 143 generates a two-dimensional image obtained by projecting the user image YI and the object image OI located at a set position in the virtual space VR from a viewpoint position as a composite image CI. According to the present disclosure, the user image YI can be composed with the object image OI in the virtual space VR to generate the composite image CI.

[0112] The image generating method according to the present disclosure includes the steps of: acquiring a user image YI including a two-dimensional image of the contour of a first space R1, and terminal information indicating the position and orientation of a terminal device 400 in a second space R2 different from the first space R1; setting a viewpoint position in a virtual space VR based on the terminal information; generating a composite image CI, which is a two-dimensional image obtained by projecting the user image YI from the viewpoint position while the user image YI is located in the virtual space VR; and outputting the composite image CI. According to the present disclosure, better communication with people in remote locations can be realized.

[0113] The program according to the present disclosure causes a computer to execute the following steps: acquiring a user image YI including a two-dimensional image of the contour of a first space R1 and terminal information indicating the position and orientation of a terminal device 400 in a second space R2 different from the first space R1; setting a viewpoint position in a virtual space VR based on the terminal information; generating a composite image CI, which is a two-dimensional image obtained by projecting the user image YI from the viewpoint position while the user image YI is located in the virtual space VR; and outputting the composite image CI. According to the present disclosure, better communication with people in remote locations can be realized. [Explanation of symbols]

[0114] CI composite image YI User Image OI Object Image R1 First space R2 2nd space U1 First User U2 Second user VR virtual space SA field of view

Claims

1. an acquisition unit that acquires a user image including a two-dimensional image of the user's outline in a first space and terminal information that is information indicating the position and orientation of a terminal device in a second space different from the first space; a position setting unit that sets a viewpoint position in a virtual space based on the terminal information; an image synthesis unit that generates a synthetic image, which is a two-dimensional image obtained by projecting the user image from the viewpoint position while the user image is located in the virtual space; an output control unit that outputs the composite image; An image generating device comprising:

2. The position setting unit also sets a field of view range in the virtual space based on the terminal information, and the image synthesis unit generates, as the synthesized image, a two-dimensional image within the field of view range by projecting the user image from the viewpoint position. The image generating device of claim 1 .

3. the image synthesis unit generates, as the synthesized image, a two-dimensional image obtained by projecting the user image and an object image located at a set position in the virtual space from the viewpoint position; 3. The image generating device according to claim 1 or 2.

4. acquiring a user image including a two-dimensional image of a contour of a first space and terminal information that is information indicating a position and orientation of a terminal device in a second space different from the first space; setting a viewpoint position in a virtual space based on the terminal information; generating a composite image, which is a two-dimensional image obtained by projecting the user image from the viewpoint position while the user image is located within the virtual space; and outputting the composite image. Image generation method.

5. acquiring a user image including a two-dimensional image of a contour of a first space and terminal information that is information indicating a position and orientation of a terminal device in a second space different from the first space; setting a viewpoint position in a virtual space based on the terminal information; generating a composite image, which is a two-dimensional image obtained by projecting the user image from the viewpoint position while the user image is located within the virtual space; outputting the composite image in the second space; A program that causes a computer to execute the following.

Images