Control System, Control Method, and Program

Abstract

This control system comprises an acquiring unit, a generating unit, and an output control unit. The acquiring unit acquires data in which imaging device identification information for identifying an imaging device installed in a real space, and frame identification information for identifying a frame in a virtual reality space are associated with each other. The generating unit generates a video of frames that can be seen by a user in the virtual reality space from a video imaged by the imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frames, on the basis of the positional relationship in the virtual reality space between the user and the frames in the virtual reality space. The output control unit displays the generated video of the frames within the frames in the virtual reality space displayed on a display device.

Description

Control system, control method, and recording medium

[0001] The present disclosure relates to control systems and the like.

[0002] In a virtual reality space, an image of a real space may be displayed.

[0003] For example, Patent Document 1 describes displaying an image on a head mounted display (HMD). Specifically, Patent Document 1 describes that the image displayed on the HMD includes a virtual reality space or an augmented reality space in which an interior space and a view from a tourist vehicle are combined, and a character, and that the view from the tourist vehicle is combined with a window of the moving part.

[0004] Furthermore, Patent Document 2 describes a technique in which, in order to have a conversation in a virtual reality space, a conversation partner is photographed by a virtual camera and the photographed image is displayed in a virtual window.

[0005] JP 2021-058528 A International Publication No. 2022 / 091832

[0006] For example, in a virtual reality space, if the scenery seen from the frame of a window or door of a building or vehicle is from a fixed viewpoint, there is a problem in that the sense of realism is low.

[0007] An example of an object of the present disclosure is to provide a control system or the like that improves the sense of realism.

[0008] A control system according to one aspect of the present disclosure includes an acquisition means for acquiring data associating imaging device identification information that identifies an imaging device installed in real space with frame identification information that identifies a frame in virtual reality space; a generation means for generating an image of the frame as seen by a user in the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame, based on a positional relationship in the virtual reality space between the user in the virtual reality space and the frame; and an output control means for displaying the generated image of the frame within the frame displayed on a display device.

[0009] A control method in one aspect of the present disclosure acquires data associating imaging device identification information that identifies an imaging device installed in real space with frame identification information that identifies a frame in virtual reality space, generates an image of the frame as seen by the user in the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame based on a positional relationship in the virtual reality space between the user in the virtual reality space and the frame, and displays the generated image of the frame within the frame displayed on a display device.

[0010] A program in one aspect of the present disclosure causes a computer to execute processing to acquire data associating imaging device identification information that identifies an imaging device installed in real space with frame identification information that identifies a frame in virtual reality space, generate an image of the frame as seen by the user in the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame based on a positional relationship in the virtual reality space between the user in the virtual reality space and the frame, and display the generated image of the frame within the frame displayed on a display device.

[0011] Each program may be stored in a non-transitory computer-readable recording medium.

[0012] According to the present disclosure, it is possible to improve the sense of realism.

[0013] 1 is a block diagram showing an example of a configuration of a control system according to a first embodiment; FIG. 2 is a flowchart showing an example of an operation of the control system according to the first embodiment; FIG. 3 is an explanatory diagram showing an example of a dome-shaped display used as a display device; FIG. 4 is an explanatory diagram showing an example of a connection between a control system and a dome-shaped display or the like; FIG. 5 is an explanatory diagram showing an example of an HMD used as a display device; FIG. 6 is an explanatory diagram showing an example of a connection between a control system and an HMD or the like; FIG. 7 is an explanatory diagram showing an example of a seat in a moving body; FIG. 8 is a block diagram showing an example of a configuration of a control system according to a second embodiment; FIG. 9 is an explanatory diagram showing an example of an imaging device installed in a moving body; FIG. 10 is an explanatory diagram showing an example of a correspondence relationship between one imaging device in real space and a window in virtual reality space; FIG. 11 is an explanatory diagram showing an example of an association table associating imaging devices with windows; FIG. 12 is an explanatory diagram showing an example of cutting out an image of a window from an imaged image; FIG. 13 is an explanatory diagram showing an example of an association table associating multiple imaging devices with windows; FIG. 14 is an explanatory diagram showing an example of a correspondence relationship between two imaging devices in real space and a window in virtual reality space; FIG. 10 is an explanatory diagram showing an example of selecting and generating an image of a window from images captured by each of a plurality of imaging devices. FIG. 11 is an explanatory diagram showing an example of a display on a dome-shaped display. FIG. 12 is an explanatory diagram showing an example of graphically displaying the voices of other users. FIG. 13 is an explanatory diagram showing an example of switching from graphic output to audio output. FIG. 14 is a flowchart showing an example of an operation of a control system according to a second embodiment. FIG. 15 is an explanatory diagram showing an example of the hardware configuration of a computer.

[0014] Hereinafter, with reference to the drawings, embodiments of a control system, a control method, a program, and a non-transitory recording medium for recording a program according to the present disclosure will be described in detail. The disclosed technology is not limited to these embodiments.

[0015] Furthermore, virtual reality may be referred to as VR (Virtual Reality). In addition, in the control system, a virtual reality space is displayed on a display device. The type of the display device is not particularly limited, and may be an HMD, a dome display, or the like. Furthermore, the usage scenario of the virtual reality space is not particularly limited.

[0016] First Embodiment First, in the first embodiment, basic functions of a control system will be described. Fig. 1 is a block diagram showing an example of the configuration of a control system according to the first embodiment. For example, a control system 10 includes an acquisition unit 101, a generation unit 102, and an output control unit 103.

[0017] The acquisition unit 101 acquires data associating imaging device identification information that identifies an imaging device installed in real space with frame identification information that identifies a frame in the VR space. The type of imaging device is not particularly limited, and a 360-degree camera or the like may be used. As a specific data acquisition method, the acquisition unit 101 may acquire data associating frame identification information that identifies a frame in the VR space visible to a user in the VR space with imaging device identification information that identifies an imaging device installed in real space from a database. Alternatively, the acquisition unit 101 may acquire data via an input device or a communication network. For example, the frame may be the frame of a vehicle or a building in the VR space. The vehicle may be, but is not limited to, a bus, an airplane, a train, a ship, etc. The building may be, but is not limited to, a company, a school, a house, etc. Examples of the frame include a window or a door. In the following description, for the sake of simplicity, the association between frame identification information and imaging device identification information may be referred to as the association between a frame and an imaging device. Also, for the sake of simplicity, an imaging device identified by imaging device identification information associated with frame identification information that identifies a frame in a VR space may be referred to as an imaging device associated with the frame.

[0018] Next, the generation unit 102 generates an image of the frame as seen by the user in the VR space from the image captured by the imaging device associated with the frame, based on the positional relationship in the VR space between the user in the VR space and the frame in the VR space. Examples of the positional relationship include the distance from the frame, the height position, and the horizontal position relative to the frame (left and right). A specific method for generating the image will be described in detail in the second embodiment.

[0019] The output control unit 103 displays the image of the generated frame within the frame in the VR space displayed on the display device.

[0020] 2 is a flowchart showing an example of an operation of the control system 10 according to the first embodiment. The acquisition unit 101 acquires data that associates an imaging device installed in real space with a frame in the VR space (step S101).

[0021] The generation unit 102 generates an image of the frame in the VR space from an image captured by an imaging device associated with the frame in the VR space based on the positional relationship in the VR space between the user in the VR space and the frame in the VR space (step S102). The output control unit 103 displays the generated image of the frame within the frame in the VR space (step S103).

[0022] In real space, the view seen through a frame such as a window or door of a building or vehicle varies depending on the user's position. For example, in a VR space, the view seen through a frame such as a window or door of a building or vehicle may be from a fixed viewpoint. In such cases, there is a problem of a low sense of realism. In embodiment 1, the control system 10 generates an image of a frame in the VR space as seen by the user in the VR space from an image captured by an imaging device associated with the frame in the VR space, based on the positional relationship in the VR space between the user in the VR space and the frame in the VR space. Then, the control system 10 displays the image of the generated frame within the frame in the VR space. This makes it possible to provide an image of the real space as seen from the user's position in the VR space. Therefore, the sense of realism can be improved.

[0023] (Embodiment 2) Next, embodiment 2 will be described in detail with reference to the drawings. In embodiment 2, an imaging device is installed on a moving object in real space, and a travel is used as an example of a usage scene of the VR space, and a vehicle window is used as an example of a frame in the VR space. Below, explanations of content that overlaps with the above explanation will be omitted to the extent that the explanation of embodiment 2 is not unclear.

[0024] 3 is an explanatory diagram showing an example of a dome-shaped display used as a display device. In FIG. 3, a dome-shaped display 21 is taken as an example of the display device. In real space, for example, a user is sitting in front of the dome-shaped display 21.

[0025] When the dome display 21 displays an image, the projection device 2102 projects the image onto the screen 2101 .

[0026] An imaging device 22 and a recording device 23 may be installed where the dome-shaped display 21 is installed. For example, the imaging device 22 and the recording device 23 are used to detect the direction of the user's face, the direction of the user's gaze, the user's movements, and the user's conversation. The control system can detect the user's position and movements from the video captured by the imaging device 22. The control system can detect the user's conversation from the audio acquired by the recording device 23. Although not shown, an audio output device such as a speaker that outputs the voices of other users and the voices of a moving object is also installed where the dome-shaped display 21 is installed. A controller 24 may also be installed where the dome-shaped display 21 is installed. The controller 24 is, for example, an example of an input device that accepts user operations. For example, in the second embodiment, the controller 24 accepts input of the user's movement through the user's operation.

[0027] There is no particular limitation on the number of imaging devices 22 and recording devices 23. For example, the imaging devices 22 may transmit captured images to a control system or the like, and the recording devices 23 may transmit audio to a control system or the like.

[0028] The dome-shaped display 21 is, for example, a device in which at least a portion of the screen 2101 is curved, so as to cover the user's field of view. The dome-shaped display 21 does not have to be a complete dome shape, such as a 360-degree dome, but may have a missing portion, such as a 180-degree dome. That is, the dome-shaped display 21 may have a hemispherical dome shape. Furthermore, the dome-shaped display 21 does not require that a portion of the screen 2101 be curved. Thus, the size and type of the dome-shaped display 21 are not particularly limited. For example, the dome-shaped display 21 may be a 180-degree dome display 21 or a 360-degree dome display 21. Furthermore, the size of the dome-shaped display 21 may be, for example, approximately 1 to 2 meters in length, 1 to 2 meters in width, and 1 to 2 meters in height.

[0029] Furthermore, there is no particular limitation on the installation location of the dome-shaped display 21. For example, the dome-shaped display 21 may be installed in the user's home, office, or the like, or may be installed in a location that anyone can use.

[0030] 4 is an explanatory diagram showing an example of a connection between the control system and the dome-shaped display 21 and the like. The dome-shaped display 21, the imaging device 22, the sound recording device 23, and the controller 24 are installed in the user's home, a shared space, or the like. The control system 20 is connected to the dome-shaped display 21, the imaging device 22, the sound recording device 23, the controller 24, and the like via a communication network. Note that, for example, an edge terminal device or the like may be installed on the user side. The edge terminal device may be connected to the dome-shaped display 21, the imaging device 22, the sound recording device 23, the controller 24, and the like via a communication network, and the control system 20 may be connected to the edge terminal device via the communication network or the like.

[0031] For example, the imaging device 26 in the real space may be installed on a moving body 25. Alternatively, the imaging device 26 may be attached to a robot arm or the like. The imaging device 26 is connected to the control system 20 via a communication network. The number of imaging devices 26 is not particularly limited.

[0032] The control system 20 may also be configured as an entire system including the dome display 21 , the imaging device 22 , the sound recording device 23 , the imaging device 26 , and the controller 24 .

[0033] Furthermore, the communication network through which the control system 20 is connected to the device on the user side and the communication network through which the control system 20 is connected to the device on the mobile object 25 side may be the same.

[0034] 5 is an explanatory diagram showing an example of an HMD used as a display device. In FIG. 5, an HMD 27 is taken as an example of the display device. In real space, a user wears the HMD 27.

[0035] The HMD 27 may have a function to detect, for example, the direction of the user's face, the direction of the user's gaze, and the user's movements. The HMD 27 may also have a function to output audio and collect audio. The controller 24 may also be used to input the user's movements.

[0036] 6 is an explanatory diagram showing an example of a connection between the control system 20 and the HMD 27, etc. The control system 20 is connected to the HMD 27, the controller 24, etc. via a communication network. Note that, for example, an edge terminal device or the like may be installed on the user side. The edge terminal device may be connected to the HMD 27, the controller 24, etc. via a communication network, and the control system 20 may be connected to the edge terminal device via the communication network, etc.

[0037] 4, the imaging device 26 is installed on the moving body 25. Alternatively, the imaging device 26 may be attached to a robot arm or the like. The imaging device 26 is connected to the control system 20 via a communication network. The number of imaging devices 26 is not particularly limited.

[0038] The control system 20 may also be configured as an entire system including the HMD 27 , the imaging device 26 , and the controller 24 .

[0039] FIG. 7 is an explanatory diagram showing an example of seats in a moving body 25. When the window w1 in the VR space is the window w1 of a vehicle such as a bus or train, there may be seats in the VR space. For example, in FIG. 7, there are seats A to D. Here, seat A is the seat of user X. Seat B is the seat of user Y. Seat C is the seat of user Z.

[0040] 8 is a block diagram showing an example of a configuration of a control system 20 according to embodiment 2. For example, the control system 20 includes an acquisition unit 201, a generation unit 202, an output control unit 203, a detection unit 204, an identification unit 205, a reception unit 206, a product determination unit 207, a registration unit 208, and a settlement unit 209.

[0041] The control system 20 further includes a detection unit 204, an identification unit 205, a reception unit 206, a product determination unit 207, a registration unit 208, and a settlement unit 209 in addition to the control system 10 of the first embodiment.

[0042] The acquiring unit 201 has, as a basic function, the function of the acquiring unit 101 according to the first embodiment. The generating unit 202 has, as a basic function, the function of the generating unit 102 according to the first embodiment. The output control unit 203 has, as a basic function, the function of the output control unit 103 according to the first embodiment.

[0043] For example, the control system 20 has an association table 2001, a user DB (Database) 2002, and a product DB 2003. Each functional unit of the control system 20 can refer to and update various databases and tables as appropriate.

[0044] The association table 2001 stores data associating the imaging devices 26 in real space with windows in the VR space. Specifically, the association table 2001 stores, for example, imaging device identification information for identifying the imaging devices 26 and window identification information for identifying windows, in association with each other. The imaging device identification information may be any information capable of identifying the imaging devices 26, and the representation format of the imaging device identification information is not particularly limited. The window identification information may be any information capable of identifying the windows, and the representation format of the window identification information is not particularly limited. Note that multiple imaging devices 26 may be associated with one window. An example of the storage of the association table 2001 will be described later using drawings along with an example of installation of the imaging devices 26.

[0045] Furthermore, the user DB 2002 stores user information for each user. Examples of user information include the user's name and an image of the user. The image of the user may be a photograph of the actual user or an avatar, and is not particularly limited. The image of the user is used, for example, as an image of the user in a VR space. For example, the user DB 2002 may store display device identification information that identifies the display device and user information in association with each other for each user. Furthermore, the user DB 2002 may separately store display device identification information, user identification information that identifies the user, and user information in association with each other.

[0046] Furthermore, the product DB 2003 stores product information for each product. The product DB 2003 stores product identification information and product information in association with each other. The product identification information is not particularly limited as long as it can identify the product. The product information may include, for example, the product name, product price, product image, product features, etc.

[0047] Next, the processing of each functional unit will be described by taking a specific example of associating the image capture device 26 in the real space with a window in the VR space.

[0048] <Example of Associating One Imaging Device 26 with a Window> First, an example of associating one imaging device 26 in real space with a window in VR space will be described.

[0049] Fig. 9 is an explanatory diagram showing an example in which an imaging device 26 is installed on a moving body 25. For example, in real space, the imaging device 26 is installed on the moving body 25. In Fig. 9, the moving body 25 is exemplified as a vehicle such as a truck, but is not particularly limited to an airplane, a train, a drone, a ship, or the like.

[0050] 10 is an explanatory diagram showing an example of the correspondence between one imaging device 26 in real space and a window in VR space. Although the imaging device 26 in real space is not installed at the window w1 in VR space, for ease of understanding, FIG. 10 shows the positional relationship between one imaging device 26 in real space and the window w1 in VR space.

[0051] 11 is an explanatory diagram showing an example of an association table 2001 that associates image capture devices 26 with windows. The association table 2001 stores image capture device IDs (Identifiers) and window IDs in association with each other. Here, the image capture device ID is an example of image capture device identification information that identifies the image capture device 26. The window ID is an example of window identification information that identifies the window.

[0052] In Fig. 11, an imaging device ID "C0001" is associated with a window ID "w1." For example, the imaging device identified by the imaging device ID "C0001" is the imaging device 26 shown in Fig. 9. An imaging device ID "C0002" is associated with a window ID "w2." An imaging device ID "C0003" is associated with a window ID "w3." An imaging device ID "C0004" is associated with a window ID "w4."

[0053] The association table 2001 may store position information or location information indicating where the imaging device 26 is installed in real space, orientation information indicating the orientation of the imaging device 26 in real space, etc., in association with the imaging device ID.

[0054] The acquisition unit 201 acquires an association table 2001 as data associating, for example, the image capture device 26 installed in the real space with a window in the VR space.

[0055] Next, the generation unit 202 generates an image of the window as seen by the user in the VR space from the image captured by the imaging device 26 associated with the window, based on the positional relationship in the VR space between the user in the VR space and the window in the VR space.

[0056] When there are multiple users, for example, the generation unit 202 generates an image of the window visible to the user in the VR space from the image captured by the camera for each of the multiple users, based on the positional relationship in the VR space between the user in the VR space and the window in the VR space.

[0057] The generation unit 202 generates a new image of the frame seen by the user in the VR space based on the positional relationship between the user's position in the VR space and the position of the window in the VR space, and the orientation of the user in the VR space. The orientation of the user includes, for example, the orientation of the user's face or head, the direction of the user's line of sight, etc.

[0058] Specifically, as a method of generating an image of a window, for example, the generation unit 202 may cut out an image of a window visible to the user in the VR space from the captured image based on the positional relationship in the VR space between the user in the VR space and the window in the VR space.

[0059] 12 is an explanatory diagram showing an example of cutting out an image of a window from a captured image. In FIG. 12, the generation unit 202 cuts out an image of the window w1 seen by the user X in the VR space from the captured image based on the positional relationship in the VR space between the user X in the VR space and the window w1 in the VR space and the direction of the user's line of sight.

[0060] Also, in Figure 12, the generation unit 202 cuts out an image of window w1 visible to user X in the VR space from the captured image based on the positional relationship in the VR space between user X in the VR space and window w1 in the VR space and the direction of the user's line of sight.

[0061] As another method for generating an image of a window, for example, the generation unit 202 generates a new image of a frame visible to the user in the VR space from the image captured by the imaging device 26 based on the positional relationship in the VR space between the user in the VR space and the window w1 in the VR space. Generating a new image may involve adding another image to the captured image. More specifically, for example, the generation unit 202 may generate a new image to complement an image that is not visible due to weather conditions in the captured image. The generation unit 202 is not limited to combining, correcting, or complementing past images, but may also combine another image, text, or the like with the captured image.

[0062] The methods for generating each image may be combined.

[0063] <Example of Associating Multiple Image Capturing Devices 26 with Windows> Next, an example of associating multiple image capturing devices 26 in real space with windows in VR space will be described.

[0064] 13 is an explanatory diagram showing an example in which a plurality of image capturing devices 26 are installed on a moving body 25. For example, in real space, two image capturing devices 26-1 and 26-2 are installed on the moving body 25 at different positions.

[0065] If the imaging device 26 is a camera other than a 360-degree camera, the imaging devices 26-1 and 26-2 may be installed in the same position but facing different directions.

[0066] 14 is an explanatory diagram showing an example of the correspondence between two image capture devices 26 in real space and a window in VR space. Although an image capture device 26 is not installed at window w1 in VR space, for ease of understanding, FIG. 14 shows the positional relationship between the two image capture devices 26-1 and 26-2 in real space and the window w1 in VR space.

[0067] 15 is an explanatory diagram showing an example of an association table 2001 that associates a plurality of image capture devices 26 with windows. The association table 2001 stores image capture device IDs (Identifiers) and window IDs in association with each other. As with the example of the association table 2001 in FIG. 11 , the image capture device IDs in FIG. 15 are an example of image capture device identification information that identifies the image capture devices 26. The window IDs are an example of window identification information that identifies the windows.

[0068] 15, in the association table 2001, an imaging device ID "C0001" is associated with a window ID "w1," and an imaging device ID "C0002" is associated with a window ID "w1." For example, the imaging device identified by the imaging device ID "C0001" is the imaging device 26-1 shown in FIG. 13, and the imaging device identified by the imaging device ID "C0002" is the imaging device 26-2 shown in FIG. 13.

[0069] In addition, the association table 2001 may store position information and location information indicating where each of the multiple imaging devices 26 is installed in real space, and orientation information indicating the orientation of the device in real space, in association with the imaging device ID.

[0070] Next, the acquisition unit 201 acquires an association table 2001 as data associating a plurality of image capture devices 26 installed at different positions in the real space with one window in the VR space.

[0071] Based on the positional relationship in the VR space between the user in the VR space and a window in the VR space, the generation unit 202 generates a video from videos captured by the multiple imaging devices 26 associated with the window in the association table 2001. The multiple imaging devices 26 associated with the window in the association table 2001 are the multiple imaging devices 26 identified by each of the multiple imaging device IDs associated with the window ID in the association table 2001.

[0072] For example, the generation unit 202 generates an image seen by the user in the VR space from multiple images based on the positional relationship in the VR space between the user in the VR space and a window in the VR space. Specifically, as a method of generating an image seen by the user from multiple images, the generation unit 202 may combine multiple images and then extract an image seen by the user in the VR space from the combined image. When the window is large, a large number of imaging devices 26 may be installed to increase the density, allowing the generation unit 202 to combine multiple images to generate a more detailed image. Alternatively, the generation unit 202 may generate an image seen by the user in the VR space by extracting an image seen by the user in the VR space from each of the multiple images and then combining the extracted images. Specifically, for example, the generation unit 202 may determine a coordinate position from which to extract the image from each of the multiple images and then composite the multiple images based on the determined coordinate positions. When the window is small and multiple imaging devices are installed without high density, there is no need to synthesize the parts of the image that the user does not see, so the generation unit 202 can generate a new image of the window that is visible to the user in the VR space in a short time.

[0073] FIG. 16 is an explanatory diagram showing an example of generating a window image from images captured by each of multiple imaging devices 26. In FIG. 16, the generation unit 202 generates an image seen by user X in the VR space from images captured by multiple imaging devices 26 based on the positional relationship in the VR space between user X in the VR space and window w1 in the VR space and the direction of the user's line of sight. In FIG. 16, the generation unit 202 generates an image seen by user X in the VR space from images captured by imaging devices 26-1 and 26-2. Note that while FIG. 16 shows an example in which the ranges captured by the multiple images captured by imaging devices 26-1 and 26-2 do not overlap, there may be cases in which the ranges captured by the multiple images overlap. In such a case, there is no particular limitation on which image the generation unit 202 uses for the overlapping portion.

[0074] More specifically, the generation unit 202 may synthesize multiple videos to generate one video, and then extract from the single video a video that can be seen by the user X in the VR space. Alternatively, the generation unit 202 may extract from each of the multiple videos a video that can be seen by the user X in the VR space, and synthesize the multiple extracted videos.

[0075] Furthermore, the generation unit 202 generates an image for user Y that can be seen by user X in the VR space from a plurality of images, in the same way as for user X. This concludes the description of the example in FIG.

[0076] As another method for generating an image visible to the user from a plurality of images, the generation unit 202 selects an image including an image visible to the user in the VR space from the plurality of images based on the positional relationship in the VR space between the user in the VR space and a window in the VR space. Then, the generation unit 202 may extract an image visible to the user in the VR space outside the window in the VR space from the selected image.

[0077] FIG. 17 is an explanatory diagram showing an example of selecting and generating a window image from images captured by each of multiple imaging devices 26. In FIG. 17 , the generation unit 202 generates an image seen by user X in the VR space from the images captured by the multiple imaging devices 26 based on the positional relationship in the VR space between user X in the VR space and window w1 in the VR space, and the direction of the user's line of sight in the VR space. Specifically, the generation unit 202 selects the image captured by imaging device 26-1 from the images captured by imaging device 26-1 and the images captured by imaging device 26-2, based on the positional relationship in the VR space between user X in the VR space and window w1 in the VR space, and the direction of the user's line of sight in the VR space. Then, the generation unit 202 extracts the image seen by user X in the VR space from the selected images.

[0078] This concludes the description of an example of generating an image seen by a user in a VR space from multiple images. Note that the generation process described in the case where one imaging device 26 is associated with a window and the generation process described in the case where multiple imaging devices 26 are associated with windows may be combined as appropriate.

[0079] Next, the output control unit 203 displays the generated image within a frame in the VR space displayed on the display device. As described above, the display device is not particularly limited, but examples thereof include the HMD 27 and the dome-shaped display 21. Here, the dome-shaped display 21 is taken as an example of the display device.

[0080] 18 is an explanatory diagram showing a display example of the dome-type display 21. For example, the dome-type display 21 displays an image of the inside of a vehicle in a VR space.

[0081] 18, more specifically, user Y, user Z, and window w1 are displayed in the VR space for user X on the dome-shaped display 21. In the VR space, an image generated by the generation unit 202 is displayed in the window w1.

[0082] <User Movement> Next, a case where a user moves in the VR space will be described. For example, there is an example where the user moves in the VR space in conjunction with the movement of the user in the real space, and an example where the user moves in the VR space by operating an input device such as the controller 24 in the real space by the user in the real space. Note that which example is used may be determined appropriately and is not particularly limited. When the display device is the HMD 27, either example is likely to be used. For example, when the display device is a dome-shaped display 21, there is a high possibility that the user moves in the VR space by operating an input device such as the controller 24.

[0083] First, an example of linking with the movement of the user in the real space will be described.

[0084] The setting unit may associate the user's position and orientation in the real space with the user's position and orientation in the VR space at the start of use. The setting unit may store the associated information as a table in a storage unit or the like.

[0085] Next, the detection unit 204 detects the movement of the user in real space. The movement of the user in real space includes facial movement, head movement, body movement, etc. More specifically, the movement of the user includes changes in the position of the face, the position of the head, the direction of the face, the direction of the head, the direction of the line of sight, etc. When the display device is the HMD 27, the detection unit 204 may detect the movement of the user based on data detected by an acceleration sensor, a gyro sensor, a geomagnetic sensor, etc. provided in the HMD 27. When the display device is the HMD 27 or the dome-shaped display 21, the detection unit 204 may detect the movement of the user from an image captured by the imaging device 22.

[0086] The identification unit 205 identifies the new position and orientation of the user in the VR space based on the detected movement of the user. Note that, for example, the amount of movement of the user in the real space and the amount of movement of the user in the VR space are proportional, and the magnification is not particularly limited. For example, the amount of movement of the user in the VR space may be several times larger than the amount of movement of the user in the real space.

[0087] Then, the generation unit 202 generates a new image of the window as seen by the user in the VR space from the image captured by the imaging device 26 associated with the window, based on the user's new position in the VR space, the positional relationship in the VR space between the user and the window in the VR space, and the user's new orientation in the VR space.

[0088] Next, an example will be described in which a user in real space moves a user in the VR space via an input device. For example, the reception unit 206 detects the user's movement in the VR space by receiving the user's movement in the VR space through the user's operation on the input device. Note that the input device may be the controller 24 or a terminal device, and is not particularly limited.

[0089] The identification unit 205 identifies the new position and orientation of the user in the VR space based on the received movement of the user in the VR space.

[0090] Then, the generation unit 202 generates a new image of the window as seen by the user in the VR space from the image captured by the imaging device 26 associated with the window, based on the user's new position in the VR space, the positional relationship in the VR space between the user and the window in the VR space, and the user's new orientation in the VR space.

[0091] Then, the generation unit 202 generates a new image of the window as seen by the user in the VR space from the image captured by the imaging device 26 associated with the window, based on the user's new position in the VR space, the positional relationship in the VR space between the user and the window in the VR space, and the user's new orientation in the VR space.

[0092] <Graphical display of audio> For example, a group trip by vehicle may be held in a VR space. In the case of a group trip by vehicle, the users are travelers. In the case of a group trip, some of the users may know each other, while other users may not. While a group trip has the advantage of being able to share information, such as listening to conversations between users who are not acquaintances, it also has the disadvantage of having to listen to conversations that do not interest users.

[0093] Therefore, in the second embodiment, the control system 20 may allow the user to selectively hear the sound.

[0094] Specifically, for example, the output control unit 203 causes a display device to display a graphic representing the voice of a user other than the user who is conversing with the user. Note that, for example, the output control unit 203 may display the graphic together with the voice, or may display the graphic with the voice muted. Here, for example, the generation unit 202 generates a video in which a graphic representing the voice of the other user is added to the captured video. Then, the output control unit 203 causes the generated video to be displayed on the display device.

[0095] Here, graphics include photographs, illustrations, figures, symbols, letters, etc. Graphics may be graphics corresponding to audio. Audio-related graphics may have different shapes, sizes, colors, patterns, etc. depending on the audio. For example, the color of a graphic may be different for audio related to travel and audio unrelated to travel. For example, the output control unit 203 may highlight a graphic representing audio related to travel more than a graphic representing audio unrelated to travel. Similarly, in the case of graphic patterns, the pattern for audio related to travel may be different from the pattern for audio unrelated to travel.

[0096] For example, users other than the user who is having a conversation with the user may be grouped in advance, or may be grouped according to the conversation. "Pre-grouped" means that users who will travel together are registered in advance before use begins, and the registered users traveling together are treated as one group. For example, "grouped according to the conversation" means that if user X, user Y, and user Z are having a conversation, the users involved in the same conversation are grouped into one group. In this case, the groups change periodically.

[0097] 19 is an explanatory diagram showing an example of a graphic display of the voices of other users. For example, in FIG. 19, speech bubbles are used as the graphics. Note that the shape of the graphics is not limited to the shape of speech bubbles, and is not particularly limited as long as it is possible to identify that it represents voice.

[0098] There are three speech bubbles in Fig. 19. That is, in Fig. 19, there are three people speaking or three groups of conversations. Here, user Z is a user who is not conversing with user X, and the voice of user Z is assumed to be travel-related. On the other hand, the voices of the other users are assumed to be voices unrelated to travel. For example, the output control unit 203 highlights the graphic representing the voice related to travel more than the graphic representing the voice unrelated to travel.

[0099] In Fig. 19, the size and pattern of the graphic are used for highlighting. In Fig. 19, the size of the speech bubble representing user Z's voice is larger than the sizes of the other speech bubbles. Also, the pattern of the speech bubble representing user Z's voice is a dotted pattern, while the patterns of the other speech bubbles are solid. In this way, the speech bubble representing user Z's voice is emphasized more than the other speech bubbles. Also, the color of the graphic may be used for highlighting.

[0100] The receiving unit 206 may also receive a selection of a graphic representing audio through a user operation. The user operation may be, for example, an operation via an input device, an operation corresponding to the user's hand movement, or an operation corresponding to the user's voice; the operation method is not particularly limited. The input device may be, for example, the controller 24 or a terminal device, as described above. The user's hand movement may be detected, for example, from an image captured by the imaging device 26. The user's voice may be obtained, for example, by a recording device. The output control unit 203 then outputs the audio represented by the graphic selected by the user. Alternatively, for example, the output control unit 203 may not display the graphic selected by the user. Specifically, for example, the generating unit 202 adds a graphic other than the selected graphic to the captured video, thereby generating a video without the selected graphic. The output control unit 203 then displays the generated video on a display device.

[0101] Furthermore, for example, if the display device is a dome-shaped display 21 that does not cover the entire surface of the user, the output control unit 203 may output the conversation behind the user as audio and display the conversation in front of the user graphically without audio output. The front of the user is the direction the user is facing, and the back of the user is the direction opposite to the direction the user is facing. An example of a dome-shaped display 21 that does not cover the entire surface of the user is a 180-degree dome-shaped display 21.

[0102] 20 is an explanatory diagram showing an example of switching from graphic output to audio output. For example, in FIG. 20, the output control unit 203 displays a speech bubble representing the voice of user Z on the dome-shaped display 21. The reception unit 206 receives a graphic selection by operation of user X. The output control unit 203 outputs the voice of user Z represented by the selected graphic as audio.

[0103] Furthermore, the receiving unit 206 may receive an output format for the voice of another user. The output control unit 203 outputs the voice of another user in the received format. Specifically, for example, the receiving unit 206 may be able to select, in stages, between a graphic representing the voice and the voice itself. The receiving unit 206 may also be able to select, in stages, the graphic. Then, the output control unit 203 switches the output in stages. For example, the output control unit 203 may switch in stages so that more detailed content of the voice is displayed, such as a graphic representing a predetermined shape with a color and a graphic representing the voice with text.

[0104] It should be noted that group travel is an example of a usage scenario, and the system can also be used in other scenarios, such as when there are multiple users in a building.

[0105] Furthermore, the moving body 25 may be a vehicle. For example, if the moving body 25 is a vehicle, the moving body 25 may be installed outside a window of an actual vehicle, and the vehicle in the VR space may be a reproduction of the actual vehicle. The other user displayed in the virtual reality space may be a person riding in the actual vehicle. In such a case, the vehicle in the real space may be equipped with an imaging device that captures the interior of the vehicle in the real space. Then, the output control unit 203 may display an image of the captured interior of the vehicle as the interior of the vehicle in the virtual reality space displayed on the display device. Note that, if an imaging device is captured in the interior image, the generation unit 202 may generate an image corrected to remove the imaging device, and the output control unit 203 may display the generated image as an image of the vehicle in the virtual reality space.

[0106] <Shopping> For example, shopping may be done during a trip in a VR space.

[0107] The product determination unit 207 determines recommended products from the products included in the product DB 2003. Here, the products included in the product DB 2003 are products identified by product identification information stored in the product DB 2003. Specifically, for example, the product determination unit 207 determines products related to travel destinations as recommended products. Furthermore, the product determination unit 207 may determine recommended products from among products related to travel destinations based on conversations of the user.

[0108] For example, the output control unit 203 presents information about recommended products to the user. As a specific presentation method, the output control unit 203 may display the information about recommended products on a display device, or may notify the information about recommended products to a terminal device of the user.

[0109] For example, products may be sold in a manner similar to in-train or in-flight sales. Therefore, the output control unit 203 may also display an avatar of a store clerk on the display device, and the avatar of the store clerk may present information about recommended products.

[0110] The registration unit 208 accepts product registration through user operation. The settlement unit 209 settles the payment for the registered product through user operation. Note that existing technology may be used for the product registration method and the payment method.

[0111] 21 is a flowchart showing an example of an operation of the control system 20 according to the second embodiment. Here, an example will be described in which the movement of a user in the real space is linked to the movement of a user in the VR space.

[0112] First, the acquisition unit 201 acquires data that associates image capture device identification information that identifies the image capture device 26 installed in real space with frame identification information that identifies a frame in the VR space (step S201). For example, the acquisition unit 201 acquires an association table 2001 as this data.

[0113] As an initial setting, the setting unit associates the user's position in the VR space with the user's position in the real space (step S202).

[0114] Next, the generation unit 202 generates an image of the frame in the VR space from an image captured by the imaging device 26 associated with the frame in the VR space, based on the positional relationship in the VR space between the frame in the VR space and the user in the VR space, and the orientation of the user in the VR space (step S203). The output control unit 203 displays the generated image of the frame within the frame in the VR space (step S204).

[0115] The detection unit 204 determines whether movement of the user in real space has been detected (step S205). If movement of the user in real space has not been detected (step S205: No), the detection unit 204 returns to step S205.

[0116] If movement of the user in real space is detected (step S205: Yes), the identification unit 205 identifies the new position and orientation of the user in the VR space (step S206), and the process returns to step S203. This generates a new image of the frame visible to the user in the VR space at the user's new position and orientation in the VR space.

[0117] The flowchart may be terminated as appropriate.

[0118] In the second embodiment, the control system 20 extracts an image of a frame visible to the user in the VR space from the image captured by the imaging device 26. This makes it possible to provide an image of the area outside the frame that the user can see in the VR space. Alternatively, the control system 20 generates a new image of the frame that the user can see in the VR space from the image captured by the imaging device 26 based on the positional relationship in the VR space between the user in the VR space and the frame in the VR space. This makes it possible to provide an image of the area outside the frame that the user can see in the VR space. It is also possible to complement an image that is not present in the actual experience.

[0119] Furthermore, when multiple imaging devices 26 installed at different locations are associated with a frame, the control system 20 generates an image of the frame visible to the user in the VR space from the multiple images, thereby providing an image of the area outside the frame that can be seen by the user in the VR space.

[0120] Furthermore, for each of a plurality of users, the control system 20 generates an image of the frame visible to the user in the VR space from the image captured by the imaging device 26, based on the positional relationship in the VR space between the user in the VR space and the frame in the VR space. This makes it possible to provide an image according to the user's position in the VR space from images captured by the same imaging device 26 installed in real space, even if there are a plurality of users in the VR space.

[0121] Furthermore, the control system 20 generates a new image of the frame as seen by the user in the VR space based on the positional relationship in the VR space between the user in the VR space and the frame in the VR space, and the orientation of the user in the VR space. This allows the image to differ depending on the orientation of the user, making it possible to provide an image that is closer to reality.

[0122] Furthermore, the control system 20 detects the user's movement in real space and identifies the user's position and orientation in the VR space based on the user's movement in real space. The control system 20 then generates a new image of the frame as seen by the user in the VR space based on the positional relationship in VR space between the user in the identified position and the frame in the VR space, as well as the new orientation. This allows the image to change depending on the user's up / down and left / right movements in real space, thereby improving the sense of realism of the image seen outside the frame.

[0123] The control system 20 also receives user movement in the VR space through user operation in real space. Based on the received movement, the control system 20 identifies the user's position and orientation in the VR space, and generates a new image of the frame as seen by the user in the VR space based on the positional relationship in the VR space between the user and the frame in the VR space at the identified position and the user's new orientation in the VR space. This allows the image to change depending on the user's up / down or left / right movements, thereby improving the sense of realism of the image seen outside the frame.

[0124] The frame may be a window or a door. For example, the window may be a window of a vehicle or a window of a building. This allows the view seen from the window of a vehicle in the VR space to be changed depending on the user's position in the VR space. Also, the view seen from the window of a building in the VR space to be changed depending on the user's position in the VR space. Similarly, the door may be a window of a vehicle or a door of a building. This allows the sense of realism of the image seen outside the window or door to be improved. In the case of a door, when the door is open, the control system 20 may display the generated image inside the door.

[0125] Furthermore, the imaging device 26 may be installed on a moving object 25 in real space. For example, as the moving object 25 moves, the image captured by the imaging device 26 also changes. For example, the scenery seen from the window of a building or vehicle changes as the moving object 25 moves. This makes it possible to provide a travel experience, etc. Furthermore, the moving object 25 may move in real time. This makes it possible to provide real-time images.

[0126] Furthermore, the control system 20 displays, in graphics, the voices of users other than the user who is conversing with the user. Users want to enjoy realistic images, or in the case of travel, enjoy group trips on vehicles. However, there are times when users want to hear what other users are saying, and times when they don't want to. For example, users want to hear information that is useful to them. This allows users, for example, to not be bothered by the conversations of other users if they don't look at the graphics. On the other hand, if a user is interested, they can just look at the graphics. In this way, the control system 20 can provide a better video experience by combining the benefits of real travel with the benefits of VR space.

[0127] The control system 20 also outputs the sound represented by a graphic selected by the user from among the displayed graphics, allowing the user to, for example, hear a conversation in which the user is interested.

[0128] The control system 20 does not display the graphics selected by the user, thereby allowing the user to remove from view the graphics representing conversations in which the user is not interested.

[0129] In addition, if the display device is a hemispherical dome-shaped display 21, the control system 20 may output conversations behind the user in the VR space as audio and display conversations in front of the user in the VR space in graphics.

[0130] The control system 20 can accept an output format for the voices of other users and output the voices of other users in the accepted output format. For example, the control system 20 may allow selection of the output format in stages, in the order of color, text, and sound. This allows the output format of the conversations of other users to be changed according to the user's interests.

[0131] The above is the explanation of each embodiment. The embodiments may be used in combination as appropriate.

[0132] In each embodiment, the control system 20 may be configured to include some of the functional units and information. In the second embodiment, the control system 20 is described as having some of the functions of a POS (Point of Sale) system, such as the registration unit 208 and the settlement unit 209. The POS system including the registration unit 208 and the settlement unit 209 may be connected to the control system 20 via a communication network.

[0133] Furthermore, the embodiments are not limited to the examples described above and can be modified in various ways. Furthermore, the configuration of the control system 20 in each embodiment is not particularly limited. For example, the control system 20 may be realized by a single device, such as a single server. When each functional unit of the control system 20 is realized by a single device, the single device may be called, for example, a control device, an information processing device, or the like, and is not particularly limited. Alternatively, the control system 20 in each embodiment may be realized by different devices for different functions or data. For example, each functional unit may be configured by multiple servers and realized as the control system 20. For example, the control system 20 may be realized by a database server including each DB and a server having each functional unit.

[0134] In each embodiment, each piece of information or each DB may include a portion of the information described above. Each piece of information or each DB may also include information other than the information described above. Each piece of information or each DB may be divided into multiple DBs or multiple pieces of information in more detail. Thus, the method for realizing each piece of information or each DB is not particularly limited.

[0135] Furthermore, each screen is merely an example and is not particularly limited. Buttons, lists, check boxes, information display fields, input fields, etc. (not shown) may be added to each screen. Furthermore, the background color of the screen may be changed.

[0136] Furthermore, the process of generating information to be displayed on the display device may be performed by the output control units 103 and 203. This process may also be performed by the display device.

[0137] (Example of Computer Hardware Configuration) Next, an example of a hardware configuration in which each device, such as the control systems 10 and 20 described in each embodiment, is implemented by a computer will be described. Fig. 22 is an explanatory diagram showing an example of a computer hardware configuration. For example, some or all of each device can be implemented using any combination of a computer 80 and a program as shown in Fig. 22.

[0138] The computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. The computer 80 also includes a communication interface 805 and an input / output interface 806. The components are connected to each other, for example, via a bus 807. The number of each component is not particularly limited, and there may be one or more of each component.

[0139] The processor 801 controls the entire computer 80. Examples of the processor 801 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). The computer 80 includes a storage unit such as a ROM 802, a RAM 803, and a storage device 804. Examples of the storage device 804 include a semiconductor memory such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD). For example, the storage device 804 stores an operating system (OS) program, application programs, and programs according to the embodiments. Alternatively, the ROM 802 stores application programs and programs according to the embodiments. The RAM 803 is used as a work area for the processor 801 .

[0140] The processor 801 also loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. The processor 801 also functions as a part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

[0141] The communication interface 805 is connected to a communication network NT such as a LAN (Local Area Network) or a WAN (Wide Area Network) via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. As a result, the computer 80 is connected to external devices and external computers 80 via the communication networks NT. The communication interface 805 serves as an interface between the communication network NT and the inside of the computer 80. The communication interface 805 also controls the input and output of data from external devices and external computers 80.

[0142] Furthermore, the input / output interface 806 is connected to at least one of an input device, an output device, and an input / output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Examples of the input / output device include a touch panel display. Note that the input device, output device, and input / output device may be built into the computer 80 or may be external.

[0143] The hardware configuration of the computer 80 is an example. The computer 80 may have some of the components shown in FIG. 22 . The computer 80 may have components other than those shown in FIG. 22 . For example, the computer 80 may have a drive device or the like. The processor 801 may then read programs and data stored on a recording medium attached to the drive device or the like into the RAM 803. Examples of non-transitory tangible recording media include optical disks, flexible disks, magneto-optical disks, and USB (Universal Serial Bus) memories. As described above, the computer 80 may have input devices such as a keyboard and a mouse. The computer 80 may have an output device such as a display. The computer 80 may also have an input device, an output device, and an input / output device.

[0144] The computer 80 may also include various sensors (not shown). The types of sensors are not particularly limited. The computer 80 may also include an imaging device capable of capturing images or videos.

[0145] This concludes the description of the hardware configuration of each device. There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a different computer and program for each component. Furthermore, multiple components of each device may be realized by any combination of a single computer and program.

[0146] Furthermore, some or all of the components of each device may be realized by circuits for specific applications. Furthermore, some or all of the components of each device may be realized by general-purpose circuits including a processor such as an FPGA (Field Programmable Gate Array). Furthermore, some or all of the components of each device may be realized by a combination of circuits for specific applications and general-purpose circuits. Furthermore, these circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. The multiple integrated circuits may be connected via a bus or the like.

[0147] Furthermore, when some or all of the components of each device are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally located or distributed.

[0148] The control methods described in the respective embodiments are realized by being executed by the control systems 10 and 20. Furthermore, for example, the control methods are realized by having a computer such as a server or a terminal device execute a program prepared in advance.

[0149] The programs described in each embodiment are recorded on a computer-readable recording medium such as a HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. The programs are then read from the recording medium and executed by a computer. The programs may also be distributed via a communication network NT.

[0150] The functions of each of the components of the control systems 10 and 20 in each of the embodiments described above may be realized by dedicated hardware, such as a computer. Alternatively, each component may be realized by software. Alternatively, each component may be realized by a combination of hardware and software.

[0151] Although the present disclosure has been described above with reference to various embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of each of the present disclosures may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may also include embodiments in which the details described herein are appropriately combined or substituted as necessary. For example, details described using a particular embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of the descriptions does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations may be changed as long as it does not interfere with the content.

[0152] Some or all of the above-described embodiments can be described as follows: However, some or all of the above-described embodiments are not limited to the following.

[0153] (Supplementary Note 1) A control system comprising: an acquisition means for acquiring data associating imaging device identification information that identifies an imaging device installed in real space with frame identification information that identifies a frame in a virtual reality space; a generation means for generating an image of the frame as seen by the user in the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame, based on a positional relationship in the virtual reality space between the user in the virtual reality space and the frame; and an output control means for displaying the generated image of the frame within the frame that is displayed on a display device. (Supplementary Note 2) The control system described in Supplementary Note 1, wherein the generation means cuts out the image of the frame as seen by the user in the virtual reality space from the image captured by the imaging device, based on the positional relationship in the virtual reality space between the user in the virtual reality space and the frame. (Supplementary Note 3) The control system according to Supplementary Note 1, wherein the generation means generates a new image of the frame as seen by the user in the virtual reality space from an image captured by the imaging device, based on a positional relationship in the virtual reality space between the user and the frame. (Supplementary Note 4) The control system according to any one of Supplements 1 to 3, wherein the acquisition means acquires data associating imaging device identification information that identifies each of a plurality of imaging devices installed at different positions with the frame identification information that identifies the frame, and the generation means generates the image of the frame as seen by the user in the virtual reality space from a plurality of images captured by the plurality of imaging devices identified by the plurality of imaging device identification information associated with the frame identification information that identifies the frame, based on the positional relationship in the virtual reality space between the user and the frame. (Supplementary Note 5) The control system described in any one of Supplementary Notes 1 to 4, wherein the generation means generates, for each of a plurality of users, an image of the frame as seen by the user in the virtual reality space from an image captured by the imaging device based on a positional relationship in the virtual reality space between the user and the frame.(Supplementary Note 6) The control system according to any one of Supplements 1 to 5, wherein the generation means generates a new image based on a positional relationship in the virtual reality space between the user and the frame in the virtual reality space, and an orientation of the user in the virtual reality space. (Supplementary Note 7) The control system according to any one of Supplements 1 to 6, comprising: a setting means for associating a position and orientation of the user in the real space with a position and orientation of the user in the virtual reality space, a detection means for detecting movement of the user in the real space, and an identification means for identifying a position and orientation of the user in the virtual reality space based on the movement of the user in the real space, wherein the generation means generates a new image based on the positional relationship in the virtual reality space between the user and the frame in the virtual reality space at the identified position, and the identified orientation. (Supplementary Note 8) The control system according to any one of Supplements 1 to 7, comprising: a receiving means for receiving movement of the user in the virtual reality space by operation of the user in the real space; and a specifying means for specifying a position and orientation of the user in the virtual reality space based on the received movement, wherein the generating means generates a new image based on the positional relationship in the virtual reality space between the user in the virtual reality space at the specified position and the frame, and the specified orientation. (Supplementary Note 9) The control system according to any one of Supplements 1 to 8, wherein the frame is a window or a door. (Supplementary Note 10) The control system according to Supplementary Note 9, wherein the window is a window of a vehicle or a window of a building, and the door is a door of a vehicle or a door of a building. (Supplementary Note 11) The control system according to Supplementary Note 10, wherein the imaging device is installed on a moving body in the real space. (Supplementary Note 12) The control system according to any one of Supplementary Notes 1 to 11, wherein the output control means displays a graphic representing the voice of a user other than the user who is conversing with the user in the virtual reality space.(Supplementary Note 13) The control system according to Supplementary Note 12, further comprising: a receiving means for receiving a selection of a graphic from the graphics by the user in the real space, wherein the output control means causes a sound represented by the selected graphic to be output as a sound. (Supplementary Note 14) The control system according to Supplementary Note 12, further comprising: a receiving means for receiving a selection of a graphic from the graphics by the user in the real space, wherein the output control means does not cause the selected graphic to be displayed. (Supplementary Note 15) The control system according to any of Supplements 12 to 14, further comprising: when the display device is a hemispherical dome display, the output control means outputs a conversation behind the user in the virtual reality space as a sound and displays a conversation in front of the user in the virtual reality space as a graphic. (Supplementary Note 16) The control system according to any of Supplements 12 to 14, further comprising: a receiving means for receiving an output format for the voice of the other user, wherein the output control means outputs the voice of the other user in the received output format. (Supplementary Note 17) A control method comprising: acquiring data associating imaging device identification information that identifies an imaging device installed in real space with frame identification information that identifies a frame in a virtual reality space; generating an image of the frame as seen by the user in the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame, based on a positional relationship in the virtual reality space between the user in the virtual reality space and the frame; and displaying the generated image of the frame within the frame displayed on a display device.(Supplementary Note 18) A non-transitory computer-readable recording medium that records a program for causing a computer to execute the following processes: acquire data associating imaging device identification information that identifies an imaging device installed in real space with frame identification information that identifies a frame in virtual reality space; generate an image of the frame as seen by the user in the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame, based on a positional relationship in the virtual reality space between the user in the virtual reality space and the frame; and display the generated image of the frame within the frame displayed on a display device. (Supplementary Note 19) A computer is caused to execute the following process: acquire data associating imaging device identification information that identifies an imaging device installed in real space with frame identification information that identifies a frame in virtual reality space; generate an image of the frame as seen by the user in the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame, based on a positional relationship in the virtual reality space between the user in the virtual reality space and the frame; and display the generated image of the frame within the frame displayed on a display device.

[0154] 10, 20 Control system 21 Dome-shaped display 22 Imaging device 23 Sound recording device 24 Controller 25 Mobile object 26, 26-1, 26-2 Imaging device 80 Computer 101, 201 Acquisition unit 102, 202 Generation unit 103, 203 Output control unit 204 Detection unit 205 Identification unit 206 Reception unit 207 Product determination unit 208 Registration unit 209 Settlement unit 801 Processor 802 ROM 803 RAM 804 Storage device 805 Communication interface 806 Input / output interface 807 Bus 2101 Screen 2102 Projection device 2001 Association table 2002 User DB 2003 Product DB NT Communication network w1 Window X User Y User Z User

Claims

1. An acquisition means for acquiring data in which imaging device identification information for identifying an imaging device installed in the real space is associated with frame identification information for identifying a frame on the virtual reality space; A generation means for generating an image of the frame visible from the user on the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frame based on the positional relationship between the user on the virtual reality space and the frame on the virtual reality space; An output control means for displaying the generated image of the frame within the frame displayed on the display device; A control system comprising:

2. The generation means cuts out an image of the frame visible from the user on the virtual reality space from an image captured by the imaging device based on the positional relationship between the user on the virtual reality space and the frame on the virtual reality space. The control system according to claim 1.

3. The generation means generates a new image of the frame visible from the user on the virtual reality space from an image captured by the imaging device based on the positional relationship between the user on the virtual reality space and the frame on the virtual reality space. The control system according to claim 1.

4. The acquisition means acquires data in which imaging device identification information for identifying each of a plurality of imaging devices installed at different positions is associated with the frame identification information for identifying the frame, The generation means generates an image of the frame visible from the user on the virtual reality space from a plurality of images captured by the plurality of imaging devices identified by the plurality of imaging device identification information associated with the frame identification information for identifying the frame based on the positional relationship between the user on the virtual reality space and the frame on the virtual reality space. The control system according to claim 1.

5. The generation means generates an image of the frame visible from the user on the virtual reality space from an image captured by the imaging device based on the positional relationship between the position of the user on the virtual reality space and the frame on the virtual reality space for each of the plurality of users. The control system according to claim 1.

6. The generation means generates a new video based on the positional relationship in the virtual reality space between the user and the frame on the virtual reality space, and the orientation of the user on the virtual reality space. The control system according to claim 1.

7. Setting means for associating the position and orientation of the user in the real space with the position and orientation of the user on the virtual reality space; Detection means for detecting the movement of the user in the real space; Specifying means for specifying the position and orientation of the user on the virtual reality space based on the movement of the user in the real space; Comprising: The generation means generates a new video based on the positional relationship in the virtual reality space between the user on the virtual reality space at the specified position and the frame, and the specified orientation. The control system according to claim 1.

8. Receiving means for receiving the movement of the user on the virtual reality space by the operation of the user in the real space; Specifying means for specifying the position and orientation of the user on the virtual reality space based on the received movement; Comprising: The generation means generates a new video based on the positional relationship in the virtual reality space between the user on the virtual reality space at the specified position and the frame, and the specified orientation. The control system according to any one of claims 1 to 7.

9. A computer acquires data in which imaging device identification information for identifying an imaging device installed in the real space is associated with frame identification information for identifying a frame on the virtual reality space, generates an image of the frame as seen by the user on the virtual reality space from an image captured by the imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frame based on the positional relationship in the virtual reality space between the user on the virtual reality space and the frame, and causes the generated image of the frame to be displayed within the frame displayed on the display device. A control method for executing the process.

10. A computer acquires data in which imaging device identification information for identifying an imaging device installed in the real space is associated with frame identification information for identifying a frame on the virtual reality space, Based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space, a video captured by an imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frame is used to generate a video of the frame visible from the user in the virtual reality space. The generated video of the frame is displayed within the frame being displayed on the display device. A program for executing the processing.