Control system, control method, and program

Abstract

This control system comprises an acquiring unit, a generating unit, and an output control unit. The acquiring unit acquires data in which imaging device identification information for identifying an imaging device installed in a real space, and frame identification information for identifying a frame in a virtual reality space are associated with each other. The generating unit generates a video of frames that can be seen by a user in the virtual reality space from a video imaged by the imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frames, on the basis of the positional relationship in the virtual reality space between the user and the frames in the virtual reality space. The output control unit displays the generated video of the frames within the frames in the virtual reality space displayed on a display device.

Description

Technical Field

[0001] The present disclosure relates to a control system and the like.

Background Art

[0002] In a virtual reality space, it may be possible to display an image of the real space.

[0003] For example, Patent Document 1 describes displaying an image on an HMD (Head Mounted Display). Specifically, Patent Document 1 states that the image displayed on the HMD includes a virtual reality space or an augmented reality space in which the internal space and the scenery visible from a sightseeing vehicle are synthesized, and a character, and that the scenery visible from the sightseeing vehicle is synthesized on the window of the moving part.

[0004] Also, Patent Document 2 describes that in a virtual reality space, in order to interact, a virtual camera captures a conversation partner and the captured video is displayed on a virtual window.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] For example, in a virtual reality space, if the scenery visible through a frame such as a window or a door of a building or a vehicle is a fixed viewpoint, there is a problem that the sense of immersion is low.

[0007] An example of the object of the present disclosure is to provide a control system and the like that improve the sense of immersion.

Means for Solving the Problems

[0008] A control system in one aspect of the present disclosure includes: acquisition means for acquiring data that associates imaging device identification information for identifying an imaging device installed in real space with frame identification information for identifying a frame in virtual reality space; generation means for generating an image of the frame as seen by the user in the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frame, based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space; and output control means for displaying the generated image of the frame within the frame displayed on a display device.

[0009] A control method in one aspect of the present disclosure acquires data that associates imaging device identification information for an imaging device installed in real space with frame identification information for a frame in virtual reality space, generates an image of the frame as seen by the user in the virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information for the frame, based on the positional relationship between the user in the virtual reality space and the frame, and displays the generated image of the frame within the frame displayed on a display device.

[0010] A program in one aspect of the present disclosure causes a computer to acquire data that associates imaging device identification information for an imaging device installed in real space with frame identification information for a frame in virtual reality space, and, based on the positional relationship between a user in virtual reality space and the frame in the virtual reality space, generates an image of the frame as seen by the user in virtual reality space from an image captured by an imaging device identified by the imaging device identification information associated with the frame identification information for the frame, and displays the generated image of the frame within the frame displayed on a display device.

[0011] Each program may be stored on a non-temporary storage medium that is readable by the computer. [Effects of the Invention]

[0012] According to this disclosure, it is possible to improve the sense of realism. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing an example configuration of the control system according to Embodiment 1. [Figure 2] This flowchart shows an example of operation of the control system according to Embodiment 1. [Figure 3] This is an explanatory diagram showing an example of a dome-shaped display used as a display device. [Figure 4] This is an explanatory diagram showing an example of connecting a control system to a dome-shaped display, etc. [Figure 5] This is an explanatory diagram showing an example of an HMD used as a display device. [Figure 6] This is an explanatory diagram showing an example of connecting a control system to an HMD (Head-Mounted Display). [Figure 7] This is an explanatory diagram showing an example of seating arrangements in a mobile vehicle. [Figure 8] This is a block diagram showing an example configuration of a control system according to Embodiment 2. [Figure 9] This is an explanatory diagram showing an example of an imaging device being installed on a moving object. [Figure 10] This is an explanatory diagram illustrating an example of the correspondence between an imaging device in real space and a window in virtual reality space. [Figure 11] This is an explanatory diagram showing an example of an association table that links the imaging device to the window. [Figure 12] This is an explanatory diagram showing an example of extracting an image of a window from captured video footage. [Figure 13] This is an explanatory diagram showing an example where multiple imaging devices are installed on a moving object. [Figure 14] This is an explanatory diagram illustrating an example of the correspondence between two imaging devices in real space and a window in virtual reality space. [Figure 15] This is an explanatory diagram illustrating an example of an association table that links multiple imaging devices to windows. [Figure 16] It is an explanatory diagram showing an example of generating an image of a window from images captured by each of a plurality of imaging devices. [Figure 17] It is an explanatory diagram showing an example of selecting and generating an image of a window from images captured by each of a plurality of imaging devices. [Figure 18] It is an explanatory diagram showing a display example of a dome-shaped display. [Figure 19] It is an explanatory diagram showing an example in which voices of other users are graphically displayed. [Figure 20] It is an explanatory diagram showing an example of switching from graphic output to voice output. [Figure 21] It is a flowchart showing an example of an operation of a control system according to Embodiment 2. [Figure 22] It is an explanatory diagram showing an example of a hardware configuration of a computer.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of a control system, a control method, a program, and a non-temporary recording medium for recording the program according to the present disclosure will be described in detail with reference to the drawings. This embodiment does not limit the disclosed technology.

[0015] Also, virtual reality may be represented as VR (Virtual Reality). In the control system, for example, a virtual reality space is displayed on a display device. The type of the display device is not particularly limited, such as an HMD or a dome-shaped display. Also, the usage scene of the virtual reality space is not particularly limited.

[0016] (Embodiment 1) First, in Embodiment 1, the basic functions of the control system will be described. FIG. 1 is a block diagram showing a configuration example of a control system according to Embodiment 1. For example, the control system 10 includes an acquisition unit 101, a generation unit 102, and an output control unit 103.

[0017] The acquisition unit 101 acquires data that associates imaging device identification information, which identifies an imaging device installed in the real world, with frame identification information, which identifies a frame in the VR space. The type of imaging device is not particularly limited, but a 360-degree camera may be used. As a specific method of acquiring data, the acquisition unit 101 may acquire data from a database that associates frame identification information, which identifies a frame in the VR space that can be seen by the user in the VR space, with imaging device identification information, which identifies an imaging device installed in the real world. Alternatively, the acquisition unit 101 may acquire data via an input device or a communication network. For example, the frame may be the frame of a vehicle or a building in the VR space. Vehicles are not particularly limited, such as buses, airplanes, trains, and ships. Buildings are not particularly limited, such as companies, schools, and houses. Examples of frames include windows and doors. In the following explanation, for the sake of simplicity, associating frame identification information with imaging device identification information may be expressed as associating a frame with an imaging device. Furthermore, for the sake of simplicity, an imaging device identified by imaging device identification information associated with frame identification information that identifies a frame in VR space may be referred to as an imaging device associated with a frame.

[0018] Next, the generation unit 102 generates an image of the frame as seen by the user in the VR space, based on the positional relationship between the user in the VR space and the frame in the VR space, from the image captured by the imaging device associated with the frame. The positional relationship here includes the distance from the frame, the height, and the horizontal position relative to the frame, such as left and right. The specific method of generating the image will be described in detail in Embodiment 2.

[0019] The output control unit 103 displays the generated frame image within the frame in the VR space displayed on the display device.

[0020] (flowchart) Figure 2 is a flowchart showing an example of the operation of the control system 10 according to Embodiment 1. The acquisition unit 101 acquires data that associates an imaging device installed in the real space with a frame in the VR space (step S101).

[0021] The generation unit 102 generates an image of the frame in the VR space from the image captured by the imaging device associated with the frame in the VR space, based on the positional relationship between the user in the VR space and the frame in the VR space (step S102). The output control unit 103 displays the generated image of the frame within the frame in the VR space (step S103).

[0022] In real space, the view seen from frames such as windows and doors of buildings and vehicles varies depending on the user's position. For example, in VR space, the view seen from frames such as windows and doors of buildings and vehicles may be from a fixed viewpoint. In such cases, there is a problem of low realism. In Embodiment 1, the control system 10 generates an image of the frame in the VR space as seen from the user in the VR space, based on the positional relationship between the user in the VR space and the frame in the VR space, from the image captured by the imaging device associated with the frame in the VR space. The control system 10 then displays the generated frame image within the frame in the VR space. This makes it possible to provide an image of the real space as seen from the user's position in the VR space. Therefore, the realism can be improved.

[0023] (Embodiment 2) Next, Embodiment 2 will be described in detail with reference to the drawings. In Embodiment 2, the imaging device is installed on a moving object in real space, travel is given as an example of a VR space usage scenario, and a vehicle window is given as an example of a frame in the VR space. To the extent that the description of Embodiment 2 does not become unclear, explanations that overlap with the above description will be omitted.

[0024] Figure 3 is an explanatory diagram showing an example of a dome-shaped display used as a display device. In Figure 3, a dome-shaped display 21 is given as an example of a display device. In real space, for example, a user is sitting in front of the dome-shaped display 21.

[0025] When the dome-shaped display 21 displays an image, it means that the projection device 2102 projects an image onto the screen 2101.

[0026] An imaging device 22 and a recording device 23 may also be installed at the location where the dome-shaped display 21 is installed. For example, the imaging device 22 and the recording device 23 are used to detect the direction of the user's face, the direction of the user's gaze, the user's movements, and the user's conversations. The control system can detect the user's position and movements from the images captured by the imaging device 22. The control system can detect the user's conversations from the audio obtained by the recording device 23. In addition, although not shown, an audio output device such as a speaker that outputs the voices of other users and the voices of moving objects is installed at the location where the dome-shaped display 21 is installed. Furthermore, a controller 24 may also be installed at the location where the dome-shaped display 21 is installed. The controller 24 is, for example, an example of an input device that receives user operations. For example, in Embodiment 2, the controller 24 receives input of the user's movement due to user operations.

[0027] The number of imaging devices 22 and recording devices 23 is not particularly limited. For example, the imaging device 22 can transmit the captured video to a control system, etc., and the recording device 23 can transmit the audio to a control system, etc.

[0028] The dome-shaped display 21 is a device in which, for example, at least a part of the screen 2101 is curved, and is structured to cover the user's field of view. The dome-shaped display 21 does not have to be a complete dome, such as a 360-degree dome; a portion may be missing, such as a 180-degree dome. In other words, the dome-shaped display 21 may be a hemispherical dome. Furthermore, the dome-shaped display 21 does not have to have a curved portion of the screen 2101. Thus, the size and type of the dome-shaped display 21 are not particularly limited. For example, the dome-shaped display 21 may be a 180-degree dome-shaped display 21 or a 360-degree dome-shaped display 21. Also, for example, the size of the dome-shaped display 21 may be approximately 1 to 2 meters in length, 1 to 2 meters in width, and 1 to 2 meters in height.

[0029] Furthermore, the installation location of the dome-shaped display 21 is not particularly limited. For example, the dome-shaped display 21 may be installed in the user's home, office, or any other location accessible to anyone.

[0030] Figure 4 is an explanatory diagram showing an example of connection between the control system and the dome-shaped display 21, etc. The dome-shaped display 21, imaging device 22, recording device 23, and controller 24 are installed in the user's home or a shared space. The control system 20 is connected to the dome-shaped display 21, imaging device 22, recording device 23, controller 24, etc. via a communication network. For example, an edge terminal device may be installed on the user's side. The edge terminal device may be connected to the dome-shaped display 21, imaging device 22, recording device 23, controller 24, etc. via a communication network, and the control system 20 may be connected to the edge terminal device via a communication network, etc.

[0031] For example, the imaging device 26 in real space may be installed on the mobile body 25. Alternatively, the imaging device 26 may be attached to a robotic arm or the like. The imaging device 26 is connected to the control system 20 via a communication network. The number of imaging devices 26 is not particularly limited.

[0032] Furthermore, the control system 20 may be configured as a whole system comprising a dome-shaped display 21, an imaging device 22, a recording device 23, an imaging device 26, and a controller 24.

[0033] Furthermore, the communication network to which the control system 20 is connected to the user's device and the communication network to which the control system 20 is connected to the mobile device 25 may be the same.

[0034] Figure 5 is an explanatory diagram showing an example of an HMD used as a display device. In Figure 5, the HMD27 is used as an example of a display device. In the real world, the user is wearing the HMD27.

[0035] The HMD27 may have functions to detect, for example, the orientation of the user's face, the direction of the user's gaze, and the user's movements. The HMD27 may also have functions to output sound and collect sound. A controller 24 may be used to allow input of the user's movements.

[0036] Figure 6 is an explanatory diagram showing an example of connection between the control system 20 and the HMD 27, etc. The control system 20 is connected to the HMD 27, controller 24, etc. via a communication network. For example, an edge terminal device may be installed on the user side. The edge terminal device may be connected to the HMD 27, controller 24, etc. via a communication network, and the control system 20 may be connected to the edge terminal device via a communication network, etc.

[0037] For example, as in Figure 4, the imaging device 26 is installed on the mobile body 25. Alternatively, the imaging device 26 may be attached to a robot arm or the like. The imaging device 26 is connected to the control system 20 via a communication network. The number of imaging devices 26 is not particularly limited.

[0038] Furthermore, the control system 20 may be configured as a whole system comprising the HMD 27, the imaging device 26, and the controller 24.

[0039] Figure 7 is an explanatory diagram showing an example of seating in the mobile device 25. When the window w1 in the VR space is the window w1 of a vehicle such as a bus or train, there may be seats in the VR space. For example, in Figure 7, there are seats A through D. Here, seat A is the seat of user X. Seat B is the seat of user Y. Seat C is the seat of user Z.

[0040] Figure 8 is a block diagram showing an example configuration of the control system 20 according to Embodiment 2. For example, the control system 20 includes an acquisition unit 201, a generation unit 202, an output control unit 203, a detection unit 204, a specification unit 205, a reception unit 206, a product determination unit 207, a registration unit 208, and a settlement unit 209.

[0041] The control system 20 is further equipped with a detection unit 204, a identification unit 205, a reception unit 206, a product determination unit 207, a registration unit 208, and a settlement unit 209, in addition to the control system 10 according to Embodiment 1.

[0042] The acquisition unit 201 has the functions of the acquisition unit 101 according to Embodiment 1 as its basic functions. The generation unit 202 has the functions of the generation unit 102 according to Embodiment 1 as its basic functions. The output control unit 203 has the functions of the output control unit 103 according to Embodiment 1 as its basic functions.

[0043] For example, the control system 20 includes an association table 2001, a user database (DB) 2002, and a product database 2003. Each functional unit of the control system 20 can appropriately refer to and update various databases and tables.

[0044] The association table 2001 stores data that associates the imaging device 26 in real space with the windows in VR space. Specifically, the association table 2001 stores, for example, imaging device identification information that identifies the imaging device 26 and window identification information that identifies the windows, in association with each other. The imaging device identification information only needs to be able to identify the imaging device 26, and the format of the imaging device identification information is not particularly limited. The window identification information only needs to be able to identify the windows, and the format of the window identification information is not particularly limited. Note that multiple imaging devices 26 may be associated with a single window. An example of how the association table 2001 is stored will be described later with reference to a diagram, along with an example of how the imaging device 26 is installed.

[0045] Furthermore, the user database 2002 stores user information for each user. User information may include, for example, the user's name and image. The user's image may be an actual photograph of the user or an avatar, and is not particularly limited. The user's image may be used, for example, as the user's image in the VR space. For example, the user database 2002 may store, for each user, a device identification information that identifies the device, in association with the user's information. In addition, separately, the user database 2002 may store, in association with the device identification information, user identification information that identifies the user, and the user's information.

[0046] Furthermore, Product DB2003 stores product information for each product. Product DB2003 stores product identification information and product information in association. Product identification information is not particularly limited as long as it can identify the product. Product information is not particularly limited, for example, product name, product price, product image, product characteristics, etc.

[0047] Next, we will explain the processing of each functional unit by giving a specific example of associating an imaging device 26 in real space with a window in VR space.

[0048] <Example of correspondence between one imaging device 26 and a window> First, we will explain an example of associating a single imaging device 26 in real space with a window in VR space.

[0049] Figure 9 is an explanatory diagram showing an example in which an imaging device 26 is installed on a mobile body 25. For example, in real space, an imaging device 26 is installed on the mobile body 25. In Figure 9, the mobile body 25 is shown as a vehicle such as a truck, but it is not particularly limited to airplanes, trains, drones, ships, etc.

[0050] Figure 10 is an explanatory diagram illustrating an example of the correspondence between an imaging device 26 in real space and a window in VR space. Although the imaging device 26 in real space is not installed in the window w1 in VR space, for ease of understanding, Figure 10 shows the positional relationship between the imaging device 26 in real space and the window w1 in VR space.

[0051] Figure 11 is an explanatory diagram showing an example of an association table 2001 that associates the imaging device 26 with a window. The association table 2001 stores the imaging device ID (Identifier) ​​and the window ID in association. Here, the imaging device ID is an example of imaging device identification information that identifies the imaging device 26. The window ID is an example of window identification information that identifies a window.

[0052] In Figure 11, the imaging device ID "C0001" is associated with the window ID "w1". For example, the imaging device identified by imaging device ID "C0001" is imaging device 26 shown in Figure 9. Imaging device ID "C0002" is associated with the window ID "w2". Imaging device ID "C0003" is associated with the window ID "w3". Imaging device ID "C0004" is associated with the window ID "w4".

[0053] The association table 2001 may store location information, position information, and orientation information, which indicate where the imaging device 26 is installed in real space, in association with the imaging device ID.

[0054] The acquisition unit 201 acquires, for example, an association table 2001 as data that associates an imaging device 26 installed in the real space with a window in the VR space.

[0055] Next, the generation unit 202 generates an image of the window as seen by the user in the VR space, based on the positional relationship between the user in the VR space and the window in the VR space, from the image captured by the imaging device 26 associated with the window.

[0056] If there are multiple users, for example, the generation unit 202 generates an image of the window as seen by the user in the VR space from the image captured by the camera, based on the positional relationship between the user in the VR space and the window in the VR space for each of the multiple users.

[0057] The generation unit 202 generates a new image of the frame visible to the user in the VR space, based on the positional relationship between the user's position in the VR space and the position of the window in the VR space, as well as the user's orientation in the VR space. The user's orientation refers to, for example, the direction of the user's face or head, or the direction of the user's gaze.

[0058] Specifically, as a method for generating the window image, for example, the generation unit 202 may extract the image of the window as seen by the user in the VR space from the captured image, based on the positional relationship between the user in the VR space and the window in the VR space.

[0059] Figure 12 is an explanatory diagram illustrating an example of extracting an image of a window from captured video. In Figure 12, the generation unit 202 extracts an image of the window w1 as seen by user X in the VR space from the captured video, based on the positional relationship between user X in the VR space and window w1 in the VR space, and the direction of the user's gaze.

[0060] Furthermore, in Figure 12, the generation unit 202 generates data for the user in the VR space. Y Based on the positional relationship between the VR window w1 and the user's gaze direction in the VR space, the captured video is used to determine the user's position in the VR space. Y Extract the image of window w1 visible from there.

[0061] Another method for generating window images is, for example, the generation unit 202 generates a new image of the frame visible to the user in the VR space from the image captured by the imaging device 26, based on the positional relationship in the VR space between the user in the VR space and the window w1 in the VR space. Generating a new image may involve adding other images to the captured image. More specifically, for example, the generation unit 202 may generate a new image to the captured image to complement images that are not visible due to weather. The generation unit 202 is not limited to combining, correcting, and complementing past images, but may also combine other images or text with the captured image.

[0062] Furthermore, the methods for generating each video may be combined.

[0063] <Example of correspondence between multiple imaging devices 26 and a window> Next, we will explain an example of associating multiple imaging devices 26 in real space with windows in VR space.

[0064] Figure 13 is an explanatory diagram showing an example in which multiple imaging devices 26 are installed on a mobile body 25. For example, in real space, two imaging devices 26-1 and 26-2 are installed on the mobile body 25 at different positions.

[0065] If the imaging device 26 is a camera other than a 360-degree camera, imaging devices 26-1 and 26-2 may be installed in the same position but in different orientations.

[0066] Figure 14 is an explanatory diagram illustrating an example of the correspondence between two imaging devices 26 in real space and a window in VR space. Although no imaging device 26 is installed in window w1 in VR space, for ease of understanding, Figure 14 shows the positional relationship between the two imaging devices 26-1 and 26-2 in real space and window w1 in VR space.

[0067] Figure 15 is an explanatory diagram showing an example of an association table 2001 that associates multiple imaging devices 26 with windows. The association table 2001 stores imaging device IDs (Identifiers) and window IDs in association. Similar to the example of the association table 2001 in Figure 11, in Figure 15, the imaging device ID is an example of imaging device identification information that identifies the imaging device 26. The window ID is an example of window identification information that identifies a window.

[0068] In Figure 15, in association table 2001, the imaging device ID "C0001" is associated with the window ID "w1", and the imaging device ID "C0002" is associated with the window ID "w1". For example, the imaging device identified by imaging device ID "C0001" is imaging device 26-1 shown in Figure 13, and the imaging device identified by imaging device ID "C0002" is imaging device 26-2 shown in Figure 13.

[0069] Furthermore, the association table 2001 may store location information, place information, and orientation information, indicating where each of the multiple imaging devices 26 is installed in real space, in association with the imaging device ID.

[0070] Next, the acquisition unit 201 acquires an association table 2001 as data that associates multiple imaging devices 26 installed at different locations in real space with a single window in the VR space.

[0071] The generation unit 202 generates an image from images captured by multiple imaging devices 26 associated with a window in the association table 2001, based on the spatial relationship between the user in the VR space and the window in the VR space. The multiple imaging devices 26 associated with a window in the association table 2001 are multiple imaging devices 26 identified by each of the multiple imaging device IDs associated with the window ID in the association table 2001.

[0072] For example, the generation unit 202 generates an image visible to the user in the VR space from multiple images based on the positional relationship in the VR space between the user in the VR space and the window in the VR space. Specifically, as a method for generating an image visible to the user from multiple images, the generation unit 202 may synthesize the multiple images and then extract the image visible to the user in the VR space from the synthesized image. When the window is large, a large number of imaging devices 26 are installed to increase density, allowing the generation unit 202 to synthesize multiple images and generate a more detailed image. Alternatively, the generation unit 202 may extract the image visible to the user in the VR space from each of the multiple images and then synthesize the extracted images to generate the image visible to the user in the VR space. Specifically, for example, the generation unit 202 may determine the coordinate positions to be extracted from each of the multiple images and then synthesize the multiple images based on the determined coordinate positions. When the window is small and the density is not high, and multiple imaging devices are installed, the generation unit 202 can generate a new image of the window visible to the user in the VR space in a short time because it does not need to synthesize the parts of the image that the user does not see.

[0073] Figure 16 is an explanatory diagram illustrating an example of generating a window image from images captured by each of the multiple imaging devices 26. In Figure 16, the generation unit 202 generates an image visible to user X in the VR space from the images captured by the multiple imaging devices 26, based on the positional relationship between user X in the VR space and window w1 in the VR space, and the direction of the user's gaze. In Figure 16, the generation unit 202 generates an image visible to user X in the VR space from the image captured by imaging device 26-1 and the image captured by imaging device 26-2. Note that in Figure 16, an example is given where the areas captured by the multiple images captured by imaging devices 26-1 and 26-2 do not overlap, but there are also cases where the areas captured by the multiple images overlap. In such cases, the generation unit 202 is not particularly limited in which image to use for the overlapping portion.

[0074] More specifically, the generation unit 202 may synthesize multiple videos to generate a single video, and then extract the video visible to user X in the VR space from that single video. Alternatively, the generation unit 202 may extract the video visible to user X in the VR space from each of the multiple videos, and then synthesize the resulting multiple videos.

[0075] Furthermore, the generation unit 202, for user Y, as with user X, generates a user in VR space from multiple images. Y This generates the image visible from that point. This concludes the explanation of the example in Figure 16.

[0076] Alternatively, as another method for generating a user-viewable image from multiple images, the generation unit 202 selects an image from multiple images that includes the image visible to the user in the VR space, based on the positional relationship between the user in the VR space and the window in the VR space. The generation unit 202 may then extract from the selected image the image the user in the VR space sees outside the window in the VR space.

[0077] Figure 17 is an explanatory diagram illustrating an example of selecting and generating a window image from images captured by each of the multiple imaging devices 26. In Figure 17, the generation unit 202 generates an image visible to user X in the VR space from the images captured by the multiple imaging devices 26, based on the positional relationship between user X in the VR space and window w1 in the VR space, and the direction of the user's gaze in the VR space. Specifically, the generation unit 202 selects the image captured by imaging device 26-1 from the images captured by imaging device 26-1 and the images captured by imaging device 26-2, based on the positional relationship between user X in the VR space and window w1 in the VR space, and the direction of the user's gaze in the VR space. Then, the generation unit 202 extracts the image visible to user X in the VR space from the selected image.

[0078] This concludes the explanation of an example of generating an image visible to a user in a VR space from multiple video sources. Note that the generation process described for a single imaging device 26 associated with a window and the generation process described for multiple imaging devices 26 associated with windows may be combined as appropriate.

[0079] Next, the output control unit 203 displays the generated image within a frame in the VR space shown on the display device. The display device is not particularly limited, as mentioned above, but examples include the HMD 27 and the dome-shaped display 21. Here, the dome-shaped display 21 is used as an example of the display device.

[0080] Figure 18 is an explanatory diagram showing an example of what the dome-shaped display 21 can display. For example, the dome-shaped display 21 can display images of the inside of a vehicle in a VR space.

[0081] In Figure 18, more specifically, the dome-shaped display 21 shows user X, user Y, user Z, and window w1 in the VR space. In the VR space, window w1 displays the image generated by the generation unit 202.

[0082] <User movement> Next, we will explain how users move within the VR space. For example, there are two approaches: one in which the user in the VR space is moved in conjunction with the user's movement in the real world, and another in which the user in the VR space is moved by the user's operation of an input device such as a controller 24 in the real world. The choice of which approach to use is determined as appropriate and is not particularly limited. When the display device is an HMD 27, both approaches are likely to be used. For example, when the display device is a dome-shaped display 21, it is highly likely that the user in the VR space will be moved by an operation of an input device such as a controller 24.

[0083] First, let's explain an example of linking the system to the user's movement in the real world.

[0084] The configuration unit associates the user's position and orientation in the real world with the user's position and orientation in the VR space at the start of use. The configuration unit may store this associated information as a table in a storage device or similar.

[0085] Next, the detection unit 204 detects the user's movement in real space. User movement in real space includes facial movements, head movements, and body movements. More specifically, user movement refers to changes in the position of the face, head position, face orientation, head orientation, and gaze direction. If the display device is an HMD 27, the detection unit 204 may detect user movement based on data detected by an acceleration sensor, gyro sensor, geomagnetic sensor, etc., provided in the HMD 27. Alternatively, if the display device is an HMD 27 or a dome-shaped display 21, the detection unit 204 may detect user movement from images captured by the imaging device 22.

[0086] The identification unit 205 identifies the new position and orientation of the user in the VR space based on the detected user movement. For example, the amount of user movement in the real world and the amount of user movement in the VR space are proportional, and the multiplier is not particularly limited. For instance, the amount of user movement in the VR space may be several times greater than the amount of user movement in the real world.

[0087] Then, the generation unit 202 generates a new image of the window as seen by the user in the VR space, based on the new position of the user in the VR space, the positional relationship between the user and the window in the VR space, and the new orientation of the user in the VR space, from the image captured by the imaging device 26 associated with the window.

[0088] Next, we will explain using an example where a user in the real world moves the user in the VR space via an input device. For example, the reception unit 206 detects the user's movement in the VR space by receiving the user's operation on the input device. The input device may be a controller 24, a terminal device, or the like, and is not particularly limited.

[0089] The identification unit 205 identifies the new position and orientation of the user in the VR space based on the user's movement in the VR space that has been received.

[0090] Then, the generation unit 202 generates a new image of the window as seen by the user in the VR space, based on the new position of the user in the VR space, the positional relationship between the user and the window in the VR space, and the new orientation of the user in the VR space, from the image captured by the imaging device 26 associated with the window.

[0091] Then, the generation unit 202 generates a new image of the window as seen by the user in the VR space, based on the new position of the user in the VR space, the positional relationship between the user and the window in the VR space, and the new orientation of the user in the VR space, from the image captured by the imaging device 26 associated with the window.

[0092] <Graphic display of audio> For example, in a VR space, group tours using vehicles may take place. In the case of group tours using vehicles, the users are travelers. In a group tour, some users may know each other, while others may not. Group tours have the advantage of allowing information sharing, such as listening to conversations of users who do not know each other, but they also have the disadvantage of requiring users to listen to conversations they are not interested in.

[0093] Therefore, in Embodiment 2, the control system 20 may be configured to allow the user to selectively hear the audio.

[0094] Specifically, for example, the output control unit 203 displays a graphic on the display device representing the voices of other users other than the user who is conversing with the user. For example, the output control unit 203 may display the graphic along with the voice, or it may mute the voice and display the graphic. Here, for example, the generation unit 202 generates a video by adding a graphic representing the voices of other users to the captured video. Then, the output control unit 203 displays the generated video on the display device.

[0095] Here, "graphics" refer to photographs, illustrations, shapes, symbols, characters, etc. The graphics may also correspond to the audio. Graphics corresponding to audio may differ in shape, size, color, pattern, etc., depending on the audio. For example, the color of the graphics may differ from the color of audio related to travel. For instance, the output control unit 203 may highlight graphics representing audio related to travel more than graphics representing audio unrelated to travel. Similarly, in the case of graphic patterns, the patterns of audio related to travel may differ from the patterns of audio unrelated to travel.

[0096] For example, users other than the user they are conversing with may be pre-grouped, or they may be grouped according to the conversation. Pre-grouping means that before the start of use, users who will be traveling together are registered in advance, and these registered users can be treated as a single group. For example, grouping according to the conversation means that if users X, Y, and Z are conversing, the users included in the same conversation will be grouped together as a single group. In this case, the group will change periodically.

[0097] Figure 19 is an explanatory diagram showing an example of a graphical representation of another user's voice. For example, in Figure 19, a speech bubble is used as the graphic. However, the shape of the graphic is not limited to the shape of a speech bubble; it is not particularly limited as long as it is possible to identify that it represents the voice.

[0098] In Figure 19, there are three speech bubbles. That is, in Figure 19, there are three utterances or three groups of conversations. Here, user Z is a user who is not conversing with user X, and user Z's voice is related to travel. On the other hand, the voices of the other users are not related to travel. For example, the output control unit 203 highlights the graphics representing travel-related voices more than the graphics representing voices that are not related to travel.

[0099] In Figure 19, graphic size and pattern are used for emphasis. In Figure 19, the size of the speech bubble representing user Z's voice is larger than the sizes of the other speech bubbles. Also, the pattern of the speech bubble representing user Z's voice is a dot pattern, while the patterns of the other speech bubbles are solid. In this way, the speech bubble representing user Z's voice is emphasized more than the other speech bubbles. Graphic color may also be used for emphasis.

[0100] Furthermore, the reception unit 206 may accept the selection of a graphic representing voice through user operation. User operation may be, for example, operation via an input device, operation corresponding to the user's hand movements, or operation corresponding to the user's voice, and the method of operation is not particularly limited. The input device may be, as mentioned above, a controller 24 or a terminal device. The user's hand movements can be detected, for example, from an image captured by the imaging device 26. The user's voice can be obtained, for example, by a recording device. The output control unit 203 then outputs the voice represented by the graphic selected by the user. Alternatively, for example, the output control unit 203 may not display the graphic selected by the user. Specifically, for example, the generation unit 202 adds graphics other than the selected graphic to the captured video and generates a video without the selected graphic. The output control unit 203 then displays the generated video on the display device.

[0101] Furthermore, for example, if the display device is a dome-shaped display 21 that does not cover the entire user, the output control unit 203 may output audio of conversations behind the user and display graphics of conversations in front of the user without outputting audio. The front of the user is the direction the user is facing, and the rear of the user is the direction opposite to the direction the user is facing. An example of a dome-shaped display 21 that does not cover the entire user is a 180-degree dome-shaped display 21.

[0102] Figure 20 is an explanatory diagram illustrating an example of switching from graphic output to audio output. For example, in Figure 20, the output control unit 203 displays a speech bubble on the dome-shaped display 21 representing the voice of user Z. The reception unit 206 accepts the selection of a graphic based on the operation of user X. The output control unit 203 outputs the voice of user Z represented by the selected graphic as audio.

[0103] Furthermore, the reception unit 206 may accept the output format for the voices of other users. The output control unit 203 will process the voices of other users in the accepted output format. Specifically, for example, the reception unit 206 may be able to select in stages a graphic representing the voice and the voice itself. The reception unit 206 may also be able to select the graphic in stages. The output control unit 203 then switches the output in stages. For example, the output control unit 203 may switch in stages to display more detailed content of the voice, such as a graphic representing a predetermined shape with color, and a graphic representing the voice with text.

[0104] Please note that group travel is just one example of a usage scenario. It can also be used in other situations, such as when multiple users are present in a building.

[0105] Furthermore, the mobile object 25 may be a vehicle. For example, if the mobile object 25 is a vehicle, it may be installed outside the window of an actual vehicle, and the vehicle in the VR space may be a reproduction of the actual vehicle. Other users displayed in the virtual reality space may be people riding in an actual vehicle. In such a case, the vehicle in the real space may be equipped with an imaging device that takes pictures of the interior of the vehicle in the real space. The output control unit 203 may then display the captured image of the vehicle's interior as the interior of the vehicle in the virtual reality space displayed on the display device. If the imaging device is visible in the interior image, the generation unit 202 may generate a corrected image that removes the imaging device, and the output control unit 203 may then display the generated image as the image of the vehicle in the virtual reality space.

[0106] <Shopping> For example, shopping may take place while traveling in a VR space.

[0107] The product selection unit 207 determines recommended products from the products included in the product database 2003. Here, the products included in the product database 2003 are those identified by the product identification information stored in the product database 2003. Specifically, for example, the product selection unit 207 determines products related to the travel destination as recommended products. Alternatively, the product selection unit 207 may determine recommended products from among the products related to the travel destination based on the user's conversation.

[0108] For example, the output control unit 203 presents information on recommended products to the user. Specifically, the output control unit 203 may display the information on recommended products on a display device, or it may notify the user of the information on recommended products via a terminal device.

[0109] For example, products may be sold in a format similar to in-car or in-flight sales. In addition, the output control unit 203 may display an avatar of a store employee on the display device, and the avatar of the store employee may present information on recommended products.

[0110] The registration unit 208 accepts product registrations based on user operations. The settlement unit 209 settles the registered products based on user operations. Existing technologies may be used for product registration and settlement methods.

[0111] (flowchart) Figure 21 is a flowchart showing an example of the operation of the control system 20 according to Embodiment 2. Here, we will explain an example in which the user's movement in the real world and the user's movement in the VR space are linked.

[0112] First, the acquisition unit 201 acquires data that associates imaging device identification information, which identifies the imaging device 26 installed in the real space, with frame identification information, which identifies the frame in the VR space (step S201). For example, the acquisition unit 201 acquires an association table 2001 as this data.

[0113] The settings unit, as an initial setting, associates the user's position in the VR space with the user's position in the real world (step S202).

[0114] Next, the generation unit 202 generates an image of the frame in the VR space from the image captured by the imaging device 26 associated with the frame in the VR space, based on the positional relationship between the frame in the VR space and the user in the VR space, and the orientation of the user in the VR space (step S203). The output control unit 203 displays the generated image of the frame within the frame in the VR space (step S204).

[0115] The detection unit 204 determines whether it has detected the user's movement in the real world (step S205). If the user's movement in the real world has not been detected (step S205: No), the detection unit 204 returns to step S205.

[0116] If movement of the user in the real world is detected (step S205: Yes), the identification unit 205 identifies the new position and orientation of the user in the VR space (step S206), and returns to step S203. As a result, a new image of the frame visible to the user in the VR space at the new position and orientation of the user in the VR space is generated.

[0117] The flowchart can be terminated as needed.

[0118] In Embodiment 2, the control system 20 extracts the frame visible to the user in the VR space from the image captured by the imaging device 26. This allows the system to provide images outside the frame visible to the user in the VR space. Alternatively, the control system 20 generates a new image of the frame visible to the user in the VR space from the image captured by the imaging device 26, based on the positional relationship between the user in the VR space and the frame in the VR space. This allows the system to provide images outside the frame visible to the user in the VR space. Furthermore, it can complement images that are not present in the actual experience.

[0119] Furthermore, when multiple imaging devices 26 installed at different locations are associated with a frame, the control system 20 generates an image of the frame visible to the user in the VR space from the multiple images. This makes it possible to provide images outside the frame that the user can see in the VR space.

[0120] Furthermore, for each of the multiple users, the control system 20 generates an image of the frame as seen by the user in the VR space from the image captured by the imaging device 26, based on the positional relationship in the VR space between the user in the VR space and the frame in the VR space. This makes it possible to provide images corresponding to the user's position in the VR space, even if there are multiple users in the VR space, using images captured by the same imaging device 26 installed in the real space.

[0121] Furthermore, the control system 20 generates a new image of the frame as seen by the user in the VR space, based on the positional relationship between the user in the VR space and the frame in the VR space, as well as the orientation of the user in the VR space. As a result, the image differs depending on the user's orientation, providing a more realistic image.

[0122] Furthermore, the control system 20 detects the user's movement in the real world and determines the user's position and orientation in the VR space based on the user's movement in the real world. The control system 20 then generates a new image of the frame as seen by the user in the VR space, based on the positional relationship between the user in the VR space and the frame in the VR space at the determined position, as well as the new orientation. As a result, the image changes in response to the user's vertical and horizontal movements in the real world, thereby improving the sense of realism of the image visible outside the frame.

[0123] Furthermore, the control system 20 receives the user's movement in the VR space based on the user's actions in the real world. Based on the received movement, the control system 20 identifies the user's position and orientation in the VR space, and generates a new image of the frame as seen by the user in the VR space, based on the positional relationship between the user and the frame in the VR space at the identified position, and the user's new orientation in the VR space. As a result, the image changes in response to the user's up-and-down movements and left-and-right movements, which improves the sense of realism of the image visible outside the frame.

[0124] Furthermore, the frame can be a window or a door. For example, the window may be a window of a vehicle or a window of a building. This allows the view from a vehicle window in the VR space to be changed according to the user's position in the VR space. Similarly, the view from a building window in the VR space can be changed according to the user's position in the VR space. Likewise, the door may be a window of a vehicle or a door of a building. This can improve the sense of realism of the images visible outside the window or door. In the case of a door, if the door is open, the control system 20 may display the generated image inside the door.

[0125] Furthermore, the imaging device 26 may be installed on a moving object 25 in real space. For example, as the moving object 25 moves, the image captured by the imaging device 26 also changes. For instance, the view from a window of a building or vehicle changes as the moving object 25 moves. This can provide a travel experience, for example. Also, the moving object 25 may be moving in real time. This can provide real-time video.

[0126] Furthermore, the control system 20 displays the voices of other users, excluding the user currently conversing with the user, as graphics. Users want to enjoy realistic visuals, or, in the case of travel, enjoy group travel on a vehicle. However, users may or may not want to listen to the conversations of other passengers. For example, they may want to listen if the information is useful to them. This allows users to avoid being bothered by other users' conversations by simply not looking at the graphics. On the other hand, users can look at the graphics if they are interested. In this way, the control system 20 can provide a better visual experience by combining the advantages of real-world travel with the advantages of VR space.

[0127] Furthermore, the control system 20 outputs audio representing the sound of the graphic selected by the user from among the displayed graphics. For example, the user can listen to audio to confirm a conversation that interests them.

[0128] The control system 20 does not display graphics selected by the user. This allows the user to remove graphics representing conversations they are not interested in from their field of vision.

[0129] Furthermore, if the display device is a hemispherical dome-shaped display 21, the control system 20 may output audio of conversations behind the user in the VR space and graphically display conversations in front of the user in the VR space.

[0130] The control system 20 can accept output formats for other users' voices and outputs them in the accepted format. For example, the control system 20 may allow users to select output formats in stages, such as color, text, and sound. This allows the output format of other users' conversations to be changed according to the user's interests.

[0131] This concludes the description of each embodiment. The embodiments may be used in combination as appropriate.

[0132] Furthermore, in each embodiment, the control system 20 may be configured to include some of the functional units and information. In Embodiment 2, the control system 20 was described as having some of the functions of a POS (Point of Sale) system, such as a registration unit 208 and a settlement unit 209. The control system 20 may be configured to be connected to a POS system equipped with a registration unit 208 and a settlement unit 209 via a communication network.

[0133] Furthermore, each embodiment is not limited to the examples described above and can be modified in various ways. Also, the configuration of the control system 20 in each embodiment is not particularly limited. For example, the control system 20 may be implemented by a single device, such as a single server. When each functional part of the control system 20 is implemented by a single device, for example, the single device may be called a control device, an information processing device, etc., and is not particularly limited. Alternatively, the control system 20 in each embodiment may be implemented by different devices for each function or data. For example, each functional part may be composed of multiple servers and implemented as the control system 20. For example, the control system 20 may be implemented by a database server including each DB and a server having each functional part.

[0134] Furthermore, in each embodiment, each piece of information and each database may include a portion of the aforementioned information. Also, each piece of information and each database may include information other than the aforementioned information. Each piece of information and each database may be further divided into multiple databases or multiple pieces of information. Thus, the method of implementing each piece of information and each database is not particularly limited.

[0135] Furthermore, each screen is merely an example and is not particularly limited. Buttons, lists, checkboxes, information display fields, input fields, etc., not shown in the illustrations may be added to each screen. Also, the background color of the screen may be changed.

[0136] Furthermore, the process of generating information to be displayed on the display device may be performed by the output control units 103 and 203. Alternatively, this process may be performed by the display device itself.

[0137] (Example of computer hardware configuration) Next, we will describe an example of a hardware configuration when each device, such as the control systems 10 and 20 described in each embodiment, is implemented using a computer. Figure 22 is an explanatory diagram showing an example of a computer hardware configuration. For example, some or all of each device can be implemented using any combination of computer 80 and program as shown in Figure 22.

[0138] Computer 80 includes, for example, a processor 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, and a storage device 804. Computer 80 also includes a communication interface 805 and an input / output interface 806. Each component is connected, for example, via a bus 807. The number of components is not particularly limited, and each component may be one or more.

[0139] The processor 801 controls the entire computer 80. The processor 801 may include, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or a GPU (Graphics Processing Unit). The computer 80 has memory units such as ROM 802, RAM 803, and a storage device 804. The storage device 804 may include, for example, semiconductor memory such as flash memory, an HDD (Hard Disk Drive), or an SSD (Solid State Drive). For example, the storage device 804 stores OS (Operating System) programs, application programs, and programs related to each embodiment. Alternatively, ROM 802 stores application programs and programs related to each embodiment. The RAM 803 is used as the work area for the processor 801.

[0140] The processor 801 also loads programs stored in the memory device 804, ROM 802, etc. Then, the processor 801 executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. Furthermore, the processor 801 functions as part or all of the computer 80. The processor 801 may also execute processes or instructions in the illustrated flowchart based on the program.

[0141] The communication interface 805 is connected to a communication network NT, such as a LAN (Local Area Network) or WAN (Wide Area Network), via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. This allows the computer 80 to connect to external devices and external computers 80 via the communication network NT. The communication interface 805 manages the interface between the communication network NT and the internal workings of the computer 80. Furthermore, the communication interface 805 controls the input and output of data from external devices and external computers 80.

[0142] Furthermore, the input / output interface 806 is connected to at least one of the input device, output device, and input / output device. The connection method may be wireless or wired. Examples of input devices include keyboards, mice, and microphones. Examples of output devices include display devices, lighting devices, and audio output devices. Examples of input / output devices include touch panel displays. The input device, output device, and input / output device may be built into the computer 80 or may be external.

[0143] The hardware configuration of computer 80 is an example. Computer 80 may have some of the components shown in Figure 22. Computer 80 may have components other than those shown in Figure 22. For example, computer 80 may have a drive device. The processor 801 may read programs and data stored on a recording medium attached to the drive device into RAM 803. Examples of non-temporary tangible recording media include optical discs, flexible discs, magneto-optical discs, and USB (Universal Serial Bus) memory. Also, as mentioned above, computer 80 may have input devices such as a keyboard and a mouse. Computer 80 may have output devices such as a display. Furthermore, computer 80 may have input devices, output devices, and input / output devices, respectively.

[0144] Furthermore, the computer 80 may have various sensors (not shown). The type of sensor is not particularly limited. Also, the computer 80 may be equipped with an imaging device capable of capturing images or videos.

[0145] This concludes the description of the hardware configuration of each device. Furthermore, there are various variations in how each device can be implemented. For example, each device may be implemented by any combination of different computers and programs for each component. Alternatively, the multiple components of each device may be implemented by any combination of a single computer and program.

[0146] Furthermore, some or all of the components of each device may be implemented by application-specific circuits. Alternatively, some or all of the components of each device may be implemented by general-purpose circuits, including processors such as FPGAs (Field Programmable Gate Arrays). Furthermore, some or all of the components of each device may be implemented by a combination of application-specific circuits and general-purpose circuits. These circuits may also be a single integrated circuit, or they may be divided into multiple integrated circuits. These multiple integrated circuits may be connected via a bus or the like.

[0147] Furthermore, if some or all of the components of each device are implemented by multiple computers or circuits, these computers or circuits may be centrally located or distributed.

[0148] The control methods described in each embodiment are implemented by the control systems 10 and 20. Alternatively, the control method can be implemented by a computer, such as a server or terminal device, executing a pre-prepared program.

[0149] The programs described in each embodiment are recorded on a computer-readable recording medium such as an HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. The programs are then executed by being read from the recording medium by a computer. The programs may also be distributed via a communication network NT.

[0150] Each component of the control systems 10 and 20 in the embodiments described above may be implemented using dedicated hardware, such as a computer. Alternatively, each component may be implemented using software. Alternatively, each component may be implemented using a combination of hardware and software.

[0151] The present disclosure has been described above with reference to the embodiments described herein, but the present disclosure is not limited to the embodiments described above. The structure and details of each present disclosure may include embodiments that apply various modifications that can be grasped by those skilled in the art within the scope of the present disclosure. The present disclosure may include embodiments that combine or substitute the matters described herein as appropriate. For example, matters described using a particular embodiment may be applied to other embodiments to the extent that they do not cause a contradiction. For example, although multiple operations are described sequentially in the form of a flowchart, the order in which they are described does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations may be changed to the extent that it does not impair the content.

[0152] Some or all of the above embodiments may also be described as follows. However, some or all of the above embodiments are not limited to the following.

[0153] (Note 1) An acquisition means for acquiring data that associates imaging device identification information, which identifies an imaging device installed in real space, with frame identification information, which identifies a frame in virtual reality space. A generation means for generating an image of the frame as seen by the user in the virtual reality space, based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space, from an image captured by an imaging device identified by imaging device identification information associated with frame identification information that identifies the frame, An output control means for displaying the generated image of the frame within the frame shown on the display device, A control system equipped with the following features. (Note 2) The generation means extracts an image of the frame as seen by the user in the virtual reality space from the image captured by the imaging device, based on the positional relationship between the user and the frame in the virtual reality space. The control system described in Appendix 1. (Note 3) The generation means generates a new image of the frame as seen by the user in the virtual reality space, based on the positional relationship between the user and the frame in the virtual reality space, from the image captured by the imaging device. The control system described in Appendix 1. (Note 4) The acquisition means acquires data that associates imaging device identification information, which identifies each of a plurality of imaging devices installed at different locations, with frame identification information, which identifies the frame. The generation means generates an image of the frame as seen by the user in the virtual reality space from a plurality of images captured by a plurality of imaging devices identified by a plurality of imaging device identification information associated with the frame identification information that identifies the frame, based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space. A control system as described in any of the appendices 1 to 3. (Note 5) The generation means generates, for each of the multiple users, an image of the frame as seen by the user in the virtual reality space, from the image captured by the imaging device, based on the positional relationship between the user and the frame in the virtual reality space. A control system as described in any of the appendices 1 to 4. (Note 6) The generation means generates a new image based on the positional relationship between the user and the frame in the virtual reality space, and the orientation of the user in the virtual reality space. A control system as described in any of the appendices 1 to 5. (Note 7) A setting means for associating the user's position and orientation in the real space with the user's position and orientation in the virtual reality space, A detection means for detecting the movement of the user in the aforementioned real space, A means for identifying the position and orientation of the user in the virtual reality space based on the user's movement in the real space, Equipped with, The generation means generates a new image based on the positional relationship between the user and the frame in the virtual reality space at the specified location, and the specified orientation. A control system as described in any of the appendices 1 to 6. (Note 8) A reception means that accepts the user's movement in the virtual reality space based on the user's actions in the real space, Based on the received movement, a means for identifying the position and orientation of the user in the virtual reality space, Equipped with, The generation means generates a new image based on the positional relationship between the user in the virtual reality space and the frame at the specified location in the virtual reality space, and the specified orientation. A control system as described in any of the appendices 1 to 7. (Note 9) The aforementioned frame is a window or a door. A control system as described in any of the appendices 1 to 8. (Note 10) The aforementioned window is a window of a vehicle or a window of a building. The aforementioned door is a vehicle door or a building door. The control system described in Appendix 9. (Note 11) The imaging device is installed on the moving object in the real space. The control system described in Appendix 10. (Note 12) The output control means displays graphics representing the voices of other users in the virtual reality space who are not conversing with the user in question. A control system as described in any of the appendices 1 to 11. (Note 13) Among the aforementioned graphics, a receiving means for accepting the selection of a graphic by the user in the real space, Equipped with, The output control means causes the selected graphic to output as sound. The control system described in Appendix 12. (Note 14) Among the aforementioned graphics, a receiving means for accepting the selection of a graphic by the user in the real space, Equipped with, The output control means prevents the display of the selected graphic. The control system described in Appendix 12. (Note 15) When the display device is a hemispherical dome-shaped display, The output control means outputs audio of conversations behind the user in the virtual reality space and displays graphics of conversations in front of the user in the virtual reality space. A control system as described in any of the appendices 12 to 14. (Note 16) A receiving means for receiving the output format of the voice of the aforementioned other user, Equipped with, The output control means outputs the voice of the other user in the received output format. A control system as described in any of Appendix 12 to 14. (Note 17) By acquiring data that associates imaging device identification information, which identifies an imaging device installed in the real world, with frame identification information, which identifies a frame in the virtual reality space, Based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space, an image of the frame as seen by the user in the virtual reality space is generated from the image captured by the imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame. The generated image of the frame is displayed within the frame shown on the display device. Control method. (Note 18) On the computer, By acquiring data that associates imaging device identification information, which identifies an imaging device installed in the real world, with frame identification information, which identifies a frame in the virtual reality space, Based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space, an image of the frame as seen by the user in the virtual reality space is generated from the image captured by the imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame. The generated image of the frame is displayed within the frame shown on the display device. A non-temporary recording medium readable by the computer, which records a program that executes a process. (Note 19) On the computer, By acquiring data that associates imaging device identification information, which identifies an imaging device installed in the real world, with frame identification information, which identifies a frame in the virtual reality space, Based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space, an image of the frame as seen by the user in the virtual reality space is generated from the image captured by the imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame. The generated image of the frame is displayed within the frame shown on the display device. Execute the process program . [Explanation of Symbols]

[0154] 10,20 Control Systems 21 Dome-shaped displays 22 Imaging device 23 Recording device 24 controllers 25 Mobile Unit 26, 26-1, 26-2 Imaging device 80 Computers 101,201 Acquisition Department 102,202 Generation part 103,203 Output control unit 204 Detection Unit 205 Specific section 206 Reception Department 207 Product Determination Department 208 Registration Department 209 Settlement Department 801 Processor 802 ROM 803 RAM 804 Storage device 805 Communication Interface 806 Input / Output Interface 807 Bus 2101 Screen 2102 Projection device 2001 Related Tables 2002 User DB 2003 Product DB NT communication network w1 window X User Y User Z user

Claims

1. An acquisition means for acquiring data that associates imaging device identification information, which identifies an imaging device installed in real space, with frame identification information, which identifies a frame in virtual reality space. A generation means for generating an image of the frame as seen by the user in the virtual reality space, based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space, from an image captured by an imaging device identified by imaging device identification information associated with frame identification information that identifies the frame, An output control means for displaying the generated image of the frame within the frame shown on the display device, A control system equipped with the following features.

2. The generation means extracts an image of the frame as seen by the user in the virtual reality space from the image captured by the imaging device, based on the positional relationship between the user and the frame in the virtual reality space. The control system according to claim 1.

3. The generation means generates a new image of the frame as seen by the user in the virtual reality space, based on the positional relationship between the user and the frame in the virtual reality space, from the image captured by the imaging device. The control system according to claim 1.

4. The acquisition means acquires data that associates imaging device identification information, which identifies each of a plurality of imaging devices installed at different locations, with frame identification information, which identifies the frame. The generation means generates an image of the frame as seen by the user in the virtual reality space from a plurality of images captured by a plurality of imaging devices identified by a plurality of imaging device identification information associated with the frame identification information that identifies the frame, based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space. The control system according to claim 1.

5. The generation means generates an image of the frame as seen by the user in the virtual reality space, based on the positional relationship between the user's position in the virtual reality space and the frame in the virtual reality space, for each of the multiple users, from the image captured by the imaging device. The control system according to claim 1.

6. The generation means generates a new image based on the positional relationship between the user and the frame in the virtual reality space, and the orientation of the user in the virtual reality space. The control system according to claim 1.

7. A setting means for associating the user's position and orientation in the real space with the user's position and orientation in the virtual reality space, A detection means for detecting the movement of the user in the aforementioned real space, A means for identifying the position and orientation of the user in the virtual reality space based on the user's movement in the real space, Equipped with, The generation means generates a new image based on the positional relationship between the user and the frame in the virtual reality space at the specified location, and the specified orientation. The control system according to claim 1.

8. A reception means that accepts the user's movement in the virtual reality space based on the user's actions in the real space, Based on the received movement, a means for identifying the position and orientation of the user in the virtual reality space, Equipped with, The generation means generates a new image based on the positional relationship between the user in the virtual reality space and the frame at the specified location in the virtual reality space, and the specified orientation. A control system according to any one of claims 1 to 7.

9. A computer, By acquiring data that associates imaging device identification information, which identifies an imaging device installed in the real world, with frame identification information, which identifies a frame in the virtual reality space, Based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space, an image of the frame as seen by the user in the virtual reality space is generated from the image captured by the imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame. The generated image of the frame is displayed within the frame shown on the display device. A control method for executing a process.

10. On the computer, By acquiring data that associates imaging device identification information, which identifies an imaging device installed in the real world, with frame identification information, which identifies a frame in the virtual reality space, Based on the positional relationship between the user in the virtual reality space and the frame in the virtual reality space, an image of the frame as seen by the user in the virtual reality space is generated from the image captured by the imaging device identified by the imaging device identification information associated with the frame identification information that identifies the frame. The generated image of the frame is displayed within the frame shown on the display device. A program that executes a process.

Images