Control system, control method, and program

Abstract

This control system comprises an acquisition unit, a detection unit, an imaging device control unit, and an output control unit. The acquisition unit acquires data in which imaging device identification information on an imaging device attached to a mobile body is associated with display device identification information on a display device of a user. The detection unit detects an orientation which the user wishes to view. The imaging device control unit controls the orientation of the imaging device on the basis of the detected orientation and the position of the user in a virtual reality space. The output control unit displays a video captured by the imaging device on the display device identified by the display device identification information associated with the imaging device identification information on the imaging device.

Description

[Technical Field]

[0001] The present disclosure relates to control systems and the like. [Background technology]

[0002] For example, a user may experience travel by watching a video of a tourist spot. In an actual trip, the user may travel by vehicle.

[0003] Patent Document 1 describes displaying an image on an HMD (Head Mounted Display). Specifically, for example, Patent Document 1 describes that the image displayed on the HMD includes a virtual reality space or an augmented reality space in which the interior space and the scenery seen from the tourist vehicle are combined, and a character, and that the scenery seen from the tourist vehicle is combined with the window of the moving part. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-058528 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, when a user travels and views videos of tourist spots, the videos may have been recorded in advance. In such cases, unlike actual travel, it may be difficult for users to view the videos they want to see in real time. As such, there is a problem in that it is difficult to provide highly realistic videos when viewing videos remotely.

[0006] An example of an objective of the present disclosure is to provide a control system or the like that improves the sense of realism when viewing video remotely. [Means for solving the problem]

[0007] A control system according to one aspect of the present disclosure includes an acquisition means for acquiring data associated with imaging device identification information of an imaging device attached to a moving body and display device identification information of a user's display device, a detection means for detecting the direction in which the user wishes to view, an imaging device control means for controlling the orientation of the imaging device based on the detected direction and the position of the user in a virtual reality space, and an output control means for displaying an image captured by the imaging device on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device.

[0008] A control method in one aspect of the present disclosure acquires data that associates imaging device identification information of an imaging device attached to a moving body with display device identification information of a user's display device, detects the direction in which the user wants to view, and, based on the detected direction and the user's position in a virtual reality space, displays the image captured by the imaging device on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device.

[0009] A program in one aspect of the present disclosure causes a computer to execute a process of acquiring data that associates imaging device identification information of an imaging device attached to a moving object with display device identification information of a user's display device, detecting a direction in which the user wants to view, and displaying an image captured by the imaging device on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device, based on the detected direction and the user's position in a virtual reality space.

[0010] Each program may be stored in a non-transitory computer-readable recording medium. [Effects of the Invention]

[0011] According to the present disclosure, it is possible to improve the sense of reality when viewing video remotely. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing a configuration example of a control system according to a first embodiment; [Figure 2] 4 is a flowchart showing an example of an operation of the control system according to the first embodiment. [Figure 3] FIG. 1 is an explanatory diagram showing Example 1 of a display device in a control system. [Figure 4] FIG. 10 is an explanatory diagram showing a second example of a display device in the control system. [Figure 5] FIG. 1 is an explanatory diagram showing an example of a seat in a moving body. [Figure 6] FIG. 1 is an explanatory diagram showing an example in which an imaging device is installed on a moving body. [Figure 7] FIG. 10 is an explanatory diagram showing an example in which the imaging device is attached to a robot arm. [Figure 8] FIG. 10 is a block diagram showing a configuration example of a control system according to a second embodiment. [Figure 9] FIG. 10 is an explanatory diagram illustrating an example of an association table. [Figure 10] FIG. 1 is an explanatory diagram (part 1) showing the correspondence between the direction of a user's face in real space and the direction of a user's face in virtual reality space. [Figure 11] FIG. 2 is an explanatory diagram (part 2) showing the correspondence between the direction of the user's face in the real space and the direction of the user's face in the virtual reality space. [Figure 12] FIG. 10 is an explanatory diagram showing a display example of a dome-shaped display. [Figure 13] FIG. 10 is an explanatory diagram showing an example in which the voices of other users are graphically displayed. [Figure 14] FIG. 10 is an explanatory diagram showing an example of switching from graphic output to audio output. [Figure 15] 10 is a flowchart showing an example of an operation of the control system according to the second embodiment. [Figure 16] FIG. 1 is an explanatory diagram illustrating an example of a control system. [Figure 17] FIG. 2 is an explanatory diagram illustrating an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, with reference to the drawings, embodiments of a control system, a control method, a program, and a non-transitory recording medium for recording a program according to the present disclosure will be described in detail. The disclosed technology is not limited to these embodiments.

[0014] In addition, although each embodiment will be described taking travel as an example of a use case, the use case is not limited to travel. Virtual reality may also be referred to as VR (Virtual Reality).

[0015] (Embodiment 1) First, in the first embodiment, basic functions of a control system will be described. Fig. 1 is a block diagram showing an example of the configuration of a control system according to the first embodiment. For example, a control system 10 includes an acquisition unit 101, a detection unit 102, an imaging device control unit 103, and an output control unit 104.

[0016] The acquisition unit 101 acquires data in which the imaging device identification information of the imaging device attached to the moving body is associated with the display device identification information of the user's display device.

[0017] Here, the moving body is not particularly limited to vehicles such as buses, trains, airplanes, boats, drones, etc. The imaging device identification information is not particularly limited as long as it can uniquely identify the imaging device. The display device identification information is not particularly limited as long as it can uniquely identify the user's display device. For example, if the user and the display device are in a paired relationship, user identification information may be used instead of the display device identification information. The display device is not particularly limited to, for example, a general display, an HMD, a dome-shaped display, etc. However, if an HMD, a dome-shaped display, etc. is used, the sense of immersion can be improved.

[0018] The detection unit 102 detects the direction in which the user wants to look. For example, the direction in which the user wants to look may be the direction of the user's face, or may be a direction specified by a controller or the like.

[0019] First, an example will be described in which the direction the user wants to look is the direction of the user's face. For example, the detection unit 102 detects the direction of the user's face when using a display device identified by display device identification information. As a specific method for detecting the direction of the face, for example, when the display device is an HMD, the detection unit 102 may detect the direction of the face based on data detected by an acceleration sensor, a gyro sensor, a geomagnetic sensor, or the like provided in the HMD. Alternatively, the detection unit 102 may detect the direction of the user's face from an image captured by an imaging device different from the imaging device provided in the mobile object. Furthermore, the detection unit 102 may detect the direction of the user's line of sight in more detail.

[0020] Next, an example will be described in which the direction the user wants to look is the direction specified via an input device. For example, the detection unit 102 detects the direction the user wants to look in the virtual reality space by receiving the user's direction in the virtual reality space through the user's operation on the input device. Note that the input device may be a controller, a terminal device, or the like, and is not particularly limited.

[0021] The image capture device control unit 103 controls the orientation of the image capture device installed on the moving body based on the direction in which the user wants to look and the user's position in the virtual reality space.

[0022] Then, the output control unit 104 displays the video captured by the imaging device on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device.

[0023] (flowchart) 2 is a flowchart showing an example of an operation of the control system 10 according to the first embodiment. The acquiring unit 101 acquires data in which the imaging device identification information of the imaging device and the display device identification information of the display device are associated with each other (step S101). For example, the data in which the imaging device identification information of the imaging device and the display device identification information of the display device are associated with each other may be stored in a storage unit or the like. Then, the acquiring unit 101 may acquire the data from the storage unit.

[0024] The detection unit 102 detects the direction in which the user wants to view (step S102). The imaging device control unit 103 controls the orientation of the imaging device based on the detected orientation and the user's position in the virtual reality space (step S103). The output control unit 104 displays the video captured by the imaging device on a display device identified by display device identification information associated with the imaging device identification information of the imaging device (step S104).

[0025] As described above, in the first embodiment, the control system 10 controls the orientation of the imaging device installed on the moving object based on the direction the user wants to view, and displays the captured image on the user's display device. This improves the sense of reality when viewing images remotely.

[0026] (Embodiment 2) Next, the second embodiment will be described in detail with reference to the drawings. In the second embodiment, a specific use case of travel will be described using travel as an example. In the second embodiment, the description of the second embodiment will be omitted for the content that overlaps with the above description, as long as the description of the second embodiment is not unclear.

[0027] Fig. 3 is an explanatory diagram showing Example 1 of a display device in a control system. In Fig. 3, a dome-shaped display 21 is taken as an example of the display device. In real space, for example, a user is sitting in front of the dome-shaped display 21.

[0028] When the dome display 21 displays an image, the projection device 2102 projects the image onto the screen 2101 .

[0029] An imaging device 22 and a sound recording device 23 may be installed where the dome-shaped display 21 is installed. For example, the imaging device 22 and the sound recording device 23 are used to detect the direction of the user's face, the direction of the user's gaze, the user's movements, and the user's conversation. The control system can detect the direction of the user's face, the direction of the user's gaze, and the user's movements from the video captured by the imaging device 22. The control system can detect the user's conversation from the audio obtained by the sound recording device 23. Although not shown, an audio output device such as a speaker that outputs the voices of other users and the voices of a moving object is also installed where the dome-shaped display 21 is installed. A controller 28 may also be installed where the dome-shaped display 21 is installed. The controller 28 is, for example, a device that accepts user operations. The controller 28 can also accept input of the direction the user wants to view.

[0030] There is no particular limitation on the number of imaging devices 22 and recording devices 23. For example, the imaging devices 22 may transmit captured images to a control system or the like, and the recording devices 23 may transmit audio to a control system or the like.

[0031] The dome-shaped display 21 is, for example, a device in which at least a portion of the screen 2101 is curved, so as to cover the user's field of vision. Note that the dome-shaped display 21 does not have to be a complete dome shape, such as a 360-degree dome, but may have a missing portion, such as a 180-degree dome. That is, the dome-shaped display 21 may have a hemispherical dome shape. Furthermore, the dome-shaped display 21 does not require that a portion of the screen 2101 be curved. Thus, the size and type of the dome-shaped display 21 are not particularly limited. For example, the dome-shaped display 21 may be a 180-degree dome-shaped display 21 or a 360-degree dome-shaped display 21. Furthermore, for example, the size of the dome-shaped display 21 may be approximately 1 to 2 meters long, 1 to 2 meters wide, and 1 to 2 meters high.

[0032] Furthermore, there is no particular limitation on the installation location of the dome-shaped display 21. For example, the dome-shaped display 21 may be installed in the user's home, office, etc., or in a location that anyone can use.

[0033] Fig. 4 is an explanatory diagram showing Example 2 of a display device in a control system. In Fig. 4, an HMD 24 is taken as an example of the display device. In the real space, a user wears the HMD 24.

[0034] The HMD 24 has a function to detect, for example, the direction of the user's face, the direction of the user's gaze, and the user's movement. The HMD 24 may also have an audio output function and an audio collection function.

[0035] FIG. 5 is an explanatory diagram showing an example of seats in a moving body. When the moving body is a vehicle such as a bus or train, there may be seats. For example, in FIG. 5, there are seats A to D. Here, seat A is the seat for user X, and seat B is the seat for user Y.

[0036] Fig. 6 is an explanatory diagram showing an example in which an imaging device is installed in a moving body. In Fig. 6, moving body 25 is a vehicle such as a bus or a train, and imaging device 26 is installed in one of a number of seats. In Fig. 6, imaging device 26 is installed in seat A and seat D for user X.

[0037] FIG. 7 is an explanatory diagram showing an example in which the imaging device 26 is attached to the robot arm 27. For example, in FIG. 7, a plurality of robot arms 27 (e.g., robot arms 27a, 27b, etc.) are attached to the moving body 25. An imaging device 26 (imaging device 26a, 26b, etc.) is attached to each robot arm 27. As a result, the control system can control the imaging position of the imaging device 26 by controlling the robot arm 27. Note that the control system is not limited to the example in which the imaging device 26 is attached to the robot arm 27, as long as it can capture an image in a direction corresponding to the direction of the user's face or the user's line of sight, or at a position corresponding to the user's movement.

[0038] 5 and 6, as use cases of the control system, moving body 25 on which imaging device 26 is installed is exemplified as a vehicle such as a bus or a train, but moving body 25 may be one that is not intended to carry a user. That is, imaging device 26 in moving body 25 may capture a landscape, and an image of the captured landscape may be synthesized with an image of the vehicle in a virtual reality space, and the synthesized image may be displayed on a display device.

[0039] 8 is a block diagram showing an example of the configuration of a control system according to embodiment 2. For example, the control system 20 includes an acquisition unit 201, a detection unit 202, an imaging device control unit 203, an output control unit 204, a video generation unit 205, a reception unit 206, a product determination unit 207, a registration unit 208, and a settlement unit 209.

[0040] The control system 20 further includes a video generation unit 205, a reception unit 206, a product determination unit 207, a registration unit 208, and a checkout unit 209 in addition to the control system 10 according to the first embodiment.

[0041] The acquiring unit 201 has the function of the acquiring unit 101 according to the first embodiment as a basic function. The detecting unit 202 has the function of the detecting unit 102 according to the first embodiment as a basic function. The imaging device control unit 203 has the function of the imaging device control unit 103 according to the first embodiment as a basic function. The output control unit 204 has the function of the output control unit 104 according to the first embodiment as a basic function.

[0042] For example, the control system 20 has an association table 2001, a user DB 2002, and a product DB 2003. Each functional unit of the control system 20 can refer to and update various databases and tables as appropriate.

[0043] 9 is an explanatory diagram showing an example of the association table 2001. The association table 2001 stores a camera ID (Identifier), a seat, and a display ID in association with each other. Here, the camera ID is an example of image capture device identification information that identifies the image capture device 26. The seat indicates a seat. The display ID is an example of display device identification information that identifies a display device.

[0044] 9, the camera ID "C0001", seat A, and display ID "D001" are associated with each other. In FIG. 9, the camera ID "C0002", seat D, and display ID "D002" are associated with each other.

[0045] Furthermore, the user DB 2002 stores user information for each user. Examples of user information include the user's name and an image of the user. The image of the user may be a photograph of the actual user or an avatar, and is not particularly limited. For example, the user DB 2002 may store display device identification information and user information in association with each other for each user. Furthermore, the user DB 2002 may separately store display device identification information, user identification information, and user information in association with each other.

[0046] Furthermore, the product DB 2003 stores product information for each product. The product DB 2003 stores product identification information and product information in association with each other. The product identification information is not particularly limited as long as it can identify the product. The product information may include, for example, the product name, product price, product image, product features, etc.

[0047] Next, the functional blocks in FIG. 8 will be described.

[0048] The acquisition unit 201 acquires an association table 2001 as data in which, for example, the imaging device identification information of the imaging device 26 attached to the moving object 25 and the display device identification information of the user's display device are associated with each other.

[0049] Next, the detection unit 202 detects the direction in which the user wants to look. The method for detecting the direction in which the user wants to look is as described in the first embodiment.

[0050] The imaging device control unit 203 controls the orientation of the imaging device 26 installed on the moving body 25 based on the direction the user wants to see and the user's position in the virtual reality space. As a specific method for controlling the orientation of the imaging device 26, for example, the imaging device control unit 203 may control the orientation of the imaging device 26 by controlling a robot arm 27 to which the imaging device 26 is attached. Existing technology may be used as a method for controlling the robot arm 27.

[0051] Furthermore, when the moving body 25 is a vehicle, the position of the user in the virtual reality space may be determined by the position and orientation of the user's seat in the vehicle in the virtual reality space. Therefore, when the moving body 25 is a vehicle, the imaging device control unit 203 controls the orientation of the imaging device 26 according to the orientation of the user's seat in the vehicle in the virtual reality space. As a result, the user can view the moving body 25 according to the seat. Inside You can see the scenery in

[0052] Then, the output control unit 204 displays the video captured by the imaging device 26 on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device 26.

[0053] For example, if moving body 25 to which imaging device 26 is attached is not a vehicle, output control unit 204 may cause the display device to display an image obtained by combining the captured image with an image of the vehicle in the virtual reality space. Specifically, image generation unit 205 generates an image obtained by combining the captured image with an image of the vehicle in the virtual reality space. Then, output control unit 204 causes the display device to display the generated image.

[0054] Furthermore, for example, if the moving body 25 to which the imaging device 26 is attached is a vehicle and another imaging device 26 attached to the moving body 25 is captured in the image, the image generation unit 205 generates an image from the image captured by the imaging device 26, in which the other imaging device 26 that was captured in the image has been deleted. The deleted image may be, for example, an image in which no one is sitting in the seat where the other imaging device 26 is installed, or an image in which the seat where the other imaging device 26 is installed has been replaced with a predetermined image. The predetermined image may be an image of a user using a display device identified by display device identification information associated with the imaging device identification information of the other imaging device 26. There may be cases in which an imaging device 26 is installed but no user is present. Therefore, the predetermined image may be an image of any user, as if the user were actually sitting in the moving body 25, and is not particularly limited.

[0055] 10 and 11 are explanatory diagrams showing the correspondence relationship between the orientation of the user's face in real space and the orientation of the user's face in virtual reality space. Here, the orientation of the user's face in real space will be used as the orientation the user wants to see. The orientation of the user's face in real space and the orientation of the user's face in virtual reality space are associated at the start of use. In FIG. 10, the orientation of the user's face in real space is facing north. For example, if the imaging device 26 is a 360-degree camera, the output control unit 204 displays an image captured by the imaging device 26 that faces north. An image that appears to face north, like the orientation of the user's face in the virtual reality space, is displayed on the display device.

[0056] 11, the user's face in the real space is facing northeast. For example, if the imaging device 26 is a 360-degree camera, the output control unit 204 displays an image facing northeast from among the images captured by the imaging device 26. As a result, an image that appears to face northeast, like the orientation of the user's face in the virtual reality space, is displayed on the display device.

[0057] Furthermore, the image generation unit 205 may generate an image corresponding to the orientation of the user from the image captured by the imaging device 26. For example, if the imaging device 26 is a 360-degree camera, i.e., an omnidirectional camera, it may capture images corresponding to each orientation. In such a case, the image generation unit 205 generates an image corresponding to the orientation of the user from the omnidirectional images captured by the imaging device 26.

[0058] As described in the first embodiment, the detection unit 202 may also detect the line of sight of the user.

[0059] The detection unit 202 may also detect the movement of the user. Specifically, for example, the detection unit 202 may detect the movement of the user's head as the movement of the user. The movement of the user's head may be movement of the head forward, backward, left, or right. For example, the user's line of sight may change when standing or sitting. The detection method may be detection by various sensors, or may be detection from an image captured by the imaging device 26. Also, for example, if the display device is the HMD 24, the user may move, for example, by walking. Therefore, specifically, the detection unit 202 may detect the movement of the user's position as the movement of the user.

[0060] Then, the imaging device control unit 203 changes the imaging position of the imaging device 26 by controlling the robot arm 27 on which the imaging device 26 is installed based on the movement of the user. For example, the imaging device control unit 203 controls the robot arm 27 so that the imaging device 26 can capture an image at a position where the imaging device 26 has moved by the amount of movement of the user in each of the vertical direction, horizontal direction, upward direction, downward direction, etc. In this way, the imaging position of the imaging device 26 changes in accordance with the movement of the user, thereby enhancing the user's sense of immersion.

[0061] For example, if there is an imaging device 26 for each seat, the output control unit 204 displays other users and the interior of the vehicle in accordance with each user's seat in the vehicle in the virtual reality space. Specifically, for example, the image generation unit 205 generates an image by combining an image captured by the imaging device 26 with an image of other users and the interior of the vehicle in accordance with each user's seat in the vehicle in the virtual reality space. Then, the output control unit 204 displays the generated image on a display device.

[0062] Furthermore, for example, even if the moving object 25 is a car or a train, it may not have seats. For example, the output control unit 204 may display an image of the vehicle in the virtual reality space. Specifically, for example, the image generation unit 205 generates an image in which an image captured by the imaging device 26 installed in the moving object 25 is reflected in a window of the vehicle in the virtual reality space. Then, the output control unit 204 may display the generated image on a display device.

[0063] FIG. 12 is an explanatory diagram showing a display example of the dome-type display 21. For example, an image of the inside of a vehicle is displayed on the dome-type display 21. In the example of FIG. 6, user Y is in a vehicle that is a moving body 25 in the real space, whereas user Z is not in a vehicle that is a moving body 25 in the real space. In user Z's seat D, an imaging device 26 d has been installed.

[0064] For example, user X's seat A Imaging device 26 installed at a Assume that the camera captures images of the scenery at seats B and D in the moving body 25 and the scenery outside the window of the moving body 25 in a direction that the user wants to see and in accordance with the position of the user's seat.

[0065] In FIG. 12, the image generating unit 205 generates an image of the seat of user X. A Imaging device 26 installed at a From the video captured by the imaging device 26 installed in seat B, an image is generated in which the imaging device 26 is replaced with an avatar of user Z. Then, the output control unit 204 causes the dome-shaped display 21 to display the generated image.

[0066] In addition, the image generating unit 205 A Imaging device 26 installed at a It is also possible to generate an image that has been changed from the image captured by the camera to show the interior scenery of a vehicle in accordance with the user's preferences.

[0067] <Graphical display of audio> For example, in the case of a group tour, some users may know each other, while others may not. Group tours have the advantage of being able to share information by listening to conversations between users who are not acquaintances, but they also have the disadvantage of having to listen to conversations that you are not interested in.

[0068] Therefore, in the second embodiment, the control system 20 may allow the user to selectively hear the sound.

[0069] Specifically, for example, the output control unit 204 causes a display device to display a graphic representing the voice of a user other than the user who is conversing with the user. Note that, for example, the output control unit 204 may display the graphic together with the voice, or may display the graphic with the voice muted. Here, for example, the video generation unit 205 generates a video in which a graphic representing the voice of the other user is added to the captured video. Then, the output control unit 204 causes the generated video to be displayed on the display device.

[0070] Here, graphics include photographs, illustrations, figures, symbols, characters, etc. Graphics may be graphics corresponding to audio. Audio-related graphics may have different shapes, sizes, colors, patterns, etc. depending on the audio. For example, the color of a graphic may be different for audio related to travel and audio unrelated to travel. For example, the output control unit 204 may highlight a graphic representing audio related to travel more than a graphic representing audio unrelated to travel. Similarly, in the case of graphic patterns, the pattern for audio related to travel may be different from the pattern for audio unrelated to travel.

[0071] For example, users other than the user who is having a conversation with the user may be grouped in advance, or may be grouped according to the conversation. "Grouped in advance" means that users who will travel together are registered in advance before use begins, and the registered users who will travel together are treated as one group. For example, "grouped according to the conversation" means that if user X, user Y, and user Z are having a conversation, the users involved in the same conversation are grouped into one group. In this case, the groups change periodically.

[0072] Fig. 13 is an explanatory diagram showing an example of a graphic display of the voices of other users. For example, in Fig. 13, speech bubbles are used as the graphics. Note that the shape of the graphics is not limited to the shape of speech bubbles, and is not particularly limited as long as it is possible to identify that it represents voice.

[0073] There are three speech bubbles in FIG. 13. That is, in FIG. 13, there are three people speaking or three groups of conversations. Here, user Z is a user who is not conversing with user X, and the voice of user Z is assumed to be travel-related. On the other hand, the voices of the other users are assumed to be voices unrelated to travel. For example, the output control unit 204 highlights the graphic representing the voice related to travel more than the graphic representing the voice unrelated to travel.

[0074] In Fig. 13, the size and pattern of the graphic are used for highlighting. In Fig. 13, the size of the speech bubble representing user Z's voice is larger than the sizes of the other speech bubbles. Also, the pattern of the speech bubble representing user Z's voice is a dotted pattern, while the patterns of the other speech bubbles are solid. In this way, the speech bubble representing user Z's voice is emphasized more than the other speech bubbles. Also, the color of the graphic may be used for highlighting.

[0075] The receiving unit 206 may also receive a selection of a graphic representing audio through a user operation. The user operation may be, for example, an operation via an input device, an operation corresponding to the user's hand movement, or an operation corresponding to the user's voice; the operation method is not particularly limited. As described above, the input device may be the controller 28 or a terminal device. The user's hand movement may be detected, for example, from an image captured by the imaging device 22. The user's voice may be obtained, for example, by the recording device 23. The output control unit 204 then outputs the audio represented by the graphic selected by the user. Alternatively, for example, the output control unit 204 may not display the graphic selected by the user. Specifically, for example, the video generation unit 205 adds a graphic other than the selected graphic to the captured video, thereby generating a video without the selected graphic added. The output control unit 204 then displays the generated video on a display device.

[0076] Furthermore, for example, when the display device is a dome-shaped display 21 that does not cover the entire surface of the user, the output control unit 204 may output the conversation behind the user as audio and display the conversation in front of the user graphically without audio output. The front of the user is the direction the user is facing, and the back of the user is the direction opposite to the direction the user is facing. An example of a dome-shaped display 21 that does not cover the entire surface of the user is a 180-degree dome-shaped display.

[0077] 14 is an explanatory diagram showing an example of switching from graphic output to audio output. For example, in FIG. 14, the output control unit 204 displays a speech bubble representing the voice of user Z on the dome-shaped display 21. The reception unit 206 receives a graphic selection by an operation of user X. The output control unit 204 outputs the voice of user Z represented by the selected graphic as audio.

[0078] Furthermore, the receiving unit 206 may receive an output format for the voice of another user. The output control unit 204 outputs the voice of another user in the received format. Specifically, for example, the receiving unit 206 may be able to select in stages between a graphic representing the voice and the voice itself. The receiving unit 206 may also be able to select the graphic in stages. Then, the output control unit 204 switches the output in stages. For example, the output control unit 204 may switch in stages so that more detailed content of the voice is displayed, such as a graphic representing a predetermined shape with a color, or a graphic representing the voice with text.

[0079] <Shopping> For example, shopping may be performed while traveling in a virtual reality space.

[0080] The product determination unit 207 determines recommended products from products included in the product DB 2003. Here, the products included in the product DB 2003 are products identified by product identification information stored in the product DB 2003. Specifically, for example, the product determination unit 207 determines products related to travel destinations as recommended products. Furthermore, the product determination unit 207 may determine recommended products from among products related to travel destinations based on a conversation of the user.

[0081] For example, the output control unit 204 presents information about recommended products to the user. As a specific presentation method, the output control unit 204 may cause a display device to display the information about recommended products, or may notify the user's terminal device or the like of the information about recommended products.

[0082] For example, products may be sold in a manner similar to in-train or in-flight sales. Therefore, the output control unit 204 may also display an avatar of a store clerk on the display device, and the avatar of the store clerk may present information about recommended products.

[0083] The registration unit 208 accepts product registration through user operation. The settlement unit 209 settles the payment for the registered product through user operation. Note that existing technology may be used for the product registration method and the payment method.

[0084] (flowchart) 15 is a flowchart showing an example of an operation of the control system 20 according to the second embodiment. The acquiring unit 201 acquires data in which the imaging device identification information of the imaging device 26 and the display device identification information of the display device are associated with each other (step S201). For example, the acquiring unit 201 acquires an association table 2001 as this data.

[0085] The detection unit 202 detects the direction in which the user wants to look (step S202). The imaging device control unit 203 controls the direction of the imaging device 26 based on the detected direction, or generates an image corresponding to the direction in which the user wants to look from an image captured by the imaging device 26 (step S203). Note that in step S203, the imaging device control unit 203 may control the direction of the imaging device 26 based on the direction in which the user wants to look, and may also generate an image corresponding to the direction of the face from the captured image. The output control unit 204 displays the image captured by the imaging device 26 or the generated image on a display device identified by display device identification information associated with the imaging device identification information of the imaging device 26 (step S204).

[0086] The detection unit 202 determines whether the movement of the user has been detected (step S205). If the movement of the user has not been detected (step S205: No), the detection unit 202 proceeds to step S207. If the movement of the user has been detected (step S205: Yes), the imaging device control unit 203 controls the imaging position of the imaging device 26 by controlling the robot arm 27 (step S206).

[0087] Next, the detection unit 202 determines whether the direction in which the user wants to look has changed (step S207). If the direction in which the user wants to look has changed (step S207: Yes), the imaging device control unit 203 returns to step S203. If the direction in which the user wants to look has not changed (step S207: No), the output control unit 204 returns to step S204.

[0088] The flowchart may be terminated as appropriate.

[0089] As described above, in the second embodiment, the control system 20 controls the orientation of the imaging device 26 in accordance with the orientation of the user's seat in the vehicle in the virtual reality space. This allows the user to view images similar to what they would see from their seat when riding in an actual vehicle. This makes it possible to provide the user with a better viewing experience.

[0090] The control system 20 detects the user's line of sight and controls the orientation of the imaging device 26 based on the line of sight and the user's position in the virtual reality space. This allows the user to view images similar to what they would see from their seat in a real vehicle, thereby providing the user with a better viewing experience.

[0091] The control system 20 detects the movement of the user and, based on the user's movement, controls the robot arm 27 on which the imaging device 26 is installed, thereby changing the imaging position of the imaging device 26. This allows the image to change depending on the user's up and down movements, etc., thereby improving the sense of realism.

[0092] Furthermore, the control system 20 generates an image according to the orientation of the user from the image captured by the imaging device 26, and displays the generated image on the display device.

[0093] Furthermore, when an image captured by the imaging device 26 includes an imaging device other than the imaging device 26, the control system 20 generates an image from the image by deleting the other imaging device from the image. For example, the user can concentrate on the experience provided by the image, and the sense of realism can be improved.

[0094] The control system 20 displays other users and the interior of the vehicle in accordance with the seat of each user in the vehicle in the virtual reality space. For example, the control system 20 may generate an image by combining an image captured by an imaging device 26 installed in the moving body 25 with an image of the interior of the vehicle in the virtual reality space, and display the generated image. In this way, the control system 20 can provide an experience that is not possible with actual images. interpolation and provide it to users.

[0095] The control system 20 displays, in graphics, the voices of users other than the user who is conversing with the user. Users want to enjoy realistic images, or in the case of travel, enjoy group trips on vehicles. However, there are times when users want to hear what other users are saying, and times when they don't want to hear it. For example, users want to hear information that is useful to them. This allows users, for example, to ignore the conversations of other users if they don't look at the graphics. On the other hand, if the user is interested, they can just look at the graphics. In this way, the control system 20 can provide a better video experience by combining the benefits of real travel with the benefits of virtual reality space.

[0096] Furthermore, the control system 20 outputs the sound represented by the graphic selected by the user from among the displayed graphics, so that the user can, for example, hear the sound of a conversation in which the user is interested.

[0097] The control system 20 does not display the graphics selected by the user, thereby allowing the user to remove from view the graphics representing conversations in which the user is not interested.

[0098] Furthermore, when the display device is a hemispherical dome display 21, the control system 20 may output the conversation behind the user in the virtual reality space as audio and display the conversation in front of the user in the virtual reality space as graphics.

[0099] The control system 20 can accept an output format for the voices of other users and output the voices of other users in the accepted output format. For example, the control system 20 may be able to select the output format in stages, in the order of color, text, and sound. This allows the output format of the conversations of other users to be changed according to the user's interests.

[0100] The above is the explanation of each embodiment. The embodiments may be used in combination as appropriate.

[0101] In each embodiment, the control system may be configured to include each functional unit and part of the information.

[0102] Furthermore, each embodiment is not limited to the above-described examples and can be modified in various ways. Furthermore, the configuration of the control system in each embodiment is not particularly limited. For example, the control system may be realized by a single device such as a single server. When each functional unit of the control system is realized by a single device, the single device may be called, for example, a control device, an information processing device, or the like, and is not particularly limited. Alternatively, the control system in each embodiment may be realized by different devices for different functions or data. For example, each functional unit may be configured by multiple servers and realized as a control system. For example, the control system may be realized by a database server including each DB (DataBase) and a server having each functional unit.

[0103] 16 is an explanatory diagram showing an example of a control system. In FIG. 16, the control system includes, for example, an edge terminal device 31 and a server 32.

[0104] For example, the edge terminal device 31, the dome-shaped display 21, the imaging device 22, and the sound recording device 23 are installed in the user's home, a shared space, etc. The edge terminal device 31, the dome-shaped display 21, the imaging device 22, the sound recording device 23, and the controller 28 are connected via a communication network.

[0105] For example, the imaging device 26 and the robot arm 27 are installed on a moving body 25. The imaging device 26 is attached to the robot arm 27. The imaging device 26, the edge terminal device 31, and the server 32 are connected via a communication network.

[0106] Furthermore, the control system 20 may be configured as an entire system including an edge terminal device 31, a server 32, a dome-shaped display 21, an imaging device 22, a sound recording device 23, an imaging device 26, a robot arm 27, and a controller 28.

[0107] 16, for example, each functional unit in each embodiment is realized by an edge terminal device 31 and a server 32. The server 32 may be a plurality of servers. In this way, each functional unit of the control systems 10 and 20 may be realized by a plurality of devices, and the plurality of devices may be installed in different locations.

[0108] Furthermore, in each embodiment, each piece of information or each DB may include a portion of the information described above. Furthermore, each piece of information or each DB may include information other than the information described above. Each piece of information or each DB may be divided into more detailed pieces of information or multiple DBs. In this way, the method for realizing each piece of information or each DB is not particularly limited.

[0109] Furthermore, each screen is merely an example and is not particularly limited. Buttons, lists, check boxes, information display fields, input fields, etc. (not shown) may be added to each screen. Furthermore, the background color of the screen may be changed.

[0110] Furthermore, the process of generating information to be displayed on the display device may be performed by the output control units 104 and 204. This process may also be performed by the display device.

[0111] (Example of computer hardware configuration) Next, an example of a hardware configuration in which each device, such as the control systems 10 and 20 described in each embodiment, is implemented by a computer will be described. Fig. 17 is an explanatory diagram showing an example of a hardware configuration of a computer. For example, some or all of each device can be implemented using any combination of a computer 80 and a program as shown in Fig. 17.

[0112] The computer 80 includes, for example, a processor 801, a read-only memory (ROM) 802, a random access memory (RAM) 803, and a storage device 804. The computer 80 also includes a communication interface 805 and an input / output interface 806. The components are connected to each other, for example, via a bus 807. The number of each component is not particularly limited, and there may be one or more of each component.

[0113] The processor 801 controls the entire computer 80. Examples of the processor 801 include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). The computer 80 has a storage unit including a ROM 802, a RAM 803, and a storage device 804. Examples of the storage device 804 include a semiconductor memory such as a flash memory, a hard disk drive (HDD), and a solid state drive (SSD). For example, the storage device 804 stores an operating system (OS) program, application programs, and programs according to the embodiments. Alternatively, the ROM 802 stores application programs and programs according to the embodiments. The RAM 803 is used as a work area for the processor 801.

[0114] The processor 801 also loads programs stored in the storage device 804, ROM 802, etc. The processor 801 then executes each process coded in the program. The processor 801 may also download various programs via the communication network NT. The processor 801 also functions as a part or all of the computer 80. The processor 801 may then execute the processes or instructions in the illustrated flowchart based on the program.

[0115] The communication interface 805 is connected to a communication network NT such as a LAN (Local Area Network) or WAN (Wide Area Network) via a wireless or wired communication line. The communication network NT may be composed of multiple communication networks NT. As a result, the computer 80 is connected to external devices and external computers 80 via the communication networks NT. The communication interface 805 serves as an interface between the communication network NT and the inside of the computer 80. The communication interface 805 also controls the input and output of data from external devices and external computers 80.

[0116] Furthermore, the input / output interface 806 is connected to at least one of an input device, an output device, and an input / output device. The connection method may be wireless or wired. Examples of the input device include a keyboard, a mouse, and a microphone. Examples of the output device include a display device, a lighting device, and an audio output device that outputs audio. Examples of the input / output device include a touch panel display. The input device, output device, and input / output device may be built into the computer 80 or may be external.

[0117] The hardware configuration of the computer 80 is an example. The computer 80 may have some of the components shown in FIG. 17. The computer 80 may have components other than those shown in FIG. 17. For example, the computer 80 may have a drive device or the like. The processor 801 may then read programs and data stored on a recording medium attached to the drive device or the like into the RAM 803. Examples of non-transitory tangible recording media include optical disks, flexible disks, magneto-optical disks, and USB (Universal Serial Bus) memories. As described above, the computer 80 may also have input devices such as a keyboard and a mouse. The computer 80 may also have an output device such as a display. The computer 80 may also have an input device, an output device, and an input / output device.

[0118] The computer 80 may also have various sensors (not shown). The types of sensors are not particularly limited. The computer 80 may also have an imaging device capable of capturing images or videos.

[0119] This concludes the description of the hardware configuration of each device. There are various variations in the method of realizing each device. For example, each device may be realized by any combination of a different computer and program for each component. Furthermore, multiple components of each device may be realized by any combination of a single computer and program.

[0120] Furthermore, some or all of the components of each device may be realized by circuits for specific applications. Furthermore, some or all of the components of each device may be realized by general-purpose circuits including a processor such as an FPGA (Field Programmable Gate Array). Furthermore, some or all of the components of each device may be realized by a combination of circuits for specific applications and general-purpose circuits. Furthermore, these circuits may be a single integrated circuit. Alternatively, these circuits may be divided into multiple integrated circuits. Furthermore, the multiple integrated circuits may be configured by being connected via a bus or the like.

[0121] Furthermore, when some or all of the components of each device are realized by a plurality of computers, circuits, etc., the plurality of computers, circuits, etc. may be centrally located or distributed.

[0122] The control method described in each embodiment is realized by a control system executing the control method. For example, the control method is realized by a computer such as a server or a terminal device executing a prepared program.

[0123] The programs described in each embodiment are recorded on a computer-readable recording medium such as a HDD, SSD, flexible disk, optical disk, magneto-optical disk, or USB memory. The programs are then read from the recording medium and executed by a computer. The programs may also be distributed via a communications network NT.

[0124] The functions of each component of the control system in each embodiment described above may be realized by dedicated hardware, such as a computer. Alternatively, each component may be realized by software. Alternatively, each component may be realized by a combination of hardware and software.

[0125] Although the present disclosure has been described above with reference to various embodiments, the present disclosure is not limited to the above embodiments. The configuration and details of each of the present disclosures may include embodiments to which various modifications that would be apparent to those skilled in the art are applied within the scope of the present disclosure. The present disclosure may also include embodiments in which the details described herein are appropriately combined or substituted as necessary. For example, details described using a particular embodiment may also be applied to other embodiments to the extent that no contradiction occurs. For example, although multiple operations are described in sequence in the form of a flowchart, the order of the descriptions does not limit the order in which the multiple operations are performed. Therefore, when implementing each embodiment, the order of the multiple operations may be changed as long as it does not interfere with the content.

[0126] Some or all of the above-described embodiments can be described as follows: However, some or all of the above-described embodiments are not limited to the following.

[0127] (Appendix 1) an acquisition means for acquiring data in which imaging device identification information of an imaging device attached to a mobile object and display device identification information of a display device of a user are associated with each other; a detection means for detecting a direction in which the user wants to view; an imaging device control means for controlling the orientation of an imaging device based on the detected orientation and the position of the user in the virtual reality space; an output control means for displaying the image captured by the imaging device on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device; A control system comprising: (Appendix 2) the imaging device control means controls the orientation of the imaging device in accordance with the orientation of the seat of the user in the vehicle in the virtual reality space. 10. The control system of claim 1. (Appendix 3) the direction in which the user wants to look is the direction of the user's face, the detection means detects the orientation of the face, The imaging device control means controls the orientation of the imaging device based on the orientation of the face and the position of the user in the virtual reality space. 3. The control system of claim 1 or 2. (Appendix 4) the detection means detects the line of sight of the user, The imaging device control means controls the orientation of the imaging device based on the line of sight and the position of the user in the virtual reality space. 10. The control system of claim 3. (Appendix 5) the detection means detects the movement of the user; the imaging device control means controls a robot arm on which the imaging device is installed based on the movement of the user, thereby changing the imaging position of the imaging device. 5. The control system of any one of claims 1 to 4. (Appendix 6) an image generating means for generating an image corresponding to the direction in which the user wants to view the image from the image captured by the imaging device; Equipped with the output control means causes the display device to display the generated video. 6. The control system of any one of claims 1 to 5. (Appendix 7) When an imaging device other than the imaging device is shown in the image captured by the imaging device, the image generation means generates an image in which the other imaging device is deleted from the image captured by the imaging device. 6. The control system of claim 6. (Appendix 8) The output control means displays other users and the interior of the vehicle in accordance with the seat of each user in the vehicle in the virtual reality space. 8. The control system of any one of claims 1 to 7. (Appendix 9) the output control means displays a graphic representing the voice of a user other than the user who is conversing with the user; 9. The control system of any one of claims 1 to 8. (Appendix 10) a receiving means for receiving a selection of a graphic by the user from among the graphics; Equipped with the output control means outputs a sound represented by the selected graphic. 10. The control system of claim 9. (Appendix 11) a receiving means for receiving a selection of a graphic by the user from among the graphics; Equipped with the output control means does not allow the selected graphic to be displayed; 10. The control system of claim 9. (Appendix 12) In the case where the display device is a hemispherical dome-type display, the output control means outputs a conversation occurring behind the user in the virtual reality space as audio and displays a conversation occurring in front of the user in the virtual reality space as graphics; 12. The control system of any one of appendixes 9 to 11. (Appendix 13) a receiving means for receiving an output format for the voice of the other user; Equipped with the output control means outputs the voice of the other user in the accepted output format; 13. The control system of any of claims 9 to 12. (Appendix 14) acquiring data in which imaging device identification information of the imaging device attached to the mobile body is associated with display device identification information of the user's display device; Detecting the direction in which the user wants to view; displaying the image captured by the imaging device on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device, based on the detected orientation and the position of the user in the virtual reality space; Control method. (Appendix 15) On the computer, acquiring data in which imaging device identification information of the imaging device attached to the mobile body is associated with display device identification information of the user's display device; Detecting the direction in which the user wants to view; displaying the image captured by the imaging device on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device, based on the detected orientation and the position of the user in the virtual reality space; A non-transitory recording medium readable by the computer that records a program for executing processing. (Appendix 16) On the computer, acquiring data in which imaging device identification information of the imaging device attached to the mobile body is associated with display device identification information of the user's display device; Detecting the direction in which the user wants to view; displaying the image captured by the imaging device on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device, based on the detected orientation and the position of the user in the virtual reality space; A program that executes a process. [Explanation of symbols]

[0128] 10,20 Control System 21 Dome-shaped display 22 Imaging device 23 Recording Devices 24 HMD 25 Mobile 26 Imaging device 27 Robot Arm 28 Controller 31 Edge terminal device 32 servers 80 Computer 101,201 Acquisition Department 102,202 Detector 103,203 Imaging device control section 104,204 Output control section 205 Image Generation Unit 206 Reception 207 Product Determination Department 208 Registration Department 209 Settlement Department 801 processor 802 ROM 803 RAM 804 Storage device 805 Communication Interface 806 Input / Output Interface 807 Bus 2001 Association Table 2002 User DB 2003 Product DB 2101 Screen 2102 Projection device

Claims

1. an acquisition means for acquiring data in which imaging device identification information of an imaging device attached to a moving object in real space is associated with display device identification information of a user's display device; a detection means for detecting a direction in which the user wants to look by detecting a facial direction or a line of sight of the user using a sensor or by receiving an operation of the user on an input device; an imaging device control means for controlling an orientation of the imaging device in the real space based on the detected orientation and a position of the user within a virtual moving body in the virtual reality space displayed on the display device; an image generating means for generating an image corresponding to the direction in which the user wants to view the image from the image captured by the imaging device in the real space; an output control means for displaying the generated image on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device in the real space; Equipped with When the image captured by the imaging device includes an imaging device other than the imaging device attached to the moving body in the real space, the image generation means generates an image in which the other imaging device is deleted from the image. Control system.

2. the virtual moving body is a virtual vehicle, the imaging device control means controls the orientation of the imaging device in accordance with the orientation of the seat of the user in the virtual vehicle in the virtual reality space. The control system of claim 1 .

3. the direction in which the user wants to look is the direction of the user's face, the detection means detects the orientation of the face using the sensor; the imaging device control means controls the orientation of the imaging device based on the orientation of the face and the position of the user within the virtual moving body in the virtual reality space.

3. A control system according to claim 1 or 2.

4. the detection means detects the line of sight of the user by the sensor; the imaging device control means controls the orientation of the imaging device based on the line of sight and the position of the user within the virtual moving body in the virtual reality space. The control system of claim 3 .

5. the detection means detects the movement of the user; the imaging device control means controls a robot arm on which the imaging device is installed based on the movement of the user, thereby changing the imaging position of the imaging device.

3. A control system according to claim 1 or 2.

6. the virtual moving body is a virtual vehicle, the output control means displays other users and the interior of the vehicle in accordance with the seat of each user in the virtual vehicle in the virtual reality space; 3. A control system according to claim 1 or 2.

7. The computer acquiring data in which imaging device identification information of an imaging device attached to a moving body in real space is associated with display device identification information of a user's display device; Detecting a direction in which the user wants to look based on data acquired from a sensor capable of detecting the direction of the user's face or line of sight, or data acquired from an input device relating to the user's operation on the input device; controlling an orientation of the imaging device in the real space based on the detected orientation and a position of the user within a virtual moving body in the virtual reality space displayed on the display device; generating an image corresponding to the direction that the user wants to view from the image captured by the imaging device in the real space; displaying the generated image on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device in the real space; Execute the process, In the generation, if the image captured by the imaging device includes an image capturing device other than the imaging device attached to the moving body in the real space, an image is generated in which the other imaging device is deleted from the image. Control method.

8. On the computer, acquiring data in which imaging device identification information of an imaging device attached to a moving body in real space is associated with display device identification information of a user's display device; Detecting a direction in which the user wants to look based on data acquired from a sensor capable of detecting the direction of the user's face or line of sight, or data acquired from an input device relating to the user's operation on the input device; controlling an orientation of the imaging device in the real space based on the detected orientation and a position of the user within a virtual moving body in the virtual reality space displayed on the display device; generating an image corresponding to the direction that the user wants to view from the image captured by the imaging device in the real space; displaying the generated image on the display device identified by the display device identification information associated with the imaging device identification information of the imaging device in the real space; Execute the process, In the generation, if the image captured by the imaging device includes an image capturing device other than the imaging device attached to the moving body in the real space, an image is generated in which the other imaging device is deleted from the image. program.

Images