Video output device, program, and video output method

Abstract

An information processing device (130) is provided with: a position identification unit (131) that identifies the position of a virtual user who moves in a virtual space; a line-of-sight direction identification unit (132) that identifies a line-of-sight direction of the virtual user; a free-viewpoint video generation unit (134) that generates a free-viewpoint video that is a video of the virtual space in the identified line-of-sight direction at the identified position; a switching determination unit (135) that, while the free-viewpoint video is being output, determines whether or not a predetermined condition is satisfied; and a switching processing unit (137) that, when the predetermined condition is satisfied, switches the video that is output from the free-viewpoint video to a real-time video.

Description

Video output device, program, and video output method

[0001] The present disclosure relates to a video output device, a program, and a video output method.

[0002] A free viewpoint video technology has been known for some time, in which a three-dimensional (3D) model is rendered in a model space, and an image is displayed in a viewing direction that the user can freely change within the model space (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2001-291120

[0004] Rendering a 3D model requires a large number of computational resources, making it very difficult to reflect the real-time situation in a space in free-viewpoint video.

[0005] Therefore, one or more aspects of the present disclosure aim to make it possible to easily reflect real-time situations in free viewpoint videos.

[0006] A video output device according to one aspect of the present disclosure is a video output device that outputs video, and is characterized by comprising: a position identification unit that identifies the position of a virtual user moving in a virtual space; a gaze direction identification unit that identifies the gaze direction of the virtual user; a first video generation unit that generates a first video, which is a video of the virtual space at the identified position in the identified gaze direction; a judgment unit that determines whether a predetermined condition is met when the first video is being output; and a video processing unit that switches the video to be output from the first video to a second video, which is a video that reflects the real-time state, when the predetermined condition is met.

[0007] A program according to one aspect of the present disclosure is a program that causes a computer to function as a video output device that outputs video, and is characterized by causing the computer to function as a position identification unit that identifies the position of a virtual user moving in a virtual space, a gaze direction identification unit that identifies the gaze direction of the virtual user, a first video generation unit that generates a first video that is a video of the virtual space at the identified position in the identified gaze direction, a judgment unit that determines whether a predetermined condition is met when the first video is being output, and a video processing unit that switches the video to be output from the first video to a second video that is a video that reflects the real-time state when the predetermined condition is met.

[0008] A video output method according to one aspect of the present disclosure is a video output method for outputting video, which includes identifying the position of a virtual user moving in a virtual space, identifying the line of sight of the virtual user, generating a first video that is a video of the virtual space at the identified position in the identified line of sight, determining whether a predetermined condition is met while the first video is being output, and, if the predetermined condition is met, switching the video to be output from the first video to a second video that is a video that reflects the real-time state.

[0009] According to one or more aspects of the present disclosure, real-time situations can be easily reflected in free viewpoint video.

[0010] FIG. 1 is a block diagram schematically showing the configuration of an information processing system according to embodiments 1 to 6. FIG. 2 is a block diagram schematically showing the configuration of an information processing device according to embodiment 1. FIG. 3 is a block diagram schematically showing the configuration of a PC. FIG. 4 is a block diagram schematically showing the configuration of an information processing device according to embodiment 2. (A) and (B) are schematic diagrams for explaining 3D data. FIG. 5 is a block diagram schematically showing the configuration of an information processing device according to embodiments 3 and 4. FIG. 6 is a block diagram schematically showing the configuration of an information processing device according to embodiment 5. FIG. 7 is a block diagram schematically showing the configuration of an information processing device according to embodiment 6.

[0011] 1 is a block diagram showing a schematic configuration of an information processing system 100 according to embodiment 1. The information processing system 100 includes an imaging device 110, a user device 120, and an information processing device 130.

[0012] The imaging device 110 and the information processing device 130 are connected to a network 101 such as the Internet, and the user device 120 and the information processing device 130 are directly connected. The connection between the user device 120 and the information processing device 130 may be wired or wireless. Note that the above connection relationship is an example, and for example, the imaging device 110 and the information processing device 130 may be directly connected. Furthermore, the user device 120 may be connected to the network 101.

[0013] The imaging device 110 captures video in real time. The video captured by the imaging device 110 is also referred to as real-time video. The real-time video is sent to the information processing device 130 via the network 101.

[0014] The user device 120 includes at least a display serving as a display unit that displays an output image from the information processing device 130, and an output unit that outputs a signal that can identify the position and line of sight of the virtual user in the virtual space provided by the information processing device 130. The user device 120 may also include an input unit that accepts input of instructions from the user. The instructions input to the input unit may be sent from the output unit to the information processing device 130.

[0015] In the first embodiment, the user device 120 is an HMD (Head Mounted Display), but it may also be a monitor, keyboard, and mouse.

[0016] The information processing device 130 functions as a video output device that outputs video to the user device 120 .

[0017] 2 is a block diagram schematically illustrating the configuration of information processing device 130 according to Embodiment 1. Information processing device 130 includes position identification unit 131, gaze direction identification unit 132, 3D data storage unit 133, free viewpoint video generation unit 134, switching determination unit 135, real-time video acquisition unit 136, switching processing unit 137, and video output unit 138.

[0018] The position identification unit 131 receives a signal from the user-side device 120 via, for example, a connection unit (not shown), and identifies the position of the virtual user moving in the virtual space via the user-side device 120. Position information indicating the identified position is provided to the free viewpoint video generation unit 134 and the switching determination unit 135.

[0019] For example, the user can move a virtual user corresponding to the user within the virtual space via the user-side device 120, and the position identification unit 131 can identify the position to which the virtual user has moved based on a signal from the user-side device 120. For example, if the user-side device 120 is an HMD, the user can input various instructions to the user-side device 120 to move the virtual user corresponding to the user within the virtual space in accordance with the instructions.

[0020] The gaze direction specifying unit 132 specifies the gaze direction of the virtual user. For example, the gaze direction specifying unit 132 receives a signal from the user-side device 120 via a connection unit (not shown), and specifies the gaze direction, which is the direction in which the virtual user is looking in the virtual space, via the user-side device 120. Gaze direction information indicating the specified gaze direction is provided to the free viewpoint video generation unit 134.

[0021] Specifically, the user can change the gaze direction within a predetermined space via the user device 120, and the gaze direction identification unit 132 identifies the gaze direction changed by the user in response to an instruction from the user. If the user device 120 is an HMD, the user's gaze direction can be identified by signals from various sensors provided in the HMD.

[0022] The 3D data storage unit 133 stores 3D data, which is data in 3D of a virtual space provided to the user via the user-side device 120. The 3D data may include data necessary for generating a free viewpoint video, which is a 2D video of the virtual space, from a position and a line-of-sight direction within a certain virtual space. The 3D data may be pre-generated using known techniques such as CAD (Computer Aided Design), LiDAR (Light Detection And Ranging), photogrammetry, NeRF (Neural Radiance Fields), 3D Gaussian Splatting, etc.

[0023] The free viewpoint video generation unit 134 is a first video generation unit that generates a first video, which is a video of the virtual space in the line-of-sight direction specified by the line-of-sight direction specification unit 132 at the position specified by the position specification unit 131. For example, the free viewpoint video generation unit 134 refers to the 3D data stored in the 3D data storage unit 133, and generates, as the first video, a free viewpoint video, which is a 2D video of the virtual space in the line-of-sight direction indicated by the line-of-sight direction information from the line-of-sight direction specification unit 132 at the position indicated by the position information from the position specification unit 131. The generated free viewpoint video is provided to the switching processing unit 137.

[0024] The switching determination unit 135 is a determination unit that determines the switching between the free viewpoint video from the free viewpoint video generation unit 134 and the real-time video captured by the imaging device 110. For example, the switching determination unit 135 determines whether a predetermined condition is satisfied when the free viewpoint video is being output.

[0025] Specifically, when the position indicated by the position information from the position identification unit 131 enters a switching range, which is a predetermined range in a certain space, in other words, when the virtual user enters the switching range, the switching determination unit 135 determines to switch the video to be output to the user device 120 from the free viewpoint video to a second video, which is a video reflecting a real-time state. In the first embodiment, the second video is a real-time video. Furthermore, when the position indicated by the position information leaves the switching range, the switching determination unit 135 determines to switch the video to be output to the user device 120 from the real-time video to the free viewpoint video. The above determination is notified to the switching processing unit 137.

[0026] The real-time video acquisition unit 136 acquires real-time video from the imaging device 110 via a communication unit (not shown). The acquired real-time video is provided to the switching processing unit 137.

[0027] The switching processing unit 137 functions as a video processing unit that switches between the free viewpoint video and the real-time video upon receiving a notification from the switching determination unit 135. The switched video is provided as an output video to the video output unit 138. In other words, when a predetermined condition is satisfied, the switching processing unit switches the video to be output from the free viewpoint video, which is the first video, to the real-time video, which is the second video.

[0028] The video output unit 138 outputs the output video from the switching processing unit 137 to the user device 120 via, for example, a communication unit (not shown).

[0029] The information processing device 130 described above can be realized by, for example, a computer such as the PC 10 shown in Fig. 3. The PC 10 includes a storage 11 such as a hard disk drive (HDD) or a solid state drive (SSD), a memory 12, a processor 13 such as a central processing unit (CPU), a communication interface (I / F) 14 such as a network interface card (NIC), and a connection interface 15 such as a universal serial bus (USB) connector.

[0030] For example, the 3D data storage unit 133 can be realized by the storage 11. The position identification unit 131, the line of sight direction identification unit 132, the free viewpoint video generation unit 134, the switching determination unit 135, the real-time video acquisition unit 136, the switching processing unit 137, and the video output unit 138 can be realized by loading a program stored in the storage 11 into the memory 12 and having the processor 13 execute the program.

[0031] A communication unit (not shown) can be realized by the communication I / F 14. A connection unit (not shown) can be realized by the connection I / F 15.

[0032] The above programs may be downloaded to the storage 11 from a recording medium (not shown) via a reader / writer (not shown) or from the network 101 via the communication I / F 14, and then loaded onto the memory 12 and executed by the processor 13. Alternatively, the programs may be directly loaded onto the memory 12 from a recording medium via the reader / writer or from the network 101 via the communication I / F 14, and then executed by the processor 13. In other words, the programs may be provided by a program product such as a recording medium.

[0033] As described above, according to embodiment 1, real-time video is output only when the virtual user enters a predetermined range in the space in which the virtual user moves, so that the real-time situation can be easily reflected in the free viewpoint video.

[0034] In the first embodiment described above, the free viewpoint video and the real-time video are switched depending on the position of the virtual user, but the first embodiment is not limited to this example. For example, such switching may be performed when the user inputs a predetermined instruction to the user device 120.

[0035] 1, an information processing system 200 according to the second embodiment includes an imaging device 210, a user-side device 120, and an information processing device 230. The user-side device 120 of the information processing system 200 according to the second embodiment is similar to the user-side device 120 of the information processing system 100 according to the first embodiment.

[0036] The imaging device 210 captures real-time video, which is video that is captured in real time. In the second embodiment, the imaging device 210 is configured to capture video in a 360° direction so that 2D video from the viewpoint of the virtual user can be generated. The real-time video is sent to the information processing device 230 via the network 101.

[0037] 4 is a block diagram schematically illustrating the configuration of an information processing device 230 according to Embodiment 2. The information processing device 230 includes a position specifying unit 131, a line of sight direction specifying unit 132, a 3D data storage unit 233, a free viewpoint video generating unit 234, a switching determination unit 135, a real-time video acquisition unit 136, a switching processing unit 237, a video output unit 138, and a real-time viewpoint video generating unit 239.

[0038] The position identifying unit 131, the gaze direction identifying unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 230 in Embodiment 2 are similar to the position identifying unit 131, the gaze direction identifying unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 130 in Embodiment 1. However, the gaze direction information generated by the gaze direction identifying unit 132 is also provided to the real-time viewpoint video generation unit 239. In addition, the real-time video acquired by the real-time video acquisition unit 136 is provided to the real-time viewpoint video generation unit 239.

[0039] The 3D data storage unit 233 stores 3D data, which is 3D data of the virtual space provided to the user via the user-side device 120. In the second embodiment, for example, as shown in Fig. 5(A), when the imaging device 210 captures a 360-degree video of a certain space 102, the 3D data is data necessary to generate a free viewpoint video, which is a 2D video of a virtual space 103 of the space 102, as shown in Fig. 5(B). In other words, the virtual space 103 is a space that virtually represents the space 102 in which the imaging device 210 is installed.

[0040] Furthermore, in the 3D data in the second embodiment, a switching range 104 is defined as a predetermined range that includes a position corresponding to the position where the image capture device 210 is placed in the space 102. The switching range 104 indicates the range in the virtual space where, when a virtual user enters the switching range, the free viewpoint video as the first video is switched to the real-time viewpoint video as the second video.

[0041] The free viewpoint video generation unit 234 references the 3D data stored in the 3D data storage unit 233 and generates a 2D video of the virtual space in the line of sight direction indicated by the line of sight direction information from the line of sight direction identification unit 132 at the position indicated by the position information from the position identification unit 131. As described above, the 2D video generated here is a video of the virtual space 103 that virtually represents the space 102 in which the imaging device 210 is placed.

[0042] Furthermore, when the switching range indicated by the 3D data is included in the virtual space at the position indicated by the position information from the position identification unit 131 and in the line of sight direction indicated by the line of sight direction information from the line of sight direction identification unit 132, the free viewpoint video generation unit 234 includes a switching range image, which is an image indicating the switching range, in the 2D video. The switching range image may be any image that allows the user to recognize that the video will switch when the virtual user enters that position. Here, the switching range image is assumed to be a circular line superimposed at a position corresponding to the ground. Alternatively, a cylindrical translucent image corresponding to the switching range may be superimposed as the switching range image.

[0043] The free viewpoint video generation unit 234 then provides the 2D video or the video in which the switching range image is superimposed on the 2D video to the switching processing unit 237 as a free viewpoint video.

[0044] The real-time viewpoint video generation unit 239 generates a real-time viewpoint video by cutting out a 2D video in the line of sight direction indicated by the line of sight direction information from the line of sight direction identification unit 132 from the real-time video provided by the real-time video acquisition unit 136. In other words, the real-time viewpoint video is a video in the line of sight direction indicated by the line of sight direction information, extracted from the real-time video. The generated real-time viewpoint video is provided to the switching processing unit 237.

[0045] The switching processing unit 237 switches between the free viewpoint video and the real-time viewpoint video upon receiving the notification from the switching determination unit 135. The switched video is provided to the video output unit 138 as an output video.

[0046] The information processing device 230 described above can also be realized by, for example, a computer such as the PC 10 shown in Fig. 3. For example, the real-time viewpoint video generation unit 239 can also be realized by loading a program stored in the storage 11 into the memory 12 and having the processor 13 execute the program.

[0047] As described above, according to the second embodiment, the switching range, which is the range in which the video switches, is displayed in the free viewpoint video, thereby reducing the sense of discomfort caused by the video switching when the virtual user moves through the virtual space.

[0048] In addition, when generating 2D images, the free viewpoint image generation unit 234 sets the height of the virtual user's viewpoint to the same height as the height at which the imaging device 210 is capturing images, thereby further reducing the sense of discomfort when the images switch.

[0049] 1 , an information processing system 300 according to the third embodiment includes an imaging device 210, a user-side device 120, and an information processing device 330. The user-side device 120 of the information processing system 300 according to the third embodiment is similar to the user-side device 120 of the information processing system 100 according to the first embodiment. Moreover, the imaging device 210 of the information processing system 300 according to the third embodiment is similar to the imaging device 210 of the information processing system 200 according to the second embodiment.

[0050] 6 is a block diagram schematically illustrating the configuration of an information processing device 330 according to Embodiment 3. The information processing device 330 includes a position specifying unit 131, a line of sight direction specifying unit 132, a 3D data storage unit 333, a free viewpoint video generation unit 334, a switching determination unit 135, a real-time video acquisition unit 136, a switching processing unit 337, a video output unit 138, and a real-time viewpoint video generation unit 239.

[0051] The position identifying unit 131, the gaze direction identifying unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 330 in Embodiment 3 are similar to the position identifying unit 131, the gaze direction identifying unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 130 in Embodiment 1. However, the gaze direction information generated by the gaze direction identifying unit 132 is also provided to the real-time viewpoint video generation unit 239. In addition, the real-time video acquired by the real-time video acquisition unit 136 is provided to the real-time viewpoint video generation unit 239.

[0052] The real-time viewpoint video generation unit 239 of the information processing device 330 in the third embodiment is similar to the real-time viewpoint video generation unit 239 of the information processing device 230 in the second embodiment.

[0053] The 3D data storage unit 333 stores 3D data, which is 3D data of a virtual space provided to a user via the user-side device 120. In the third embodiment as well, when the imaging device 210 captures a 360-degree video of a certain space, the 3D data is data necessary for generating a free viewpoint video, which is a 2D video of the virtual space of that space.

[0054] The free viewpoint video generation unit 334 references the 3D data stored in the 3D data storage unit 333, and generates a 2D video of the virtual space in the line of sight indicated by the line of sight direction information from the line of sight direction identification unit 132, at the position indicated by the position information from the position identification unit 131. The free viewpoint video generation unit 334 then provides the 2D video generated here to the switching processing unit 337 as a free viewpoint video.

[0055] The switching processing unit 337 switches between the free viewpoint video and the real-time viewpoint video upon receiving a notification from the switching determination unit 135. In the third embodiment, the switching processing unit 337 performs an immersion maintenance process, which is a process for maintaining the immersion in the video, when switching from the free viewpoint video to the real-time viewpoint video. The immersion maintenance process is, for example, a process for preventing switching from the free viewpoint video to the real-time viewpoint video in one frame.

[0056] Specifically, the immersion maintaining process may use known techniques such as cross-fade, blackout, white-out, or right-to-left. Cross-fade is a method of performing a weighted averaging of the pre-switching video and the post-switching video, gradually increasing the weight of the post-switching video, and finally switching to the post-switching video. This gradually switches the video, thereby maintaining the immersion. Blackout is a method of displaying a black video between the pre-switching video and the post-switching video for a predetermined time. It is desirable that the predetermined time here is relatively short. This is expected to have the effect of suggesting a change in the worldview.

[0057] Whiteout is a method of displaying a white image between the image before switching and the image after switching for a predetermined time. It is desirable that the predetermined time here is relatively short. This is expected to have the effect of suggesting a change in the worldview. Right-to-left is a method in which the image after switching moves in from the right to the left of the image before switching. This maintains the sense of immersion because the image switches gradually from one direction.

[0058] The video switched as described above is given to the video output unit 138 as an output video.

[0059] As described above, according to the third embodiment, it is possible to maintain the user's immersion in the video when the video is switched.

[0060] In the third embodiment, the switching range may also be superimposed on the free viewpoint video, as shown in the second embodiment.

[0061] Furthermore, in the third embodiment described above, the switching processing unit 337 performs the immersion maintaining process when switching from the free viewpoint video to the real-time viewpoint video, but the third embodiment is not limited to this example. For example, the switching processing unit 337 may also perform the immersion maintaining process when switching from the real-time viewpoint video to the free viewpoint video.

[0062] 1, an information processing system 400 according to the fourth embodiment includes an imaging device 210, a user-side device 120, and an information processing device 430. The user-side device 120 of the information processing system 400 according to the fourth embodiment is similar to the user-side device 120 of the information processing system 100 according to the first embodiment. Moreover, the imaging device 210 of the information processing system 400 according to the fourth embodiment is similar to the imaging device 210 of the information processing system 200 according to the second embodiment.

[0063] As shown in Figure 6, the information processing device 430 in embodiment 4 includes a position identification unit 131, a gaze direction identification unit 132, a 3D data storage unit 333, a free viewpoint video generation unit 334, a switching determination unit 135, a real-time video acquisition unit 136, a switching processing unit 437, a video output unit 138, and a real-time viewpoint video generation unit 239.

[0064] The position identifying unit 131, the gaze direction identifying unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 430 in Embodiment 4 are similar to the position identifying unit 131, the gaze direction identifying unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 130 in Embodiment 1. However, the gaze direction information generated by the gaze direction identifying unit 132 is also provided to the real-time viewpoint video generation unit 239. In addition, the real-time video acquired by the real-time video acquisition unit 136 is provided to the real-time viewpoint video generation unit 239.

[0065] The real-time viewpoint video generation unit 239 of the information processing device 430 in Embodiment 4 is similar to the real-time viewpoint video generation unit 239 of the information processing device 230 in Embodiment 2. Furthermore, the 3D data storage unit 333 and the free viewpoint video generation unit 334 of the information processing device 430 in Embodiment 4 are similar to the 3D data storage unit 333 and the free viewpoint video generation unit 334 of the information processing device 330 in Embodiment 3.

[0066] The switching processing unit 437 switches between the free viewpoint video and the real-time viewpoint video upon receiving the notification from the switching determination unit 135. In the fourth embodiment, the switching processing unit 437 performs immersion maintenance processing when switching from the free viewpoint video to the real-time viewpoint video.

[0067] In the fourth embodiment, the immersion maintaining process is, for example, a process of gradually switching from a feature point of the free viewpoint video to the real-time viewpoint video. Specifically, when the virtual user enters the switching range, the switching processing unit 437 identifies at least one feature point in the free viewpoint video. The feature point may be predetermined, such as an edge of an object included in the free viewpoint video. Then, the switching processing unit 437 gradually switches from the free viewpoint video to the real-time viewpoint video, moving outward from the identified at least one feature point. This gradually switches the video, thereby maintaining the immersion.

[0068] The video switched as described above is given to the video output unit 138 as an output video.

[0069] As described above, according to the fourth embodiment, it is possible to maintain the user's immersion in the video when the video is switched.

[0070] In the fourth embodiment, the switching range may be superimposed on the free viewpoint video, as shown in the second embodiment.

[0071] Furthermore, in the above-described fourth embodiment, the switching processing unit 437 performs the immersion maintaining process when switching from the free viewpoint video to the real-time viewpoint video, but the third embodiment is not limited to this example. For example, the switching processing unit 437 may also perform the immersion maintaining process when switching from the real-time viewpoint video to the free viewpoint video.

[0072] 1, an information processing system 500 according to the fifth embodiment includes an imaging device 210, a user-side device 120, and an information processing device 530. The user-side device 120 of the information processing system 500 according to the fifth embodiment is similar to the user-side device 120 of the information processing system 100 according to the first embodiment. Moreover, the imaging device 210 of the information processing system 500 according to the fifth embodiment is similar to the imaging device 210 of the information processing system 200 according to the second embodiment.

[0073] 7 is a block diagram schematically illustrating the configuration of an information processing device 530 according to Embodiment 5. The information processing device 530 includes a position specifying unit 131, a line of sight direction specifying unit 132, a 3D data storage unit 333, a free viewpoint video generation unit 334, a switching determination unit 135, a real-time video acquisition unit 136, a switching processing unit 537, a video output unit 138, and a real-time viewpoint video generation unit 239.

[0074] The position identifying unit 131, the gaze direction identifying unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 530 in embodiment 5 are similar to the position identifying unit 131, the gaze direction identifying unit 132, the switching determination unit 135, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 130 in embodiment 1. However, the position information generated by the position identifying unit 131 and the gaze direction information generated by the gaze direction identifying unit 132 are also provided to the switching processing unit 537. In addition, the gaze direction information generated by the gaze direction identifying unit 132 is also provided to the real-time viewpoint video generation unit 239. Furthermore, the real-time video acquired by the real-time video acquisition unit 136 is provided to the real-time viewpoint video generation unit 239.

[0075] The real-time viewpoint video generation unit 239 of the information processing device 530 in Embodiment 5 is similar to the real-time viewpoint video generation unit 239 of the information processing device 230 in Embodiment 2. Furthermore, the 3D data storage unit 333 and the free viewpoint video generation unit 334 of the information processing device 530 in Embodiment 5 are similar to the 3D data storage unit 333 and the free viewpoint video generation unit 334 of the information processing device 330 in Embodiment 3.

[0076] The switching processing unit 537 switches between the free viewpoint video and the real-time viewpoint video upon receiving the notification from the switching determination unit 135. In the fifth embodiment, the switching processing unit 537 performs immersion maintenance processing when switching from the free viewpoint video to the real-time viewpoint video.

[0077] In the fifth embodiment, the immersion maintaining process is, for example, a process of gradually switching from the user's viewpoint in the free viewpoint video to the real-time viewpoint video. Specifically, when the virtual user enters the switching range, the switching processing unit 537 identifies the virtual user's viewpoint in the free viewpoint video from the position information from the position identifying unit 131 and the line of sight direction information from the line of sight direction identifying unit 132. Then, the switching processing unit 537 gradually switches from the free viewpoint video to the real-time viewpoint video from the identified viewpoint outward. This gradually switches the video, thereby maintaining the immersion.

[0078] The video switched as described above is given to the video output unit 138 as an output video.

[0079] As described above, according to the fifth embodiment, it is possible to maintain the user's immersion in the video when the video switches.

[0080] In the fifth embodiment, the switching range may also be superimposed on the free viewpoint video, as shown in the second embodiment.

[0081] Furthermore, in the above-described fifth embodiment, the switching processing unit 537 performs the immersion maintaining process when switching from the free viewpoint video to the real-time viewpoint video, but the third embodiment is not limited to this example. For example, the switching processing unit 537 may also perform the immersion maintaining process when switching from the real-time viewpoint video to the free viewpoint video.

[0082] Furthermore, in the second to fifth embodiments, the real-time video acquisition unit 136 receives real-time video, which is a 360-degree video, and the real-time viewpoint video generation unit 239 generates real-time viewpoint video by extracting video of the user's gaze direction from the real-time video. However, the second to fifth embodiments are not limited to this example. For example, the gaze direction identification unit 132 may send gaze direction information to the imaging device 210 via a communication unit (not shown), and the imaging device 210 may capture real-time viewpoint video corresponding to the gaze direction. In this case, the imaging device 210 may transmit the real-time viewpoint video, and in the information processing device 230, a real-time viewpoint video acquisition unit (not shown) may acquire the real-time viewpoint video via a communication unit instead of the real-time video acquisition unit 136. In this case, the real-time viewpoint video generation unit 239 is not required.

[0083] 1 , an information processing system 600 according to the sixth embodiment includes an imaging device 210, a user-side device 120, and an information processing device 630. The user-side device 120 of the information processing system 600 according to the sixth embodiment is similar to the user-side device 120 of the information processing system 100 according to the first embodiment. Moreover, the imaging device 210 of the information processing system 600 according to the sixth embodiment is similar to the imaging device 210 of the information processing system 200 according to the second embodiment.

[0084] 8 is a block diagram schematically illustrating the configuration of an information processing device 630 according to Embodiment 6. The information processing device 630 includes a position specifying unit 131, a line of sight direction specifying unit 132, a 3D data storage unit 333, a free viewpoint video generation unit 334, a switching determination unit 635, a real-time video acquisition unit 136, a video output unit 138, and an object superimposition unit 640.

[0085] The position identifying unit 131, the gaze direction identifying unit 132, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 530 in Embodiment 6 are similar to the position identifying unit 131, the gaze direction identifying unit 132, the real-time video acquisition unit 136, and the video output unit 138 of the information processing device 130 in Embodiment 1. However, the position information generated by the position identifying unit 131 and the gaze direction information generated by the gaze direction identifying unit 132 are also provided to the object superimposition unit 640. In addition, the video output unit 138 acquires the output video from the object superimposition unit 640.

[0086] The 3D data storage unit 333 and the free viewpoint video generation unit 334 of the information processing device 630 in embodiment 6 are similar to the 3D data storage unit 333 and the free viewpoint video generation unit 334 of the information processing device 330 in embodiment 3.

[0087] The switching determination unit 635 determines whether to switch between the free viewpoint video from the free viewpoint video generation unit 334 and the superimposed free viewpoint video, which is a video in which a dynamic object extracted from real-time video captured by the imaging device 110 is superimposed on the free viewpoint video. In the sixth embodiment, the superimposed free viewpoint video serves as the second video. For example, the switching determination unit 635 determines to switch from the free viewpoint video to the superimposed free viewpoint video when the position indicated by the position information from the position identification unit 131 enters a switching range, which is a predetermined range in the virtual space. Furthermore, the switching determination unit 635 determines to switch the video to be output to the user-side device 120 from the superimposed free viewpoint video to the free viewpoint video when the position indicated by the position information leaves the switching range. The object superimposition unit 640 is notified of the above determination.

[0088] The object superimposition unit 640 functions as a video processing unit that superimposes a dynamic object on the free viewpoint video upon receiving a notification from the switching determination unit 135. In other words, the object superimposition unit 640 extracts, as a dynamic object, a moving object from the video being captured in real time by the imaging device 210, and when predetermined conditions are satisfied and the dynamic object is included in the free viewpoint video, generates a superimposed free viewpoint video by superimposing the dynamic object on the free viewpoint video, and switches the video to be output from the free viewpoint video to the superimposed free viewpoint video.

[0089] Specifically, when the virtual user enters the switching range, the object superimposition unit 640 extracts, as a dynamic object, an object in the real-time video that shows a difference between the video at a certain time and the video after a certain time has passed. The method of extracting objects by difference may be a known technique such as a "foreground / background region segmentation algorithm based on a mixed normal distribution (Efficient Adaptive Density Estimation per Image Pixel for the Task of Background Subtraction)" or an "algorithm combining a statistical background estimation method and region segmentation based on Bayesian estimation on a pixel-by-pixel basis (Visual Tracking of Human Visitors under Variable-Lighting Conditions for a Responsive Audio Art Installation)."

[0090] Then, based on the line of sight direction indicated by the line of sight direction information, if the extracted dynamic object is included in the free viewpoint video, the object superimposing section 640 superimposes the dynamic object on the free viewpoint video.

[0091] Furthermore, when the virtual user leaves the switching range, the object superimposition unit 640 stops extracting the dynamic object and stops superimposing the dynamic object. The video switched in this manner is provided to the video output unit 138 as the output video.

[0092] As described above, according to the sixth embodiment, when the video is switched, only a part of the video is different, not the entire video, so that the user's immersion in the video can be maintained.

[0093] In the sixth embodiment, the switching range may also be superimposed on the free viewpoint video, as shown in the second embodiment.

[0094] 100, 200, 300, 400, 500, 600 Information processing system, 110 Imaging device, 120 User side device, 130, 230, 330, 430, 530, 630 Information processing device, 131 Position identification unit, 132 Line of sight direction identification unit, 133, 233, 333 3D data storage unit, 134, 234, 334 Free viewpoint video generation unit, 135 Switching determination unit, 136 Real time video acquisition unit, 137, 237, 337, 437, 537 Switching processing unit, 138 Video output unit, 239 Real time viewpoint video generation unit, 640 Object superimposition unit.

Claims

1. A video output device that outputs video, comprising: a position specifying unit that specifies the position of a virtual user moving in a virtual space; a line-of-sight direction specifying unit that specifies the line-of-sight direction of the virtual user; a first video generation unit that generates a first video, which is the video of the virtual space in the specified line-of-sight direction at the specified position; a determination unit that determines whether a predetermined condition is satisfied when the first video is being output; and a video processing unit that switches the video to be output from the first video to a second video, which is a video reflecting the real-time state, when the predetermined condition is satisfied. A video output device characterized by comprising the above.

2. The video output device according to claim 1, characterized in that the second video is a video being captured in real time by an imaging device.

3. The virtual space is a space that virtually represents the space where the imaging device is installed, and the predetermined condition is that the virtual user enters a predetermined range including the position corresponding to the position where the imaging device is installed in the virtual space. The video output device according to claim 2, characterized by the above.

4. The virtual space is a space that virtually represents the space where an imaging device that captures real-time video, which is real-time video, is installed. The predetermined condition is that the virtual user enters a predetermined range including the position corresponding to the position where the imaging device is installed in the virtual space. The second video is a video in the specified line-of-sight direction extracted from the real-time video. The video output device according to claim 1, characterized by the above.

5. The first video generation unit includes an image indicating the predetermined range in the first video when the predetermined range is included in the first video. The video output device according to claim 4, characterized by the above.

6. The video processing unit performs an immersion maintenance process, which is a process for maintaining the sense of immersion in the first video, when switching from the first video to the second video. The video output device according to claim 4, characterized by the above.

7. The immersion maintenance process is a process for preventing switching from the first video to the second video in one frame. The video output device according to claim 6, characterized by the above.

8. The video processing unit extracts a moving object from the video being captured in real time by the imaging device as a dynamic object, and when the predetermined condition is satisfied and the dynamic object is included in the first video, generates the second video by superimposing the dynamic object on the first video, and switches the output video from the first video to the second video. The video output device according to claim 1, characterized in that.

9. A program for causing a computer to function as a video output device that outputs video, the program causing the computer to function as a position specifying unit that specifies the position of a virtual user moving in a virtual space, a line-of-sight direction specifying unit that specifies the line-of-sight direction of the virtual user, a first video generation unit that generates a first video that is a video of the virtual space in the specified line-of-sight direction at the specified position, a determination unit that determines whether a predetermined condition is satisfied when the first video is being output, and a video processing unit that switches the output video from the first video to a second video that is a video reflecting the real-time state when the predetermined condition is satisfied. A program characterized by that.

10. A video output method for outputting video, the method comprising specifying the position of a virtual user moving in a virtual space, specifying the line-of-sight direction of the virtual user, generating a first video that is a video of the virtual space in the specified line-of-sight direction at the specified position, determining whether a predetermined condition is satisfied when the first video is being output, and switching the output video from the first video to a second video that is a video reflecting the real-time state when the predetermined condition is satisfied. A video output method characterized by that.

Images