Reproduction device, reproduction method, and program

Abstract

The present invention relates to: a reproduction device and a reproduction method, by which a wide-view video and a two-dimensional video constituting a part of the wide-view video can be switched so that these videos can be viewed without causing a sense of strangeness; and a program. In the present invention, a video control unit determines whether the line quality is sufficient to display the wide-view video based on the line speed of the wide-view video provided from a server or the line quality represented by the remaining buffered data amount of the wide-view video and the partial video, and in the case where the line quality is insufficient to display the wide-view video, after the wide-view video accompanied by a white frame indicating the view angle of the partial video corresponding to the wide-view video has been displayed, switches to the partial video at a timing immediately before the display switching between the wide-view video and the partial video is performed. The present invention can be applied to a reproduction device.

Description

Technical Field

[0001] The present disclosure relates to a reproduction device, a reproduction method, and a program, and particularly to a reproduction device, a reproduction method, and a program capable of switching between a wide-viewing angle video and a partial video that is a part of the wide-viewing angle video without a sense of viewing discomfort. Background Art

[0002] There is proposed a technology in which a wide-viewing-angle video such as an entire celestial sphere video is distributed and displayed on a display device such as a head-mounted display (HMD) (see Patent Document 1).

[0003] Citation List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2018-113616 Summary of the invention

[0006] Problems to be solved by the present invention

[0007] By the way, the distribution of wide-angle video enables the distribution of video with a realistic feel. However, depending on the line conditions in a show or live performance, there is a scene for switching between a wide-angle video and a partial video (hereinafter also referred to as a flat video) that is a part of the wide-angle video.

[0008] In the technology according to Patent Document 1, when switching from a wide-angle video to a flat video, it is not clearly displayed to which part of the wide-angle video the flat video is switched.

[0009] For this reason, when the wide-view video is switched to the flat video, the user as the viewer cannot recognize from which part of the wide-view video that the user has been watching until then the flat video displayed after the switch is obtained. Therefore, in some cases, there is a sense of incongruity when watching.

[0010] The present disclosure has been made in view of such circumstances, and in particular, has an object to enable viewing without a sense of incongruity when switching between a wide-viewing angle video and a flat video which is a part of the wide-viewing angle video.

[0011] Solution to the problem

[0012] According to one aspect of the present disclosure, a reproduction device includes: a switching determination unit, which is configured to determine whether a switching condition for switching display between a wide-angle video and a partial video corresponding to a partial angle of view of the wide-angle video is satisfied; and a video controller, which is configured to selectively display the wide-angle video and the partial video based on a determination result of the switching determination unit, wherein, when the display is switched between the wide-angle video and the partial video based on the determination result of the switching determination unit, the video controller displays the wide-angle video to which display information indicating the angle of view of the partial video corresponding to the wide-angle video has been added.

[0013] A reproducing method according to one aspect of the present disclosure corresponds to a reproducing device.

[0014] According to one aspect of the present disclosure, whether a switching condition for switching display between a wide-angle video and a partial video corresponding to a partial angle of view of the wide-angle video is satisfied, the wide-angle video and the partial video are selectively displayed based on a determination result, and when the display is switched between the wide-angle video and the partial video based on the determination result, the wide-angle video to which display information indicating the angle of view of the partial video corresponding to the wide-angle video has been added is displayed. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a diagram illustrating an overview of the present disclosure.

[0016] Figure 2 is a diagram showing a configuration example of a reproduction system in the first embodiment of the present disclosure.

[0017] Figure 3 It is shown by Figure 2 Flow chart of the entire celestial sphere video capture process performed by the entire celestial sphere camera device.

[0018] Figure 4 It is shown by Figure 2 Flow chart of a planar video capture process performed by a planar camera device.

[0019] Figure 5 It is shown by Figure 2 Flowchart of metadata generation processing performed by a PC.

[0020] Figure 6 It is shown by Figure 2 Flowchart of video transmission processing performed by a server.

[0021] Figure 7 It is shown by Figure 2 Flowchart of the reproduction processing performed by the reproduction device.

[0022] Figure 8 It is shown by Figure 2Flowchart of display processing performed by the HMD.

[0023] Fig. 9 : is a diagram showing an application example of the reproduction system in the first embodiment of the present disclosure.

[0024] Fig.10 It is shown by Fig. 9 Flowchart of metadata generation processing performed by a PC.

[0025] Fig.11 It is shown by Fig. 9 Flowchart of video transmission processing performed by a server.

[0026] Fig.12 is a diagram showing a configuration example of a reproduction system in a second embodiment of the present disclosure.

[0027] Fig.13 It is shown by Fig.12 Flow chart of the process of generating a framed video of the entire celestial sphere performed by a PC.

[0028] Fig.14 It is shown by Fig.12 Flowchart of video transmission processing performed by a server.

[0029] Fig.15 It is shown by Fig.12 Flowchart of the reproduction processing performed by the reproduction device.

[0030] Fig.16 It is shown by Fig.12 Flowchart of display processing performed by the HMD.

[0031] Fig.17 is a diagram showing a configuration example of a general-purpose personal computer. DETAILED DESCRIPTION

[0032] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in the present specification and the accompanying drawings, components having substantially the same functional configuration are denoted by the same reference numerals to omit repeated descriptions.

[0033] Hereinafter, a method for implementing the present technology will be described. The description will be given in the following order:

[0034] 1. Overview of the Disclosure

[0035] 2. First Implementation

[0036] 3. Application Examples of the First Embodiment

[0037] 4. Second Implementation

[0038] 5. Examples executed by software

[0039] <<1. Overview of the present disclosure>>

[0040] The present disclosure aims to switch between a wide-view video and a flat video that is a part of the wide-view video without a sense of viewing discomfort.

[0041] Therefore, first, an overview of the present disclosure will be described.

[0042] The user wears a head mounted display (HMD) on the head, and a wide viewing angle video is displayed according to the movement of the user's head. Consider a configuration in which, for example, Figure 1 As shown in the left part of FIG. 1 , a wide-viewing angle video P1 including a broadcast video of a football match is displayed on the display unit D of the HMD.

[0043] In such a configuration, in some cases, Figure 1 The wide-view video P1 on the left side switches to Figure 1 The planar video P2 shown in the right part of FIG. 1 is used to enlarge and display an area F′ in which two players including a player holding the ball and a player of the opponent team who is marking the player exist in the wide-view video P1.

[0044] It should be noted that the wide-angle video P1 is a video that has been captured by a first camera device for capturing wide-angle video, while the planar video P2 is a video that has a field of view within the wide-angle video P1 and has been captured by a second camera device, which is different from the first camera device and captures a video with a field of view narrower than that of video P1.

[0045] However, at a predetermined timing, Figure 1 The wide viewing angle video P1 shown in the left part of the image is suddenly switched to Figure 1 In the case of the plane video P2 shown in the right part of , the user who is watching the video displayed on the display unit D can recognize that the video the user is watching has been switched.

[0046] However, the user may not be able to recognize what positional relationship the planar video P2 to be displayed after the switch has with respect to the immediately preceding wide-viewing-angle video P1 , and may feel a sense of discomfort.

[0047] Therefore, in this disclosure, Figure 1 As shown in the central part of the image, the area corresponding to the area F' is displayed as a frame F in the wide-view video P1, and the area F' becomes the planar video P2 to be switched next.

[0048] Then, in the present disclosure, it is shown as Figure 1Add box F to the center of Figure 1 The wide-view video P1 on the left side of the wide-view video P1, and then switch it to Figure 1 The plane video P2 is shown in the right part.

[0049] In this way, at the timing immediately before the wide-view video P1 is switched to the plane video P2 to be displayed, the position corresponding to the plane video P2 is displayed as a frame F in the wide-view video P1. This configuration allows the user to recognize the video at which position in the wide-view video P1 immediately before the switching the plane video P2 to be switched to at the next timing corresponds, and then switch the video.

[0050] Therefore, the user watching the display unit D recognizes the frame F displayed in the wide-view video P1 so that he can watch the planar video P2 after switching, and recognizes the positional relationship between the planar video P2 to be switched next and the wide-view video P1 immediately before the switch.

[0051] As a result, when the wide-viewing-angle video P1 is switched to the flat video P2, the user can continue watching without feeling uncomfortable.

[0052] <<2. First embodiment>>

[0053] Next, we will refer to Figure 2 The block diagram describes a configuration example of the reproduction system in the first embodiment of the present disclosure.

[0054] When switching between a wide-angle video and a flat video that is a partial video in the wide-angle video, the reproduction system according to the present disclosure displays the position of the flat video to be switched to next as a box in the wide-angle video, and then switches to the flat video, so as to switch between videos without a sense of disharmony.

[0055] Figure 2 The reproduction system 1 includes an entire celestial sphere imaging device 11, a planar imaging device 12, a personal computer (PC) 13, a server 14, a reproduction device 15, a head mounted display (HMD) 16, and encoding devices 21 and 22.

[0056] Figure 2 The reproduction system 1 is a system that relays and distributes content such as sports, drama, etc. from a venue, and the entire spherical camera 11, flat camera 12, and personal computer (PC) 13 are configured as venue facilities of sports facilities, theaters, etc.

[0057] Furthermore, the reproduction device 15 and the HMD 16 are configured as facilities on the user side that receives content distribution.

[0058] Then, the server 14 acquires the video as content from the venue facility via the network, and distributes the video to the facility on the user side.

[0059] The entire sphere camera 11 is a camera installed in a fixed state, and captures an entire sphere video including a wide-angle video having a range of 180 degrees, a range of 360 degrees, etc. with respect to the horizontal direction and the vertical direction, and outputs the video to the encoding device 21. The encoding device 21 encodes the entire sphere video, and outputs the encoded entire sphere video to the server 14. It is to be noted that the entire sphere camera 11 may have a function of the encoding device 21 to encode the captured entire sphere video, and may output the encoded entire sphere video to the server 14. In addition, the entire sphere video captured by the entire sphere camera 11 is encoded by the encoding device 21, and then output to the server 14. However, in the following description, the process of encoding the entire sphere video by the encoding device 21 is omitted as needed, and, for example, the process is described as "the entire sphere camera 11 outputs the captured entire sphere video to the server 14". However, this expression is merely omitted, and actually, the entire celestial sphere video that has been captured by the entire celestial sphere camera 11 is encoded by the encoding device 21 and then output to the server 14 .

[0060] The planar camera 12 is a camera set on a camera platform 12a, and while changing the video capture position and pan-tilt using the camera platform 12a, captures a partial range in the range captured by the entire spherical camera 11 as a planar video, for example, at a horizontal-vertical ratio of 16:9, and outputs the planar video to the encoding device 22. The encoding device 22 encodes the entire spherical video, and outputs the entire spherical video whose video has been encoded to the server 14. It should be noted that the planar camera 12 may have a function of the encoding device 22 to encode the captured planar video, and may output the encoded planar video to the server 14. Further, the planar video captured by the planar camera 12 is encoded by the encoding device 22, and then output to the server 14. However, in the following description, the process of encoding the planar video by the encoding device 22 is omitted as needed, and the process will be simply described as "the planar camera 12 outputs the captured planar video to the server 14". However, this expression is merely omitted, and actually, the planar video that has been captured by the planar camera 12 is encoded by the encoding device 22 and then output to the server 14 .

[0061] In addition, in the present disclosure, a description will be given about an example in which an entire celestial sphere video and a planar video including a partial video of the entire celestial sphere video are captured by the entire celestial sphere camera 11 and the planar camera 12, respectively. However, a configuration other than the entire celestial sphere camera 11 and the planar camera 12 may be used as long as such a configuration is capable of capturing a wide-angle video and a planar video including a partial video that is a part of the entire celestial sphere video.

[0062] The camera platform 12 a outputs information indicating the video capturing position and pan-tilt directions of the planar camera 12 to the PC 13 as pan-tilt information.

[0063] The PC 13 acquires the pan-tilt information supplied from the camera platform 12 a of the planar camera 12 , generates metadata based on the pan-tilt information, and outputs the metadata to the server 14 .

[0064] More specifically, the PC 13 includes a controller 31 , a data controller 32 , and a delay processing unit 33 .

[0065] The controller 31 includes a processor, a memory, and the like, and controls the overall operation of the PC 13 , and controls the data controller 32 and the delay processing unit 33 .

[0066] The data controller 32 generates metadata including the pan-tilt information based on the pan-tilt information supplied from the camera platform 12 a , and outputs the metadata to the delay processing unit 33 .

[0067] The delay processing unit 33 temporarily stores the metadata provided from the data controller 32 , delays for a predetermined period of time to adjust the timing to match the timing when the planar video captured by the planar camera 12 is output to the server 14 , and outputs the metadata to the server 14 .

[0068] That is, the planar video output from the planar camera 12 to the server 14 and the metadata including the pan-tilt information of the planar camera 12 that is capturing the planar video are provided to the server 14 at the same timing.

[0069] The server 14 receives and stores the entire sphere video supplied from the entire sphere camera 11 , the planar video supplied from the planar camera 12 , and the metadata supplied from the PC 13 , and furthermore, outputs the entire sphere video, the planar video, and the metadata to the reproduction device 15 .

[0070] More specifically, the server 14 includes a controller 51 , a metadata storage unit 52 , a plane video storage unit 53 , and a whole celestial sphere video storage unit 54 .

[0071] The controller 51 includes a processor, a memory, etc., and controls the overall operation of the server 14 .

[0072] The controller 51 controls the operations of the metadata storage unit 52 , the plane video storage unit 53 , and the entire celestial sphere video storage unit 54 .

[0073] When metadata is supplied from the PC 13, the controller 51 acquires the metadata and causes the metadata storage unit 52 to store the metadata. Furthermore, the controller 51 reads the metadata stored in the metadata storage unit 52 and transmits the metadata to the reproducing device 15.

[0074] When the planar video is supplied from the planar camera 12, the controller 51 acquires the planar video and causes the planar video storage unit 53 to store the planar video. Furthermore, the controller 51 reads the planar video stored in the planar video storage unit 53 and transmits the planar video to the reproduction device 15.

[0075] When the entire sphere video is provided from the entire sphere camera 11, the controller 51 acquires the entire sphere video and causes the entire sphere video storage unit 54 to store the entire sphere video. In addition, the controller 51 reads the entire sphere video stored in the entire sphere video storage unit 54 and sends the entire sphere video to the reproduction device 15.

[0076] The reproduction device 15 acquires the entire celestial sphere video, the planar video, and the metadata provided from the server 14 , performs rendering based on the head tracking information from the HMD 16 worn by the user, and outputs the video to the HMD 16 for display.

[0077] The playback device 15 causes the HMD 16 to display the entire spherical video when the line quality is good, and causes the HMD 16 to display the flat video according to the line quality of the communication line with the server 14 when the line quality is deteriorated.

[0078] In this case, when switching from the entire celestial sphere video to the plane video, the reproduction device 15 displays the range of the plane video to be switched to next as a box in the entire celestial sphere video to show the position of the plane video to be switched to next in the entire celestial sphere video, and then switches to the plane video and displays the plane video.

[0079] By switching from the entire sphere video to the flat video in this manner, the user can watch the video switched from the entire sphere video to the flat video without feeling any discomfort.

[0080] It is to be noted that when the planar video is switched to the entire celestial sphere video, a frame indicating the range of the planar video is displayed in the entire celestial sphere video for a predetermined period at the timing of switching to the entire celestial sphere video.

[0081] By switching from a flat video to an entire celestial sphere video in this manner, the user can watch the video switched from a flat video to an entire celestial sphere video without feeling a sense of discomfort.

[0082] More specifically, the reproducing device 15 includes a connection controller 71 , a switching determination unit 72 , a decoding unit 73 , a meta-parser unit 74 , a video controller 75 , a user interface processing unit 76 , and a rendering unit 77 .

[0083] The connection controller 71 acquires the entire celestial sphere video and the planar video from the server 14 , and outputs the videos to the decoding unit 73 .

[0084] In this case, the connection controller 71 outputs information indicating the line quality of the line used to obtain the entire celestial video from the server 14 to the switching determination unit 72, for example, information about the line speed, information about the remaining buffered data amount of the entire celestial video and planar video being received, etc.

[0085] Based on the information indicating the line quality supplied from the connection controller 71, in a case where, for example, the line quality such as the line speed is faster than a predetermined line speed and better than a predetermined quality, the switching determination unit 72 supplies the video controller 75 with switching determination information for instructing to display the entire spherical video on the HMD 16. Furthermore, in a case where the information indicating the line quality is information on the remaining buffered data amount of the entire spherical video and the planar video being received, when the remaining buffered data amount is equal to or greater than a predetermined threshold, the switching determination unit 72 may determine that the line quality is better than the predetermined quality, and may instruct to display the entire spherical video on the HMD 16.

[0086] Furthermore, in the case where the line quality is not better than a predetermined quality, the switching determination unit 72 supplies the video controller 75 with switching determination information for instructing to display the plane video on the HMD 16 .

[0087] The decoding unit 73 decodes the entire celestial sphere video and the planar video, and outputs the video to the video controller 75 .

[0088] The meta-parser unit 74 acquires metadata supplied from the server 14 , and outputs the metadata to the video controller 75 .

[0089] The user interface processing unit 76 acquires the head tracking information supplied from the HMD 16 , and outputs the head tracking information to the video controller 75 .

[0090] The head tracking information is, for example, information about the position and direction of the HMD 16 worn on the user's head, and is information indicating the position and direction that the user is facing.

[0091] The video controller 75 determines the video based on the switching determination information provided from the switching determination unit 72, the entire celestial sphere video and the plane video provided from the decoding unit 73, the metadata provided from the meta-parser unit 74, and the head tracking information provided from the user interface processing unit 76, and outputs the determined video to the rendering unit 77.

[0092] The rendering unit 77 renders the video supplied from the video controller 75 , and outputs the rendered video to the HMD 16 for display.

[0093] The HMD 16 is a display device that is worn on the user's head, and displays the video supplied from the reproduction device 15 .

[0094] More specifically, the HMD 16 includes a controller 81 , a head tracking information acquisition unit 82 , and a display unit 83 .

[0095] The controller 81 includes a processor and a memory, and controls the overall operation of the HMD 16 .

[0096] The head tracking information acquisition unit 82 includes, for example, a motion sensor or the like, acquires information on the position, direction, and the like of the user's head as head tracking information, and transmits the head tracking information to the reproduction device 15 .

[0097] The display unit 83 includes a liquid crystal display (LCD) and an organic electroluminescence (EL), is provided in a viewing direction of a user wearing the HMD 16 , and displays a video supplied from the reproduction device 15 .

[0098] <Entire celestial sphere video capture processing>

[0099] Next, we will refer to Figure 3 The flowchart of FIG. 1 describes the entire celestial sphere video capturing process performed by the entire celestial sphere camera device 11 .

[0100] In step S11 , the entire celestial sphere camera device 11 captures an entire celestial sphere video, and outputs the entire celestial sphere video to the encoding device 21 .

[0101] In step S12 , the encoding device 21 encodes the entire celestial sphere video provided from the entire celestial sphere camera 11 , and transmits the encoded entire celestial sphere video to the server 14 .

[0102] In step S13, the entire celestial sphere imaging device 11 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S11, and the subsequent processes are repeated.

[0103] Then, in step S13 , when the end of the process is instructed, the process ends.

[0104] Through the above processing, the entire celestial sphere video is captured by the entire celestial sphere camera device 11 , encoded by the encoding device 21 , and continuously sent to the server 14 .

[0105] <Plane Video Capture Processing>

[0106] Next, we will refer to Figure 4 The planar video capturing process performed by the planar camera device 12 is described with reference to a flowchart of FIG.

[0107] In step S31 , the planar camera 12 captures a planar video including a partial video of the entire celestial sphere video captured by the entire celestial sphere camera 11 , and outputs the planar video to the encoding device 22 .

[0108] In step S32 , the camera platform 12 a of the planar camera 12 acquires pan-tilt information including information on the pan-tilt as the video capturing direction of the planar camera 12 and the video capturing position.

[0109] In step S33 , the encoding device 22 encodes the captured planar video and outputs the encoded planar video to the server 14 .

[0110] In step S34 , the camera platform 12 a outputs the pan-tilt information to the PC 13 .

[0111] In step S35, the planar imaging device 12 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S31, and the subsequent processes are repeated.

[0112] Then, in step S35, when the end of the process has been instructed, the process ends.

[0113] Through the above processing, the planar video is captured by the planar camera device 12, encoded by the encoding device 22, and sent to the server 14, and in addition, the pan-tilt information including the video capture position of the planar camera device 12 at the timing when the planar video is captured and the information about the pan-tilt is output to the PC 13.

[0114] <By Figure 2 Metadata generation processing by PC in>

[0115] Next, we will refer to Figure 5 The flowchart described by Figure 2 Metadata generation processing performed by the PC 13 in .

[0116] In step S51 , the data controller 32 acquires pan-tilt information supplied from the camera platform 12 a of the planar camera 12 .

[0117] In step S52 , the data controller 32 generates metadata based on the acquired pan-tilt information, and outputs the metadata to the delay processing unit 33 .

[0118] In step S53 , the delay processing unit 33 acquires metadata supplied from the data controller 32 , stores the metadata until the timing of transmitting the corresponding planar video in the planar camera 12 to the server 14 , and delays the output.

[0119] In step S54 , the delay processing unit 33 outputs the metadata to the server 14 at the timing of transmitting the plane video corresponding to the pan-tilt information included in the stored metadata to the server 14 .

[0120] In step S55, the controller 31 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S51, and the subsequent processes are repeated.

[0121] Then, in step S55, when the end of the process has been instructed, the process ends.

[0122] Through the above processing, metadata based on the pan-tilt information including the video capture position at the timing when the planar video is captured by the planar camera device 12 and information about the pan-tilt is generated, and in addition, the metadata is output to the server 14 at the timing when the corresponding planar video is sent to the server 14.

[0123] As a result, the planar video and metadata including the corresponding pan-tilt information are adjusted at the same timing to be transmitted to the server 14 .

[0124] <By Figure 2 Video transmission processing by the server in>

[0125] Next, we will refer to Figure 6 The flowchart described by Figure 2 The server 14 performs the video transmission processing.

[0126] In step S71 , the controller 51 of the server 14 acquires the entire celestial sphere video that has been captured by the entire celestial sphere camera 11 and encoded by the encoding device 21 , and causes the entire celestial sphere video storage unit 54 to store the entire celestial sphere video.

[0127] In step S72 , the controller 51 reads the entire celestial sphere video stored in the entire celestial sphere video storage unit 54 , and outputs the entire celestial sphere video to the reproduction device 15 .

[0128] In step S73 , the controller 51 acquires the planar video that has been captured by the planar camera 12 and encoded by the encoding device 22 , and causes the planar video storage unit 53 to store the planar video.

[0129] In step S74 , the controller 51 reads the plane video stored in the plane video storage unit 53 , and outputs the plane video to the reproduction device 15 .

[0130] In step S75 , the controller 51 acquires metadata supplied from the PC 13 , and causes the metadata storage unit 52 to store the metadata.

[0131] In step S76 , the controller 51 reads the metadata stored in the metadata storage unit 52 , and outputs the metadata to the reproducing device 15 .

[0132] In step S77, the controller 51 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S71, and the subsequent processes are repeated.

[0133] Then, in step S77, when the end of the process is instructed, the process ends.

[0134] Through the above processing, the entire sphere video captured by the entire sphere camera 11 and encoded by the encoding device 21, the planar video captured by the planar camera 12 and encoded by the encoding device 22, and the metadata generated by the PC 13 are respectively sent to the reproduction device 15.

[0135] In this case, regarding the entire celestial sphere video, the plane video, and the metadata, corresponding information at the same timing is respectively provided to the reproduction device 15 .

[0136] therefore, Figure 6 The processing of steps S71 and S72 , the processing of steps S73 and S74 , and the processing of steps S75 and S76 in the flowchart are basically parallel processing.

[0137] <By Figure 2 Reproduction processing by the reproduction device in>

[0138] Next, we will refer to Figure 7 The flowchart described by Figure 2 The reproduction process is performed by the reproduction device 15.

[0139] In step S111 , the connection controller 71 acquires the encoded entire celestial sphere video and planar video from the server 14 , and outputs the videos to the decoding unit 73 .

[0140] In step S112, the decoding unit 73 decodes the entire sphere video and the planar video supplied from the connection controller 71 and encoded, and outputs the decoded video to the video controller 75. That is, the video controller 75 acquires the decoded entire sphere video and the planar video.

[0141] In step S113, the meta-parser unit 74 acquires the metadata supplied from the server 14, and outputs the metadata to the video controller 75. That is, the video controller 75 acquires the metadata.

[0142] In step S114 , the video controller 75 acquires head tracking information from the HMD 16 .

[0143] In step S115, the connection controller 71 outputs information about the line speed and the remaining buffered data amount of, for example, the entire spherical video and the planar video being received, as information about the line quality of the entire spherical video from the server 14, to the switching determination unit 72. That is, the switching determination unit 72 acquires information about the line speed and the remaining buffered data amount of the video as information about the line quality.

[0144] In step S116, the switching determination unit 72 determines whether the line speed is faster than a predetermined speed and whether the remaining buffered data amount is equal to or greater than a predetermined threshold based on information about the line quality including the line speed or the remaining buffered data amount of the entire celestial sphere video and the planar video being received, so as to determine whether the line quality is sufficient to reproduce the entire celestial sphere video, that is, whether to reproduce the entire celestial sphere video.

[0145] In the case where it is determined in step S116 that the line quality is sufficient to reproduce the entire celestial sphere video, the switching determination unit 72 outputs switching determination information for instructing reproduction of the entire celestial sphere video to the video controller 75. The process proceeds to step S117.

[0146] In step S117 , the video controller 75 selects the decoded entire sphere video based on the switching determination information indicating the reproduction of the entire sphere video and the head tracking information, and outputs the selected entire sphere video to the rendering unit 77 .

[0147] In step S118 , the rendering unit 77 renders the entire celestial sphere video supplied from the reproduction device 15 , and outputs the rendered entire celestial sphere video to the HMD 16 for display.

[0148] In step S119, the video controller 75 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S111, and the subsequent process is repeated.

[0149] Then, in step S119, in the case where it is determined that the end has been instructed, the processing ends.

[0150] On the other hand, in the event that it is determined in step S116 that the line quality is insufficient to reproduce the entire spherical video, the switching determination unit 72 outputs switching determination information for instructing reproduction of the planar video to the video controller 75 , and the process proceeds to step S120 .

[0151] In step S120, the video controller 75 reads the pan and tilt information indicating the video capture position and direction of the planar camera 12 included in the metadata based on the switching determination information, identifies the position of the planar video in the entire celestial video, and sets the display position and size of the frame indicating the corresponding position in the entire celestial video.

[0152] In step S121, the video controller 75 generates an entire sphere video to which a frame indicating the position and size corresponding to the planar video is added based on the head tracking information, the decoded entire sphere video, and information about the position and size of the frame, and outputs the entire sphere video with the frame to the rendering unit 77.

[0153] In step S122 , the rendering unit 77 renders the entire celestial sphere video with a frame indicating the position and size corresponding to the plane video, and outputs the entire celestial sphere video with the frame to the HMD 16 for display.

[0154] In step S123 , the video controller 75 generates a transition video for transitioning from the entire celestial sphere video to the plane video.

[0155] The transition video for transitioning from the entire sphere video to the plane video is, for example, a video in the process of gradually enlarging a region in the plane video displayed as a frame in the entire sphere video and transitioning to the plane video. The entire sphere video and the plane video are used for interpolation generation, for example.

[0156] In step S124 , the rendering unit 77 renders a transition video for transitioning from the entire celestial sphere video to the plane video, and outputs the transition video to the HMD 16 for display.

[0157] In step S125 , the video controller 75 outputs the decoded plane video to the rendering unit 77 .

[0158] In step S126 , the rendering unit 77 renders the planar video, and outputs the planar video to the HMD 16 for display.

[0159] That is, through the processing of steps S120 to S126, in the HMD 16, after adding a frame indicating the corresponding position of the planar video and displaying the entire spherical video, a transition video for transitioning to the planar video is displayed, and then the display is switched to the planar video.

[0160] In step S127 , the connection controller 71 acquires the entire celestial sphere video and the planar video from the server 14 , and outputs the videos to the decoding unit 73 .

[0161] In step S128, the decoding unit 73 decodes the entire sphere video and the planar video supplied from the connection controller 71, and outputs the decoded entire sphere video and the planar video to the video controller 75. That is, the video controller 75 acquires the decoded entire sphere video and the planar video.

[0162] In step S129, the meta-parser unit 74 acquires the metadata supplied from the server 14, and outputs the metadata to the video controller 75. That is, the video controller 75 acquires the metadata.

[0163] In step S130 , the video controller 75 acquires head tracking information from the HMD 16 .

[0164] In step S131, the connection controller 71 outputs information about the line speed or the remaining buffered data amount of, for example, the entire spherical video and the planar video being received, as information about the line quality of the entire spherical video from the server 14, to the switching determination unit 72. That is, the connection controller 71 acquires information about the line speed or the remaining buffered data amount of the video as information about the line quality.

[0165] In step S132, the switching determination unit 72 determines whether the line quality is sufficient to reproduce the entire celestial sphere video, that is, whether to reproduce the entire celestial sphere video, based on information about the line quality including the line speed and the remaining buffered data amount of the entire celestial sphere video and planar video being received.

[0166] In the event that it is determined in step S132 that the line quality is insufficient to reproduce the entire spherical video, the switching determination unit 72 outputs switching determination information for instructing reproduction of the planar video to the video controller 75. The process proceeds to step S133.

[0167] In step S133, the video controller 75 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S125, and the subsequent process is repeated.

[0168] Then, in step S133, in the case where it is determined that the end has been instructed, the processing ends.

[0169] That is, in the case where a state in which sufficient line quality for reproducing the entire celestial sphere video cannot be obtained continues, the processing of steps S125 to S133 is repeated, and the plane video is continuously rendered and displayed on the HMD 16 .

[0170] Furthermore, in step S132, in the event that the line quality is determined to be sufficient to reproduce the entire celestial sphere video, the switching determination unit 72 outputs switching determination information for instructing reproduction of the entire celestial sphere video to the video controller 75. The process proceeds to step S134.

[0171] In step S134 , the video controller 75 generates a transition video for transitioning from the planar video to the entire celestial sphere video, and outputs the generated transition video to the rendering unit 77 .

[0172] The transition video used to transition from the planar video to the entire sphere video is, for example, a video in the following process: the planar video is gradually reduced to be embedded in the entire sphere video at a corresponding position and size, and transitioned to the entire sphere video. The entire sphere video and the planar video are used for interpolation generation, for example.

[0173] In step S135 , the rendering unit 77 outputs the transition video to the HMD 16 for display.

[0174] In step S136, the video controller 75 reads the pan and tilt information indicating the video capture position and direction of the planar camera 12 included in the metadata, identifies the position of the planar video in the entire celestial video, and sets the display position and size of the box indicating the corresponding position in the entire celestial video.

[0175] In step S137, the video controller 75 generates an entire sphere video to which a frame indicating the position and size corresponding to the planar video is added based on the head tracking information, the decoded entire sphere video, and information about the position and size of the frame, and outputs the entire sphere video with the frame to the rendering unit 77.

[0176] In step S138, the rendering unit 77 outputs the entire celestial sphere video to which the frame has been added to the HMD 16 for display. The process returns to step S117.

[0177] That is, through the reverse processing of the case where the entire celestial sphere video transitions to the plane video, similarly when the plane video transitions to the entire celestial sphere video, the plane video transitions to the entire celestial sphere video, and then a frame corresponding to the plane video is temporarily displayed in the entire celestial sphere video, so that it is possible to identify at which position in the entire celestial sphere video the plane video has been displayed.

[0178] Note that, for example, in the above, the position and size corresponding to the planar video to be switched to next in the entire spherical video have been determined by including Figure 1 A description is given of an example of the display information representation of the frame F in FIG. However, display information other than the frame may also be applicable as long as the position and size corresponding to the plane video to be switched to next in the entire spherical video can be represented. For example, the corresponding area may be colored semi-transparently, or only the corresponding area may be displayed as a black and white video.

[0179] In addition, depending on the direction in which the user wears the HMD 16, there is no area of ​​the planar video to be switched to next in the entire spherical video in the user's field of view. Therefore, there is a possibility that a frame indicating the position and size corresponding to the planar video to be switched to next may not be displayed.

[0180] In this case, for example, switching from the entire spherical video to the planar video may be waited until a frame indicating the position and size corresponding to the planar video to be switched to next may be displayed in the user's field of view, and then the switching may be performed.

[0181] Alternatively, for example, when switching from the entire spherical video to the flat video, a video in which the entire spherical video with the added frame can be viewed is temporarily generated, and the video is displayed as a transition video. After forcibly making the user watch the video with the added frame, the switch can be made by transitioning to the flat video.

[0182] In addition, in the above, description has been given about an example in which the line speed as the line quality of the entire spherical video or the remaining buffer data amount of the entire spherical video being received and the plane video is used as a reference for switching between the display of the entire spherical video and the display of the plane video. However, switching may be performed using another reference, for example, switching may be performed in response to a user's instruction, or switching may be performed when a predetermined specific scene is displayed according to the presentation of the content, etc.

[0183] <By Figure 2 Display processing by the HMD in>

[0184] Next, we will refer to Figure 8 The flowchart described by Figure 2 The HMD 16 performs display processing.

[0185] In step S151 , the head tracking information acquisition unit 82 of the HMD 16 acquires the head tracking information of the user.

[0186] In step S152 , the head tracking information acquiring unit 82 outputs the acquired head tracking information to the reproducing device 15 .

[0187] In step S153 , the controller 81 acquires the entire celestial sphere video, the plane video, or the transition video supplied from the reproduction device 15 , and causes the display unit 83 to display the video.

[0188] In step S154, the controller 81 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S151 and the subsequent processes are repeated.

[0189] Then, in step S154, in the case where it is determined that the end has been instructed, the processing ends.

[0190] Through the above series of processing, head tracking information is acquired and continuously output to the reproduction device 15, and in addition, the entire celestial sphere video, plane video or transition video provided from the reproduction device 15 is acquired and continuously displayed on the display unit 83.

[0191] Therefore, the entire spherical video or the plane video is switched and displayed according to the line quality. When switching, a frame located at the position of the corresponding plane video and having the size of the corresponding plane video is displayed in the entire spherical video. In addition, a transition video is displayed, and then the transition video is switched to the plane video.

[0192] Therefore, even when the entire spherical video is switched to the planar video, the user can recognize at which position in the entire spherical video the planar video has been switched.

[0193] Furthermore, when the planar video is switched to the entire sphere video, a transition video is displayed, and then the entire sphere video with the frame added is displayed. This configuration enables recognition of the position in the entire sphere video to which the planar video was switched from.

[0194] As a result, it is possible to switch between the wide-viewing angle video and the flat video which is a part of the wide-viewing angle video without a sense of discomfort in viewing.

[0195] <<3. Application examples of the first embodiment>>

[0196] In the above, description has been given about an example in which the delay processing unit 33 of the PC 13 causes the planar camera 12 to capture metadata including pan-tilt information and a planar video encoded and output by the encoding device 21 and the timing is adjusted. However, the server 14 may generate time information indicating the timing of the planar video, may include the information in the metadata, and may output the metadata to the reproduction device 15.

[0197] With such a configuration, in the reproducing device 15, when a plane video is selected, the corresponding plane video and metadata can be used according to the time information included in the metadata.

[0198] Fig. 9 There is shown a configuration example of the reproduction system 1 in which the server 14 generates time information indicating the timing of a plane video, includes the time information in metadata, and outputs the metadata to the reproduction device 15 .

[0199] exist Fig. 9 In the reproduction system 1, the delay processing unit 33 is omitted in the PC 13, and further, in the server 14, instead of the metadata storage unit 52, a reception metadata storage unit 91, a video monitoring unit 92, a time information generation unit 93, and a transmission metadata storage unit 94 are provided.

[0200] That is, in Fig. 9 In the PC 13, the generated metadata is output to the server 14 without delay adjustment. Therefore, in the PC 13, the delay processing unit 33 and the like can be omitted.

[0201] Furthermore, the reception metadata storage unit 91 is controlled by the controller 51 , and stores metadata supplied from the PC 13 as reception metadata.

[0202] The video monitoring unit 92 monitors the timing at which the plane video is stored in the plane video storage unit 53 , and outputs a signal indicating the timing when the plane video is stored to the time information generating unit 93 .

[0203] The time information generation unit 93 generates time information based on a signal indicating the timing when the plane video provided from the video monitoring unit 92 is stored in the plane video storage unit 53, reads the receiving metadata stored in the receiving metadata storage unit 91, adds the generated time information, and causes the sending metadata storage unit 94 to store it as sending metadata.

[0204] Then, the controller 51 reads the transmission metadata stored in the transmission metadata storage unit 94 , and outputs it to the reproduction device 15 as metadata.

[0205] In the reproduction device 15, in the case where the planar video is selected based on the switching determination information, the pan-tilt information corresponding to the planar video can be used by selecting metadata including the corresponding time information.

[0206] <By Fig. 9 Metadata generation processing by PC in>

[0207] Next, we will refer to Fig.10 The flowchart described by Fig. 9 The metadata generation process is performed by the PC 13.

[0208] Notice, Fig.10 The processing of steps S171 to S174 in the flowchart of Figure 5 The processing of steps S51, S52, S54 and S55 is similar. Therefore, the description thereof will be omitted.

[0209] That is, in Fig.10 In the metadata generation process of Figure 5 Delay processing of metadata in step S53 in the metadata generation process.

[0210] As a result, the delay processing by the delay processing unit 33 can be omitted.

[0211] <By Fig. 9 Video transmission processing by the server in>

[0212] Next, we will refer to Fig.11 The flowchart described by Fig. 9 The server 14 performs the video transmission processing.

[0213] In step S191 , the controller 51 of the server 14 acquires the entire celestial sphere video that has been captured by the entire celestial sphere camera 11 and has been encoded and provided by the encoding device 21 , and causes the entire celestial sphere video storage unit 54 to store the entire celestial sphere video.

[0214] In step S192 , the controller 51 reads the entire celestial sphere video stored in the entire celestial sphere video storage unit 54 , and outputs the entire celestial sphere video to the reproduction device 15 .

[0215] In step S193, the controller 51 acquires metadata supplied from the PC 13, and causes the reception metadata storage unit 91 to store as reception metadata.

[0216] In step S194 , the controller 51 acquires the planar video that has been captured by the planar camera 12 and has been encoded and supplied by the encoding device 22 , and causes the planar video storage unit 53 to store the planar video.

[0217] In step S195 , the video monitoring unit 92 outputs a signal indicating the timing when the planar video that has been captured by the planar camera 12 and has been encoded and provided by the encoding device 22 is stored in the planar video storage unit 53 to the time information generating unit 93 .

[0218] In step S196 , the time information generation unit 93 generates time information at the timing of receiving the signal supplied from the video monitoring unit 92 .

[0219] In step S197 , the time information generating unit 93 reads the reception metadata stored in the reception metadata storing unit 91 , adds the generated time information, and causes the transmission metadata storing unit 94 to store as transmission metadata.

[0220] In step S198 , the controller 51 reads the plane video stored in the plane video storage unit 53 , and outputs the plane video to the reproduction device 15 .

[0221] In step S199 , the controller 51 reads the transmission metadata stored in the transmission metadata storage unit 94 , and outputs the transmission metadata to the reproduction device 15 as metadata.

[0222] In step S200, the controller 51 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S191, and the subsequent processes are repeated.

[0223] Then, in step S200 , when the end of the process is instructed, the process ends.

[0224] Through the above processing, in addition to the entire spherical video and the planar video, metadata to which time information corresponding to the timing when the planar video was acquired is added is also transmitted to the reproduction device 15 .

[0225] It should be noted that in the reproduction device 15, when processing the planar video, the pan-tilt information included in the metadata including the same time information is used to perform the same Figure 7 Therefore, the description of the reproduction process will be omitted.

[0226] <<4. Second Embodiment>>

[0227] In the above, a description has been given of an example in which, in a case where an entire celestial video, a plane video, and metadata are provided to a reproduction device 15 and in the reproduction device 15, when the entire celestial video is switched to a plane video according to line quality, a frame is added and displayed at a position in the entire celestial video corresponding to the plane video to be switched to and displayed next based on the metadata.

[0228] However, in PC 13, a framed entire celestial sphere video in which a frame indicating the position of a planar video has been added can be generated, and in addition to the entire celestial sphere video and the planar video, the framed entire celestial sphere video can be provided to the reproduction device 15 so that the reproduction device 15 can switch and display the video.

[0229] Fig.12An example configuration of a reproduction system 1 is shown, wherein a framed entire celestial sphere video in which a frame indicating the position of a planar video has been added to the entire celestial sphere video is generated in a PC 13, and the framed entire celestial sphere video is provided to a reproduction device 15 in addition to the entire celestial sphere video and the planar video, so that the reproduction device 15 switches and displays the video.

[0230] exist Fig.12 In the reproduction system 1, instead of Figure 2 The data controller 32 and the delay processing unit 33 of the PC 13 are provided with a delay processing unit 111, a synthesis processing unit 112 and a data controller 113. Figure 2 The reproduction system 1 in is different.

[0231] In addition, Fig.12 In the reproduction system 1, instead of Figure 2 The metadata storage unit 52 in the server 14 provides the framed entire celestial video storage unit 121. In addition, the meta parser unit 74 is omitted in the reproduction device 15. This configuration is similar to Figure 2 The reproduction system 1 in is different.

[0232] The delay processing unit 111 of the PC 13 temporarily stores the entire sphere video that has been captured by the entire sphere camera 11 , and outputs the entire sphere video to the synthesis processing unit 112 to match the timing when the entire sphere video is output from the entire sphere camera 11 .

[0233] The data controller 113 obtains the position and size of the planar video in the entire celestial sphere video based on the pan-tilt information supplied from the camera platform 12 a , generates frame data for identifying the corresponding range, and outputs the frame data to the synthesis processing unit 112 .

[0234] The synthesis processing unit 112 adds the framed video corresponding to the plane video to the entire sphere video to generate the framed entire sphere video based on the frame data supplied from the data controller 113 , and outputs the framed entire sphere video to the server 14 .

[0235] The controller 51 of the server 14 acquires the framed entire celestial sphere video supplied from the PC 13 , and causes the framed entire celestial sphere video storage unit 121 to store the framed entire celestial sphere video.

[0236] Then, the controller 51 reads the framed entire celestial sphere video stored in the framed entire celestial sphere video storage unit 121 , and outputs the framed entire celestial sphere video to the reproduction device 15 .

[0237] therefore, Fig.12The connection controller 71 in the reproduction device 15 acquires the framed entire sphere video in addition to the entire sphere video and the plane video provided from the server 14 , and outputs the video to the decoding unit 73 .

[0238] Furthermore, the decoding unit 73 decodes the framed entire sphere video in addition to the entire sphere video and the planar video, and outputs the video to the video controller 75 .

[0239] The video controller 75 selects any one of the decoded entire sphere video, planar video, and framed entire sphere video based on the switching determination information supplied from the switching determination unit 72 , and outputs the selected one video to the rendering unit 77 .

[0240] <Processing of generating video of the entire celestial sphere with frame>

[0241] Next, we will refer to Fig.13 The flowchart of FIG. 1 describes the framed entire celestial sphere video generation process in the PC 13 .

[0242] It should be noted that the framed entire celestial sphere video generation process is a process executed on the PC 13, rather than on the PC 13 itself. Figure 5 The flowchart describes the metadata generation process.

[0243] In step S221 , the data controller 113 acquires pan-tilt information supplied from the camera platform 12 a of the planar camera 12 .

[0244] In step S222 , the data controller 113 generates information on the position and size of a frame corresponding to the planar video in the entire celestial video as frame data based on the acquired pan-tilt information, and outputs the frame data to the synthesis processing unit 112 .

[0245] In step S223 , the delay processing unit 111 acquires the entire celestial sphere video supplied from the entire celestial sphere camera 11 , temporarily stores the entire celestial sphere video, delays for a predetermined period, and then outputs the entire celestial sphere video to the synthesis processing unit 112 .

[0246] In step S224 , the synthesis processing unit 112 sets the position and size of the frame corresponding to the planar video on the entire celestial spherical video based on the frame data.

[0247] In step S225 , the synthesis processing unit 112 adds the frame of the set position and size to the entire celestial sphere video, generates the entire celestial sphere video with the frame, encodes the entire celestial sphere video with the frame, and then outputs the entire celestial sphere video with the frame to the server 14 .

[0248] In step S226, the controller 31 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S221, and the subsequent process is repeated.

[0249] Then, in step S226, when the end of the process is instructed, the process ends.

[0250] Through the above processing, frame data based on the pan and tilt information at the timing when the planar video is captured by the planar camera device 12 is generated, and in addition, a framed entire celestial sphere video to which a frame indicating the position and size of the corresponding planar video has been added based on the frame data is generated, and then the framed entire celestial sphere video is output to the server 14.

[0251] As a result, in addition to the entire sphere video and the plane video, the entire sphere video with the frame is transmitted to the server 14 .

[0252] <By Fig.12 Video transmission processing by the server in>

[0253] Next, we will refer to Fig.14 The flowchart described by Fig.12 The server 14 performs the video transmission processing.

[0254] Notice, Fig.14 The processing of steps S241 to S244 and S247 in the flowchart of FIG. 1 is the same as that of steps S241 to S244 and S247 in FIG. 1 . Figure 6 The processing of steps S71 to S74 and S77 described in the flowchart of is similar. Therefore, the description thereof will be omitted.

[0255] That is, in step S245 , the controller 51 acquires the framed entire celestial sphere video supplied from the PC 13 , and causes the framed entire celestial sphere video storage unit 121 to store the framed entire celestial sphere video.

[0256] In step S246 , the controller 51 reads the framed entire celestial sphere video stored in the framed entire celestial sphere video storage unit 121 , and outputs the framed entire celestial sphere video to the reproduction device 15 .

[0257] Through the above processing, the entire sphere video, the planar video, and the framed entire sphere video are provided from the entire sphere camera 11 , the planar camera 12 , and the PC 13 , respectively, and are sent to the reproduction device 15 .

[0258] In this case, regarding the entire sphere video, the plane video, and the entire sphere video with a frame, the respective corresponding information at the same timing is provided to the reproduction device 15 .

[0259] therefore, Fig.14The processing of steps S241 and S242 , the processing of steps S243 and S244 , and the processing of steps S245 and S246 in the flowchart are basically parallel processing.

[0260] <By Fig.12 Reproduction processing by the reproduction device in>

[0261] Next, we will refer to Fig.15 The flowchart described by Fig.12 The reproduction process is performed by the reproduction device 15.

[0262] In step S261 , the connection controller 71 acquires the entire celestial sphere video, the plane video, and the entire celestial sphere video with a frame from the server 14 , and outputs the video to the decoding unit 73 .

[0263] In step S262, the decoding unit 73 decodes the entire sphere video, the plane video, and the framed entire sphere video supplied from the connection controller 71, and outputs the video to the video controller 75. That is, the video controller 75 acquires the decoded entire sphere video, the plane video, and the framed entire sphere video.

[0264] In step S263 , the video controller 75 acquires head tracking information from the HMD 16 .

[0265] In step S264, the connection controller 71 outputs information about the line speed or the remaining buffered data amount of, for example, the entire spherical video and the planar video being received, as information about the line quality of the entire spherical video from the server 14, to the switching determination unit 72. That is, the switching determination unit 72 acquires information about the line speed and the remaining buffered data amount of the video as information about the line quality.

[0266] In step S265, the switching determination unit 72 determines whether the line speed is faster than a predetermined speed or whether the remaining buffered data amount is equal to or greater than a predetermined threshold based on information about the line speed including the entire celestial sphere video and planar video being received and the line quality of the remaining buffered data amount, so as to determine whether the line quality is sufficient to reproduce the entire celestial sphere video, that is, whether to reproduce the entire celestial sphere video.

[0267] In the case where it is determined in step S265 that the line quality is sufficient to reproduce the entire celestial sphere video, the switching determination unit 72 outputs switching determination information for instructing reproduction of the entire celestial sphere video to the video controller 75. The process proceeds to step S266.

[0268] In step S266 , the video controller 75 outputs the decoded entire celestial sphere video to the rendering unit 77 based on the switching determination information.

[0269] In step S267 , the rendering unit 77 renders the entire celestial sphere video, and outputs the entire celestial sphere video to the HMD 16 for display.

[0270] In step S268, the video controller 75 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S261, and the subsequent process is repeated.

[0271] Then, in step S268, in the event that it is determined that the end has been instructed, the processing ends.

[0272] On the other hand, in the event that it is determined in step S265 that the line quality is not sufficient to reproduce the entire celestial sphere video, the process proceeds to step S269.

[0273] In step S269 , the video controller 75 outputs the decoded framed entire celestial sphere video to the rendering unit 77 .

[0274] In step S270 , the rendering unit 77 renders the framed entire celestial sphere video, and outputs the framed entire celestial sphere video to the HMD 16 for display.

[0275] In step S271 , the video controller 75 generates a transition video for transitioning from the entire celestial sphere video to the plane video, and outputs the transition video to the rendering unit 77 .

[0276] In step S272 , the rendering unit 77 renders a transition video for transitioning from the entire celestial sphere video to the plane video, and outputs the transition video to the HMD 16 for display.

[0277] In step S273 , the video controller 75 outputs the decoded plane video to the rendering unit 77 .

[0278] In step S274 , the rendering unit 77 renders the planar video, and outputs the planar video to the HMD 16 for display.

[0279] That is, through the processing of steps S270 to S274, in the HMD 16, after displaying the framed entire spherical video to which a frame indicating the position on the entire spherical video corresponding to the planar video has been added, a transition video for transitioning to the planar video is displayed, and the transition video is switched to display the planar video.

[0280] In step S275 , the connection controller 71 acquires the entire sphere video, the plane video, and the entire sphere video with frame from the server 14 , and outputs the video to the decoding unit 73 .

[0281] In step S276, the decoding unit 73 decodes the entire sphere video, the plane video, and the framed entire sphere video supplied from the connection controller 71, and outputs the video to the video controller 75. That is, the video controller 75 acquires the decoded entire sphere video, the plane video, and the framed entire sphere video.

[0282] In step S277 , the video controller 75 acquires head tracking information from the HMD 16 .

[0283] In step S278, the connection controller 71 outputs, for example, information on the line speed or the remaining buffered data amount of the entire spherical video and the planar video being received, as information on the line quality of the entire spherical video from the server 14, to the switching determination unit 72. That is, the switching determination unit 72 acquires information on the line speed and the remaining buffered data amount of the video as information on the line quality.

[0284] In step S279, the switching determination unit 72 determines whether the line speed is faster than a predetermined speed and whether the remaining buffered data amount is equal to or greater than a predetermined threshold based on information about the line quality including the line speed and the remaining buffered data amount of the entire celestial sphere video and planar video being received, so as to determine whether the line quality is sufficient to reproduce the entire celestial sphere video, that is, whether to reproduce the entire celestial sphere video.

[0285] In the event that it is determined in step S279 that the line quality is insufficient to reproduce the entire spherical video, the switching determination unit 72 outputs switching determination information for instructing reproduction of the planar video to the video controller 75. The process proceeds to step S280.

[0286] In step S280, the video controller 75 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S273, and the subsequent process is repeated.

[0287] Then, in step S280, in the event that it is determined that the end has been instructed, the processing ends.

[0288] That is, in the case where a state in which sufficient line quality for reproducing the entire celestial sphere video cannot be obtained continues, the processes of steps S273 to S280 are repeated, and the plane video is continuously rendered and displayed on the HMD 16 .

[0289] Furthermore, in step S279, in the event that the line quality is determined to be sufficient to reproduce the entire celestial sphere video, the switching determination unit 72 outputs switching determination information for instructing reproduction of the entire celestial sphere video to the video controller 75. The process proceeds to step S281.

[0290] In step S281 , the video controller 75 generates a transition video for transitioning from the planar video to the entire celestial sphere video, and outputs the transition video to the rendering unit 77 .

[0291] In step S282 , the rendering unit 77 outputs the transition video to the HMD 16 for display.

[0292] In step S283 , the video controller 75 outputs the decoded framed entire celestial sphere video to the rendering unit 77 .

[0293] In step S284, the rendering unit 77 outputs the framed entire celestial sphere video to the HMD 16 for display. The process returns to step S266.

[0294] That is, the reverse process of the case where the entire celestial sphere video is transitioned to the planar video, also in the case where the planar video is transitioned to the entire celestial sphere video, after the transition to the entire celestial sphere video through the transition video, the framed entire celestial sphere video is displayed, and then the entire celestial sphere video is displayed.

[0295] <By Fig.12 Display processing by the HMD in>

[0296] Next, we will refer to Fig.16 The flowchart described by Fig.12 The HMD 16 performs display processing.

[0297] In step S291 , the head tracking information acquisition unit 82 of the HMD 16 acquires the head tracking information of the user.

[0298] In step S292 , the head tracking information acquiring unit 82 outputs the acquired head tracking information to the reproducing device 15 .

[0299] In step S293 , the controller 81 acquires the entire sphere video, the plane video, the transition video, or the entire sphere video with a frame provided from the reproduction device 15 , and causes the display unit 83 to display it.

[0300] In step S294, the controller 81 determines whether the end of the process has been instructed. In the case where the end has not been instructed, the process returns to step S291 and the subsequent process is repeated.

[0301] Then, in step S294, if it is determined that the end has been instructed, the processing ends.

[0302] Through the above series of processing, head tracking information is obtained and continuously output to the reproduction device 15, and in addition, the entire celestial sphere video, plane video or framed entire celestial sphere video provided from the reproduction device 15 is also obtained and continuously displayed on the display unit 83.

[0303] Therefore, the entire sphere video or the plane video is switched according to the line quality. When switching, the framed entire sphere video is displayed in which a frame corresponding to the position and size of the plane video has been added to the entire sphere video, so that even when the entire sphere video is switched to the plane video, the user can recognize the plane video at which position it has been switched.

[0304] Furthermore, also when the planar video is switched to the entire celestial sphere video, the entire celestial sphere video is temporarily displayed with a frame after the transition so that it can be recognized at which position in the entire celestial body video the entire celestial sphere video is switched from the planar video.

[0305] As a result, it is possible to switch between the wide-viewing angle video and the flat video which is a part of the wide-viewing angle video without a sense of discomfort in viewing.

[0306] <<5. Examples to be executed by software>>

[0307] Incidentally, the above series of processing can be performed by hardware, but can also be performed by software. In the case of performing a series of processing by software, the program constituting the software is installed in a computer embedded in dedicated hardware, or various programs are installed from a recording medium, for example, in a general-purpose computer capable of performing various functions.

[0308] Fig.17 Such a personal computer includes a built-in central processing unit (CPU) 1001. An input and output interface 1005 is connected to the CPU 1001 via a bus 1004. A read only memory (ROM) 1002 and a random access memory (RAM) 1003 are connected to the bus 1004.

[0309] The input and output interface 1005 is connected to an input unit 1006 including an input device such as a keyboard and a mouse for a user to input an operation command, an output unit 1007 for outputting a processing operation screen and an image of a processing result to a display device, a storage unit 1008 including a hard disk drive and the like for storing programs and various data, and a communication unit 1009 including a local area network (LAN) adapter and the like and performing communication processing via a network represented by the Internet. A drive 1010 is also connected, which is used to read data from and write data to a magnetic disk (including a floppy disk), an optical disk (including a compact disk read-only memory (CD-ROM) and a digital versatile disk (DVD)), a magneto-optical disk (including a mini disk (MD)), or a removable storage medium 1011 (such as a semiconductor memory).

[0310] The CPU 1001 executes various processes according to a program stored in the ROM 1002 or a program that has been read from a removable storage medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, installed in the storage unit 1008, and loaded from the storage unit 1008 into the RAM 1003. In the RAM 1003, for example, data necessary for the CPU 1001 to execute various processes is also appropriately stored.

[0311] In the computer configured as described above, the CPU 1001 loads the program stored in the storage unit 1008 into the RAM 1003 via the input and output interface 1005 and the bus 1004 , and executes the above-described series of processing.

[0312] For example, the program executed by the computer (CPU 1001) may be recorded into the removable storage medium 1011 to be provided as a package medium, etc. Furthermore, the program may be provided via a wired or wireless transmission medium such as a local area network, the Internet, and digital satellite broadcasting.

[0313] On the computer, by attaching a removable storage medium 1011 to a drive 1010, the program can be installed into the storage unit 1008 via the input and output interface 1005. In addition, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed into the storage unit 1008. In addition, the program can be pre-installed into the ROM 1002 or the storage unit 1008.

[0314] Note that the program executed by the computer may be a program that is processed time-sequentially according to the order described in this specification, or may be a program that is processed in parallel or at necessary timing (for example, when a call is made).

[0315] Notice, Fig.17 The CPU 1001 in Figure 2 The functions of controllers 31, 51, 81 and video controller 75.

[0316] In addition, in this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), and it is not important whether all the components are included in the same housing. Therefore, multiple devices housed in separate housings and connected via a network and a device in which multiple modules are housed in one housing are both systems.

[0317] It is to be noted that the embodiments of the present disclosure are not limited to the above-described embodiments, and various changes may be made without departing from the gist of the present disclosure.

[0318] For example, the present disclosure may be configured as cloud computing in which a plurality of devices share one function via a network and perform processing collaboratively.

[0319] Furthermore, each step described in the above-mentioned flowchart may be performed by one device, or may be shared and performed by a plurality of devices.

[0320] Furthermore, in the case where one step includes a plurality of processes, the plurality of processes included in such one step may be executed by one device or shared by a plurality of devices.

[0321] It is to be noted that the present disclosure may also have the following configurations.

[0322] <1> A reproduction device, comprising:

[0323] a switching determination unit configured to determine whether a switching condition for switching display between a wide-viewing angle video and a partial video corresponding to a partial viewing angle of the wide-viewing angle video is satisfied; and

[0324] a video controller configured to selectively display the wide-viewing-angle video and the partial video based on a determination result of the switching determination unit,

[0325] When switching display between the wide-view video and the partial video based on the determination result of the switching determination unit, the video controller displays the wide-view video to which display information indicating the viewing angle of the partial video corresponding to the wide-view video has been added.

[0326] <2> In the above <1> The reproduction device described in claim 1 further comprises:

[0327] A video acquisition unit is configured to acquire the wide-viewing angle video and the partial video from an externally provided server.

[0328] <3> In the above <2> The reproduction device described in

[0329] Wherein, based on the line quality with the server from which the video acquisition unit acquires the wide-viewing-angle video, the switching determination unit determines whether the switching condition for switching display between the wide-viewing-angle video and the partial video is satisfied.

[0330] <4> In the above <3> The reproduction device described in

[0331] The switching condition includes at least one of the following items: whether the line quality is faster than a predetermined line speed; and whether the remaining buffered data amount of the wide-viewing angle video and the partial video is larger than a predetermined threshold.

[0332] <5> In the above <2> The reproduction device described in claim 1 further comprises:

[0333] a metadata acquisition unit configured to acquire metadata for identifying a position and a size of a viewing angle of the partial video corresponding to the wide viewing angle video from the server,

[0334] The video controller displays the wide-viewing angle video to which display information indicating the viewing angle of the portion of the video corresponding to the wide-viewing angle video has been added based on the metadata.

[0335] <6> In the above <5> The reproduction device described in

[0336] The metadata is generated based on pan-tilt information, and the pan-tilt information includes information about a video capture position and a pan-tilt of a partial video capture unit that captures the partial video.

[0337] <7> In the above <5> The reproduction device described in

[0338] The metadata acquisition unit acquires the metadata in a state adjusted to acquire the metadata at the same timing as the timing at which the video acquisition unit acquires the partial video.

[0339] <8> In the above <5> The reproduction device described in

[0340] The metadata acquisition unit acquires metadata including time information when the corresponding partial video is provided, and

[0341] The video controller displays the wide-view video to which the display information indicating the view angle of the partial video corresponding to the wide-view video is added, using the metadata including the time information corresponding to the partial video.

[0342] <9> In the above <1> to <8> The reproduction device described in any one of the above

[0343] The display information indicating the viewing angle of the portion of the video corresponding to the wide viewing angle video includes a frame indicating the viewing angle of the portion of the video corresponding to the wide viewing angle video.

[0344] <10> In the above <2> The reproduction device described in

[0345] Wherein, in addition to the wide-viewing angle video and the partial video, the video acquisition unit further acquires the wide-viewing angle video to which the display information indicating the viewing angle of the partial video corresponding to the wide-viewing angle video has been added.

[0346] <11> In the above <10> The reproduction device described in

[0347] Wherein, a wide-angle video to which the display information indicating the viewing angle of the partial video corresponding to the wide-angle video is added is generated by adding the display information to the wide-angle video based on information including information related to the panning and tilting of a partial video capture unit that captures the partial video and the panning and tilting information of the video capture position.

[0348] <12> In the above <1> The reproduction device described in

[0349] The switching determination unit determines whether the switching condition for switching display between the wide-viewing angle video and the partial video is satisfied based on presentation content including the content of the wide-viewing angle video.

[0350] <13> In the above <1> to <12> The reproduction device described in any one of the above

[0351] In which, when displaying the wide-angle video, at a timing immediately before the display is switched from the wide-angle video to the partial video based on the determination result of the switching determination unit, the video controller displays the wide-angle video to which the display information indicating the angle of view of the partial video corresponding to the wide-angle video has been added.

[0352] <14> In the above <13> The reproduction device described in

[0353] When the display is switched from the wide-viewing angle video to the partial video, the video controller displays a transition video for transitioning from the wide-viewing angle video to the partial video.

[0354] <15> In the above <14> The reproduction device described in

[0355] Wherein, when the display is switched from the wide-viewing angle video to the partial video, the video controller displays a transition video for transitioning to the partial video while expanding the viewing angle of the wide-viewing angle video.

[0356] <16> In the above <1> to <12> The reproduction device described in any one of the above

[0357] Wherein, in the case of displaying the partial video, at a timing immediately after the display is switched from the partial video to the wide-angle video based on the determination result of the switching determination unit, the video controller displays the wide-angle video to which the display information indicating the angle of view of the partial video corresponding to the wide-angle video has been added.

[0358] <17> In the above <16> The reproduction device described in

[0359] When the display is switched from the partial video to the wide-viewing-angle video, the video controller displays a transition video for transitioning from the partial video to the wide-viewing-angle video.

[0360] <18> In the above <17> The reproduction device described in

[0361] Wherein, when the display is switched from the partial video to the wide-viewing-angle video, the video controller displays a transition video for transitioning to the wide-viewing-angle video while reducing the viewing angle of the partial video.

[0362] <19> A reproduction method, comprising:

[0363] a switching determination process for determining whether a switching condition for switching display between a wide-viewing angle video and a partial video corresponding to a partial view of the wide-viewing angle video is satisfied; and

[0364] a video control process for selectively displaying the wide-viewing angle video and the partial video based on a determination result of the switching determination process,

[0365] When the display is switched between the wide-view video and the partial video based on the determination result of the switching determination processing, the video control processing displays the wide-view video to which display information indicating the viewing angle of the partial video corresponding to the wide-view video has been added.

[0366] <20> A program for causing a computer to:

[0367] a switching determination unit configured to determine whether a switching condition for switching display between a wide-viewing angle video and a partial video corresponding to a partial viewing angle of the wide-viewing angle video is satisfied; and

[0368] a video controller configured to selectively display the wide-viewing-angle video and the partial video based on a determination result of the switching determination unit,

[0369] When switching display between the wide-view video and the partial video based on the determination result of the switching determination unit, the video controller displays the wide-view video to which display information indicating the viewing angle of the partial video corresponding to the wide-view video has been added.

[0370] Reference numerals list

[0371] 1 Reproduction system

[0372] 11 The entire celestial camera

[0373] 12. Planar camera device

[0374] 13 PC

[0375] 14 Server

[0376] 15 Playback device, HMD

[0377] 31 Controller

[0378] 32 Data Controller

[0379] 33 Delay Processing Unit

[0380] 51 Controller

[0381] 52 Metadata storage unit

[0382] 53 Planar video storage unit

[0383] 54 Entire Sphere Video Storage Unit

[0384] 71 Connecting the controller

[0385] 72 Switch determination unit

[0386] 73 Decoding unit

[0387] 74 Metaparser Unit

[0388] 75 Video Controller

[0389] 76 Presentation Units

[0390] 81 Controller

[0391] 82 Head tracking information acquisition unit

[0392] 83 Display unit

[0393] 91 Receive metadata storage unit

[0394] 92 Video Monitoring Unit

[0395] 93 Time information generation unit

[0396] 94 Send metadata storage unit

[0397] 111 Delay Processing Unit

[0398] 112 Composition Processing Unit

[0399] 113 Data Controller

[0400] 121 Framed Entire Sphere Video Storage Unit

Claims

1. A reproduction device, include: a switching determination unit configured to determine whether a switching condition for switching display between a wide-viewing angle video and a partial video corresponding to a partial viewing angle of the wide-viewing angle video is satisfied; a video controller configured to selectively display the wide-viewing-angle video and the partial video based on a determination result of the switching determination unit; as well as a video acquisition unit configured to acquire the wide-viewing angle video and the partial video from an externally provided server, wherein, when display is switched between the wide-viewing angle video and the partial video based on a determination result of the switching determination unit, the video controller displays the wide-viewing angle video to which display information indicating the viewing angle of the partial video corresponding to the wide-viewing angle video has been added, and Wherein, based on the line quality with the server from which the video acquisition unit acquires the wide-viewing-angle video, the switching determination unit determines whether the switching condition for switching display between the wide-viewing-angle video and the partial video is satisfied.

2. The reproduction device according to claim 1, in, The switching condition includes at least one of the following items: whether the line quality is faster than a predetermined line speed; and whether the remaining buffered data amount of the wide-view video and the partial video is larger than a predetermined threshold.

3. The reproduction device according to claim 1, further comprising: include: a metadata acquisition unit configured to acquire metadata for identifying a position and a size of a viewing angle of the partial video corresponding to the wide viewing angle video from the server, The video controller displays the wide-viewing angle video to which display information indicating the viewing angle of the portion of the video corresponding to the wide-viewing angle video has been added based on the metadata.

4. The reproduction device according to claim 3, in, The metadata is generated based on pan-tilt information including information about a video capturing position and a pan-tilt of a partial video capturing unit that captures the partial video.

5. The reproduction device according to claim 3, in, The metadata acquisition unit acquires the metadata in a state adjusted to acquire the metadata at the same timing as the timing at which the video acquisition unit acquires the partial video.

6. The reproduction device according to claim 3, in, The metadata acquisition unit acquires metadata including time information when the corresponding partial video is provided, and The video controller displays the wide-view video to which the display information indicating the view angle of the partial video corresponding to the wide-view video is added, using the metadata including the time information corresponding to the partial video.

7. The reproduction device according to claim 1, in, The display information indicating the angle of view of the partial video corresponding to the wide angle of view video includes a frame indicating the angle of view of the partial video corresponding to the wide angle of view video.

8. The reproduction device according to claim 1, in, The video acquisition unit acquires, in addition to the wide-viewing-angle video and the partial video, the wide-viewing-angle video to which the display information indicating the viewing angle of the partial video corresponding to the wide-viewing-angle video has been added.

9. The reproduction device according to claim 8, in, A wide-angle video to which the display information indicating the angle of view of the partial video corresponding to the wide-angle video is added is generated by adding the display information to the wide-angle video based on information including information related to the pan and tilt of a partial video capturing unit that captures the partial video and pan and tilt information of a video capturing position.

10. The reproduction device according to claim 1, in, The switching determination unit determines whether the switching condition for switching display between the wide-viewing-angle video and the partial video is satisfied based on presentation contents including the contents of the wide-viewing-angle video.

11. The reproduction device according to claim 1, in, In the case of displaying the wide-angle video, at a timing immediately before the display is switched from the wide-angle video to the partial video based on a determination result of the switching determination unit, the video controller displays the wide-angle video to which the display information indicating the angle of view of the partial video corresponding to the wide-angle video has been added.

12. The reproduction device according to claim 11, in, When the display is switched from the wide-viewing-angle video to the partial video, the video controller displays a transition video for transitioning from the wide-viewing-angle video to the partial video.

13. The reproduction device according to claim 12, in, When the display is switched from the wide-viewing-angle video to the partial video, the video controller displays a transition video for transitioning to the partial video while enlarging the viewing angle of the wide-viewing-angle video.

14. The reproduction device according to claim 1, in, In the case of displaying the partial video, at a timing immediately after the display is switched from the partial video to the wide-angle video based on a determination result of the switching determination unit, the video controller displays the wide-angle video to which the display information indicating the angle of view of the partial video corresponding to the wide-angle video has been added.

15. The reproduction device according to claim 14, in, When the display is switched from the partial video to the wide-viewing-angle video, the video controller displays a transition video for transitioning from the partial video to the wide-viewing-angle video.

16. The reproduction device according to claim 15, in, When the display is switched from the partial video to the wide-viewing-angle video, the video controller displays a transition video for transitioning to the wide-viewing-angle video while reducing the viewing angle of the partial video.

17. A reproduction method, include: a switching determination process for determining whether a switching condition for switching display between a wide-viewing angle video and a partial video corresponding to a partial viewing angle of the wide-viewing angle video is satisfied; a video control process for selectively displaying the wide-viewing-angle video and the partial video based on a determination result of the switching determination process; as well as A video acquisition process for acquiring the wide-viewing angle video and the partial video from an externally provided server, wherein, when display is switched between the wide-viewing angle video and the partial video based on a determination result of the switching determination processing, the video control processing displays the wide-viewing angle video to which display information indicating the viewing angle of the partial video corresponding to the wide-viewing angle video has been added, and The switching determination process determines whether the switching condition for switching display between the wide-viewing angle video and the partial video is satisfied based on the line quality with the server from which the video acquisition process acquires the wide-viewing angle video.

18. A computer-readable storage medium having stored thereon a program which, when executed by a computer, causes the computer to function as: a switching determination unit configured to determine whether a switching condition for switching display between a wide-viewing angle video and a partial video corresponding to a partial viewing angle of the wide-viewing angle video is satisfied; a video controller configured to selectively display the wide-viewing-angle video and the partial video based on a determination result of the switching determination unit; as well as a video acquisition unit configured to acquire the wide-viewing angle video and the partial video from an externally provided server, wherein, when display is switched between the wide-viewing angle video and the partial video based on a determination result of the switching determination unit, the video controller displays the wide-viewing angle video to which display information indicating the viewing angle of the partial video corresponding to the wide-viewing angle video has been added, and Wherein, based on the line quality with the server from which the video acquisition unit acquires the wide-viewing-angle video, the switching determination unit determines whether the switching condition for switching display between the wide-viewing-angle video and the partial video is satisfied.

Images