Video distribution system, video distribution device, terminal, method, and program

Abstract

To view a panoramic moving image and a 2D moving image while switching between them.SOLUTION: A moving image distribution system according to an embodiment comprises: a first creation unit which creates a first entire image obtained by converting a first frame image constituting a first moving image to a predetermined first resolution; a second creation unit which creates a supplementary image obtained by converting a second frame image constituting a second moving image supplementary to the first moving image to a predetermined second resolution; a composition unit which creates a second entire image obtained by composing the supplementary image and the first entire image; and a distribution unit which distributes the second entire image to a user terminal.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a video distribution system, a video distribution apparatus, a terminal, a method , and and a program.

Background Art

[0002] In recent years, videos that can be viewed in all directions of 360 degrees, up, down, left, and right, are known. Such videos are called panoramic videos (or also referred to as "panoramic videos", "360-degree panoramic videos", "VR (Virtual Reality) videos", "VR videos", "3D videos", etc.). On the other hand, normal videos that are not panoramic videos are also called "2D videos" etc. in contrast to panoramic videos.

[0003] Also, for example, a video distribution server streams and distributes a panoramic video to a client terminal, and the client terminal also watches the panoramic video. However, panoramic videos generally have a large data size in many cases, and it takes time to distribute the panoramic video, and it may take time until the panoramic video is displayed on the client terminal.

[0004] On the other hand, a technique is known in which the range displayed on the client terminal (hereinafter, also referred to as the "viewing range") is distributed in normal image quality, and the range not displayed on the client terminal is distributed in low image quality (for example, Non-Patent Document 1). By using this technique, when watching a panoramic video by streaming distribution or the like, it is possible to reduce the time required for distributing the panoramic video while suppressing a decrease in the viewing quality of the user.

Prior Art Documents

Non-Patent Documents

[0005]

Non-Patent Document 1

[0006] By the way, for content that is often viewed as panoramic videos, such as sports, concerts, and theatrical performances, regular 2D videos are also frequently available, and there is a need to be able to switch between panoramic and 2D videos while watching.

[0007] One embodiment of the present invention has been made in view of the above points, and aims to enable viewing while switching between panoramic video and 2D video. [Means for solving the problem]

[0008] To achieve the above objective, a video distribution system according to one embodiment includes: a first creation unit that creates a first overall image by converting a first frame image constituting a first video to a predetermined first resolution; a second creation unit that creates an auxiliary image by converting a second frame image constituting a second video that is supplementary to the first video to a predetermined second resolution; a synthesis unit that creates a second overall image by combining the auxiliary image with the first overall image; and a distribution unit that distributes the second overall image to a user terminal. [Effects of the Invention]

[0009] You can switch between panoramic and 2D videos while watching. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram illustrates examples of partial area tiles and overall reduction tiles. [Figure 2] This figure shows an example of the overall configuration of a video distribution system according to the first embodiment. [Figure 3] This diagram illustrates an example of VR viewing mode and 2D viewing mode. [Figure 4] This figure shows an example of the functional configuration of a video distribution system according to the first embodiment. [Figure 5] This is a sequence diagram showing an example of video distribution processing according to the first embodiment. [Figure 6] This is a diagram (part 1) illustrating an example of combining a full-size reduced tile and thumbnail tiles for panoramic video. [Figure 7] This is a diagram (part 2) illustrating an example of combining a full-size reduced tile and thumbnail tiles for panoramic video. [Figure 8] This is a sequence diagram showing an example of the mode switching process according to the first embodiment. [Figure 9] This is a diagram (part 1) illustrating an example of switching from VR viewing mode to 2D viewing mode. [Figure 10] This is a diagram (part 2) illustrating an example of switching from VR viewing mode to 2D viewing mode. [Figure 11] This diagram illustrates an example of switching from 2D viewing mode to VR viewing mode. [Figure 12] This diagram illustrates an example of a ROI panoramic video. [Figure 13] This is a sequence diagram showing an example of video distribution processing according to the second embodiment. [Modes for carrying out the invention]

[0011] Hereinafter, a first embodiment and a second embodiment will be described as an embodiment of the present invention.

[0012] [First Embodiment] First, let's describe the first embodiment. In this embodiment, we will describe a video distribution system 1 that allows viewing while switching between panoramic videos and 2D videos.

[0013] <How to distribute panoramic videos> As a technology underlying this embodiment, a method for distributing panoramic videos described in Non-Patent Document 1 will be explained. In the technology described in this Non-Patent Document 1, the viewing range displayed on the client terminal is distributed in normal image quality, and the other ranges are distributed in low image quality.

[0014] For example, as shown in FIG. 1, in a panoramic video represented by an all-sky projection sphere, with the center of the all-sky projection sphere as the observation position, the range captured by a pseudo camera device at this observation position with a predetermined angle of view becomes the viewing range. Also, a panoramic video represented by an all-sky projection sphere can be represented by an orthographic cylindrical projection method where the vertical angle θ is 0 to π and the horizontal angle φ is 0 to 2π. At this time, in the technology described in Non-Patent Document 1, a partial region including the viewing range of the panoramic video (more precisely, the frame image of the panoramic video) represented by the orthographic cylindrical projection method is defined as a "partial region tile", and an image obtained by compressing the entire panoramic video is defined as an "entire reduced tile" and distributed to the client terminal. By using this technology, the time required for distributing the panoramic video can be reduced, and a high-quality video can be viewed within the viewing range. Also, even when the viewing range is moved, the video can be viewed without interruption. Here, compression means reducing the image size.

[0015] Note that the term "viewing range" was used above, but this may also be referred to as, for example, "visible range", "display range", "viewing range", etc.

[0016] <Overall Configuration of Video Distribution System 1> Figure 2 shows the overall configuration of the video distribution system 1 according to this embodiment. As shown in Figure 2, the video distribution system 1 according to this embodiment includes a video distribution server 10, one or more client terminals 20, one or more panoramic video cameras 30, and one or more 2D video cameras 40. The video distribution server 10 and each client terminal 20 are connected via a communication network N1, such as the Internet. Furthermore, the video distribution server 10, each panoramic video camera 30, and each 2D video camera 40 are connected via a communication network N2, such as a LAN (Local Area Network).

[0017] The video distribution server 10 is a computer or computer system that streams panoramic videos and 2D videos in response to requests from client terminals 20. In this case, the video distribution server 10 distributes the videos to the client terminals 20 in a way that allows them to switch between panoramic videos and 2D videos while viewing.

[0018] The client terminal 20 is a type of device capable of viewing panoramic videos and 2D images. Examples of client terminals 20 include smartphones, tablet devices, head-mounted displays, wearable devices, and PCs (personal computers). Users can switch between viewing panoramic videos and 2D videos using the client terminal 20.

[0019] Furthermore, the user can move or change the field of view in the panoramic video by operating the client terminal 20. Such operations may include, for example, swiping or flicking on the touch panel of the client terminal 20, pressing a button to move the field of view, or pressing a button to move the field of view using a pointing device such as a mouse. In addition, if the client terminal 20 is equipped with a sensor capable of detecting orientation in all directions (up, down, left, and right, 360 degrees), the user may also move or change the field of view by changing the orientation of the client terminal 20.

[0020] The panoramic video camera 30 is a shooting device such as a camera that generates images for creating panoramic videos (hereinafter also referred to as panoramic video images). The panoramic video camera 30 may be a regular camera, a camera equipped with a wide-angle lens such as a fisheye lens, or a dedicated camera for shooting panoramic videos (for example, a camera equipped with multiple wide-angle lenses such as fisheye lenses).

[0021] The 2D video camera 40 is a camera or other shooting device that generates captured images (hereinafter also referred to as 2D video captured images) for creating 2D videos.

[0022] Note that the overall configuration of the video distribution system 1 shown in Figure 1 is just one example, and other configurations are also possible. For example, if sound and audio are also to be distributed to the client terminal 20, a sound collection device such as a microphone may be included in the video distribution system 1.

[0023] Furthermore, if panoramic videos or 2D videos are delivered on demand rather than via streaming, the video distribution system 1 may include a storage device for holding the panoramic videos or 2D videos. The following explanation primarily assumes the case of streaming live content such as sports, concerts, and theatrical performances, but the following explanation is also applicable when the content is recorded and delivered on demand.

[0024] <Viewing Mode> Here, the client terminal 20 has a "VR viewing mode" for users to view panoramic videos and a "2D viewing mode" for users to view 2D videos. Note that the "VR viewing mode" may also be called, for example, "panoramic viewing mode" or "3D viewing mode".

[0025] As shown in Figure 3, in VR viewing mode, the client terminal 20 displays the VR viewing screen 1000. On this VR viewing screen 1000, a panoramic video 1001 is displayed from a partial area tile (or a full-size reduced tile), and 2D video thumbnails 1002-1003 are displayed on top of the panoramic video 1001. When the user selects either 2D video thumbnail 1002 or 1003, the system transitions from VR viewing mode to 2D viewing mode.

[0026] As shown in Figure 3, in 2D viewing mode, the client terminal 20 displays a 2D viewing screen 2000. This 2D viewing screen 2000 displays the 2D video 2001 selected by the user, as well as a thumbnail video 2002 of the panoramic video and a thumbnail video 2003 of another 2D video different from the currently displayed 2D video 2001. If the user selects the panoramic video thumbnail video 2002, the system transitions from 2D viewing mode to VR viewing mode. If the user selects the thumbnail video 2003 of another 2D video, the system remains in 2D viewing mode and displays that other 2D video. Note that the 2D video 2001, thumbnail videos 2002 and 2003 are displayed on the panoramic video represented by the overall reduced tile.

[0027] As described above, the client terminal 20 according to this embodiment displays a thumbnail video of a 2D video during VR viewing mode, and also displays thumbnail videos of a panoramic video and other 2D videos during 2D viewing mode. When any of the thumbnail videos is selected by the user, the video corresponding to that thumbnail video (panoramic video or 2D video) is displayed. This allows the user to easily switch between viewing panoramic videos and 2D videos. The thumbnail videos may also be called, for example, "auxiliary videos" or "auxiliary images."

[0028] Therefore, for example, in a sports broadcast, if both a panoramic video showing the entire stadium and 2D videos such as a player's point-of-view video or video from a camera fixed in the stadium are provided, users can watch the match while switching between the panoramic video and the 2D video.

[0029] In the example shown in Figure 3, there are two thumbnail videos in both the VR viewing screen 1000 and the 2D viewing screen 2000. However, this is just one example, and the number of thumbnail videos can be any number. Furthermore, the number of thumbnail videos can be set by the user, and the number of thumbnail videos may differ between the VR viewing screen 1000 and the 2D viewing screen 2000.

[0030] Furthermore, in the example shown in Figure 3, thumbnail videos are displayed at the bottom of the VR viewing screen 1000 and the 2D viewing screen 2000, but this is just one example, and they may also be displayed at the top, left, or right. Alternatively, thumbnail videos may be distributed across two or more parts of the screen, such as the top, bottom, left, and right. However, given the human visual field characteristics, where a wider area can be seen horizontally than vertically, it is preferable to display thumbnail videos at the bottom or top (especially the bottom).

[0031] In addition, thumbnail videos do not need to be displayed at all times on the VR viewing screen 1000 and the 2D viewing screen 2000. For example, thumbnail videos may normally be hidden and only displayed when some user action is detected or when a new thumbnail video is added. Also, once a thumbnail video has been displayed, it may be hidden again if, for example, no user action is detected for a certain period of time.

[0032] The following primarily assumes that a user can view one panoramic video, but a user may be able to view multiple panoramic videos. For example, if a first panoramic video and a second panoramic video are viewable, there may be a first VR viewing mode for viewing the first panoramic video and a second VR viewing mode for viewing the second panoramic video, and the user may be able to switch between the first VR viewing mode, the second VR viewing mode, and the 2D viewing mode. In this case, the thumbnail video of the second panoramic video is displayed in the first VR viewing mode, and similarly, the thumbnail video of the first panoramic video is displayed in the second VR viewing mode.

[0033] <Functional Configuration of Video Streaming System 1> Figure 4 shows the functional configuration of the video distribution system 1 according to this embodiment.

[0034] ≪Video Streaming Server 10≫ As shown in Figure 4, the video distribution server 10 according to this embodiment has, as functional units, a panoramic video creation unit 101, a panoramic video tile creation unit 102, a 2D video creation unit 103, a 2D video tile creation unit 104, a synthesis unit 105, and a distribution unit 106. Each of these functional units is realized, for example, by processing that one or more programs installed on the video distribution server 10 cause a processor such as a CPU (Central Processing Unit) or GPU (Graphics Processing Unit) to execute.

[0035] The panoramic video creation unit 101 creates a panoramic video using each panoramic video image acquired from the panoramic video camera 30. Specifically, the panoramic video creation unit 101 decodes each panoramic video image for each frame time width, and then creates a panoramic video by stitching together each panoramic video image using a known image processing technique called stitching. The frame time width is the interval between the generation of captured images and is the reciprocal of the frame rate.

[0036] The panoramic video tile creation unit 102 uses the panoramic video created by the panoramic video creation unit 101 to create a full-size reduced tile, a partial area tile, and a thumbnail tile for displaying a thumbnail video of the panoramic video during 2D viewing mode (hereinafter also referred to as a panoramic video thumbnail tile).

[0037] Here, a "whole reduction tile" refers to an image obtained by compressing the entire panoramic video (more precisely, the frame images of the panoramic video) represented in equirectangular projection to a predetermined resolution (i.e., a predetermined width and height). However, unlike the whole reduction tile described in Non-Patent Document 1, in this embodiment, the whole reduction tile has a panoramic video area and a thumbnail area, and the image obtained by compressing the panoramic video represented in equirectangular projection to a predetermined resolution corresponds to the panoramic video area. If an effective area is defined for the whole reduction tile, the effective area may be designated as the panoramic video area, and the other ineffective areas as thumbnail areas. Furthermore, below, the term "reduction" will be used consistently to refer to compressing the resolution of an image, but for example, since a whole reduction tile can also be described as an image obtained by converting the entire panoramic video to a predetermined resolution, the term "conversion" may be used instead of "reduction."

[0038] Furthermore, a partial region tile is an image with the same resolution as the overall reduction tile, and represents a portion of the panoramic video (more precisely, its frame images). A partial region tile is created for each frame image of the panoramic video, covering the entire frame image.

[0039] For example, if the resolution of each frame image in a panoramic video is 11520 x 6480 pixels, and the resolution of the sub-region tiles is 1920 x 1080 pixels, and the sub-region tiles are created with an overlap of 960 pixels horizontally and 540 pixels vertically, then 11 x 11 = 121 sub-region tiles will be created for each frame image in the panoramic video. This is just one example, and the resolution of the sub-region tiles and the degree of overlap can be set as appropriate for each frame image. Generally, creating sub-region tiles with more overlap can provide users with higher viewing quality, but this requires more computing resources and memory.

[0040] Furthermore, a thumbnail tile for panoramic video is an image obtained by compressing a partial region tile to a predetermined resolution for use as a thumbnail.

[0041] The 2D video creation unit 103 creates each 2D video (more precisely, their frame images) using each 2D video image captured from the 2D video camera 40. In other words, the 2D video creation unit 103 creates each 2D video by decoding each 2D video image captured for each frame time width.

[0042] The 2D video tile creation unit 104 uses the 2D video created by the 2D video creation unit 103 to create a 2D video tile for displaying the 2D video, and a thumbnail tile (hereinafter also referred to as a 2D video thumbnail tile) for displaying a thumbnail video of the 2D video during VR viewing mode or 2D viewing mode.

[0043] Here, a 2D video tile is an image with a resolution lower than or equal to that of a partial region tile. A 2D video tile is created for each 2D video. A 2D video thumbnail tile is an image with the same resolution as a panoramic video thumbnail tile, but is an image obtained by compressing the 2D video tile to a predetermined resolution for thumbnails.

[0044] The compositing unit 105 combines the 2D video thumbnail tiles of each 2D video into the thumbnail area of ​​the overall reduced tile when the client terminal 20 is in VR viewing mode. Furthermore, when the client terminal 20 is in 2D viewing mode, the compositing unit 105 combines the 2D video thumbnail tiles of other 2D videos (excluding the currently viewed 2D video) and the panoramic video thumbnail tiles into the thumbnail area of ​​the overall reduced tile.

[0045] The distribution unit 106 distributes a full-screen reduction tile and a partial-area tile corresponding to the field of view of the client terminal 20 when the client terminal 20 is in VR viewing mode. Furthermore, the distribution unit 106 distributes a full-screen reduction tile and a 2D video tile of the 2D video selected by the user (i.e., the 2D video the user is currently watching) when the client terminal 20 is in 2D viewing mode.

[0046] ≪Client Terminal 20≫ As shown in Figure 4, the client terminal 20 according to this embodiment has a video viewing unit 201 as a functional unit. This functional unit is realized, for example, by having one or more programs installed on the client terminal 20 execute processes on a processor such as a CPU or GPU.

[0047] The video viewing unit 201 sends a distribution request to the video distribution server 10 to display a panoramic video or a 2D video and a thumbnail video, depending on the current viewing mode. Here, the distribution request includes, for example, information indicating the current viewing mode and the center coordinates of the field of view or information for identifying those center coordinates. Examples of information for identifying the center coordinates of the field of view include the coordinates of each vertex of the field of view and the coordinates of the diagonal vertices.

[0048] Furthermore, the video viewing unit 201 displays panoramic videos or 2D videos and thumbnail videos using tiles (overall reduced tiles, partial area tiles, or 2D video tiles) distributed from the video distribution server 10.

[0049] <Video streaming processing> Figure 5 shows a sequence diagram of the video distribution process according to this embodiment. Here, steps S101 to S102 are repeatedly executed for each frame time width of the captured image for panoramic video, and steps S103 to S104 are repeatedly executed for each frame time width of the captured image for 2D video. Below, steps S101 to S104 for a given frame time t will be described. The frame time t is the time that can be expressed as t=t0+Δt, t0+2Δt, t0+3Δt, ..., where t0 is the start time of shooting and Δt is the frame time width.

[0050] The panoramic video creation unit 101 of the video distribution server 10 creates a panoramic video (more precisely, its frame images) using each panoramic video capture image at frame time t (step S101).

[0051] Next, the panoramic video tile creation unit 102 of the video distribution server 10 uses the panoramic video created in step S101 above to create a full-scale reduction tile, a partial region tile, and a thumbnail tile for the panoramic video at frame time t (step S102).

[0052] The 2D video creation unit 103 of the video distribution server 10 creates each 2D video (more precisely, their frame images) using the captured images for each 2D video at frame time t (step S103).

[0053] Next, the 2D video tile creation unit 104 of the video distribution server 10 creates a 2D video tile and a 2D video thumbnail tile for frame time t using each 2D video created in step S103 (step S104). For example, if there are M 2D video cameras 40 and 2D video images are obtained from each of these M 2D video cameras 40, then M 2D video tiles and M 2D video thumbnail tiles will be created.

[0054] Next, if the viewing mode of the client terminal 20 is VR viewing mode, steps S105 to S108 are executed, and if it is 2D viewing mode, steps S109 to S112 are executed. The following steps S105 to S108 and steps S109 to S112 are repeatedly executed every n × frame time width, for example, with n being a predetermined integer of 1 or more, as long as the viewing mode is not switched.

[0055] • When in VR viewing mode The video viewing unit 201 of the client terminal 20 sends a VR viewing mode distribution request to the video distribution server 10 (step S105). Here, the VR viewing mode distribution request includes information representing n or more frame times for which tile distribution is requested. However, this information representing n or more frame times is not mandatory and does not have to be included in the VR viewing mode distribution request.

[0056] The compositing unit 105 of the video distribution server 10 composites each 2D video thumbnail tile for the corresponding frame time (i.e., the frame time for which tile distribution is requested) onto the overall reduced tile for that frame time (step S106). For example, if a user can view multiple panoramic videos, the compositing unit 105 further composites the panoramic video thumbnail tiles of panoramic videos other than the panoramic video corresponding to the overall reduced tile onto the overall reduced tile.

[0057] Here, as an example, Figure 6 shows an example of compositing when there are two 2D video cameras 40 and two 2D video thumbnail tiles are created at a certain frame time. As shown in Figure 6, in this case, the compositing is performed by embedding each 2D video thumbnail tile in the thumbnail area of ​​the overall reduced tile.

[0058] Next, the distribution unit 106 of the video distribution server 10 transmits the composite overall reduced tile from step S106 and a partial region tile with the same frame time to the client terminal 20 (step S107). At this time, the distribution unit 106 uses the center coordinates of the field of view included in the VR viewing mode distribution request or information for identifying said center coordinates to identify the partial region tile that includes said field of view, and then transmits this identified partial region tile to the client terminal 20. The overall reduced tile and the partial region tile are encoded, and this encoded data is then transmitted to the client terminal 20.

[0059] Then, the video viewing unit 201 of the client terminal 20 plays the panoramic video in VR viewing mode using the overall reduced tile and partial region tile transmitted from the video distribution server 10 (step S108). That is, the video viewing unit 201 decodes the data received from the video distribution server 10, displays the partial region tile as a panoramic video, and displays the 2D video thumbnails included in the overall reduced tile as thumbnail videos on the panoramic video. As a result, the client terminal 20 displays the VR viewing screen 1000 as shown in Figure 3.

[0060] • When in 2D viewing mode The video viewing unit 201 of the client terminal 20 sends a 2D viewing mode distribution request to the video distribution server 10 (step S109). Here, the 2D viewing mode distribution request includes information indicating the 2D video currently being viewed and information representing n or more frame times for which tile distribution is requested. However, this information representing n or more frame times is not mandatory and does not have to be included in the 2D viewing mode distribution request.

[0061] The compositing unit 105 of the video distribution server 10 combines the 2D video thumbnail tiles other than the currently viewed 2D video from the 2D video thumbnail tiles for the relevant frame time (i.e., the frame time for which tile distribution is requested) with the panoramic video thumbnail tile for that frame time into the overall reduced tile for that frame time (step S110). Here, as an example, Figure 7 shows a compositing example when there are two 2D video cameras 40, and two 2D video thumbnail tiles are created at a certain frame time, but the user is viewing a 2D video from one of the 2D video cameras 40. As shown in Figure 7, in this case, the compositing is performed by embedding the 2D video thumbnail tile for the 2D video that the user is not viewing (in the example shown in Figure 7, "2D video 1") and the panoramic video thumbnail tile into the thumbnail area of ​​the overall reduced tile. However, at this time, the synthesis unit 105 uses the center coordinates of the field of view included in the 2D viewing mode distribution request or information for identifying said center coordinates to identify a partial region tile that includes said field of view, and then embeds the panoramic video thumbnail tile corresponding to this identified partial region tile into the overall reduced tile.

[0062] However, in step S110 described above, it is not necessary to composite the panoramic video thumbnail onto the overall reduced tile. In this case, the image of the panoramic video area may be used as the thumbnail video for the panoramic video, or a partial area tile containing the user's field of view may be sent to the client terminal 20 and used as the thumbnail video for the panoramic video.

[0063] In the examples shown in Figures 6 and 7, the thumbnail area is located at the bottom edge of the overall reduced tile, but this is just one example. The thumbnail area may also be located at the top edge, left edge, or right edge of the overall reduced tile. Furthermore, multiple thumbnail areas may exist, such as at both the top and bottom edges, or both the left and right edges. In particular, if there are many thumbnails, multiple thumbnail areas may be set.

[0064] Next, the distribution unit 106 of the video distribution server 10 transmits the composite overall reduced tile from step S105 and the 2D video tile for the same frame time to the client terminal 20 (step S111). At this time, the distribution unit 106 identifies the 2D video that the user is currently watching from the information indicating the 2D video currently being watched included in the 2D viewing mode distribution request, and then transmits the 2D video tile of this identified 2D video to the client terminal 20. The overall reduced tile and the 2D video tile are encoded, and this encoded data is transmitted to the client terminal 20.

[0065] Then, the video viewing unit 201 of the client terminal 20 plays a 2D video in 2D viewing mode using the overall reduced tile and 2D video tile transmitted from the video distribution server 10 (step S112). That is, the video viewing unit 201 decodes the data received from the video distribution server 10 and displays the 2D video tile as a 2D video, as well as the 2D video thumbnail tile and panoramic video thumbnail tile included in the overall reduced tile as thumbnail videos. As a result, the client terminal 20 displays a 2D viewing screen 2000 as shown in Figure 3. Note that if the resolution of the 2D video tile is lower than the resolution of the partial area tile (i.e., for example, if the resolution is lower than that of the panoramic video 1001, as in the 2D video 2001 shown in Figure 3), the image of the panoramic video area included in the overall reduced tile may be displayed as the background.

[0066] <Mode switching process> Figure 8 shows the sequence diagram of the mode switching process according to this embodiment. Hereinafter, it is assumed that the user of the client terminal 20 is viewing a panoramic video or a 2D video in either VR viewing mode or 2D viewing mode.

[0067] The video viewing unit 201 of the client terminal accepts a mode switching operation by the user (step S201). A mode switching operation is an operation to switch the viewing mode from VR viewing mode to 2D viewing mode or from 2D viewing mode to VR viewing mode. In this embodiment, this is defined as the operation to select a thumbnail video of a 2D video in VR viewing mode and the operation to select a thumbnail video of a panoramic video in 2D viewing mode. However, any other operation that allows switching the viewing mode may also be considered a mode switching operation.

[0068] Next, if a mode switching operation from VR viewing mode to 2D viewing mode is accepted, steps S202 to S206 are executed, and if a mode switching operation from 2D viewing mode to VR viewing mode is accepted, steps S207 to S211 are executed.

[0069] Switching from VR viewing mode to 2D viewing mode The video viewing unit 201 of the client terminal 20 sends a 2D viewing mode distribution request to the video distribution server 10 (step S202). Here, the 2D viewing mode distribution request also includes information indicating the 2D video selected in the mode switching operation.

[0070] Next, the video viewing unit 201 of the client terminal 20 displays a waiting screen until the 2D video of the thumbnail video selected in the mode switching operation is displayed (step S203).

[0071] The compositing unit 105 of the video distribution server 10, similar to step S110 in Figure 5, combines the 2D video thumbnail tiles other than the 2D video selected in the mode switching operation from among the 2D video thumbnail tiles for the relevant frame time, with the panoramic video thumbnail tile for that frame time, into a reduced overall tile for that frame time (step S204).

[0072] Next, the distribution unit 106 of the video distribution server 10 transmits the composite overall reduced tile from step S204 and the 2D video tile for the same frame time to the client terminal 20, similar to step S111 in Figure 5 (step S205). At this time, the distribution unit 106 identifies the 2D video selected by the user from the information indicating the 2D video selected in the mode switching operation included in the 2D viewing mode distribution request, and then transmits the 2D video tile of this identified 2D video to the client terminal 20. The overall reduced tile and the 2D video tile are encoded, and this encoded data is transmitted to the client terminal 20.

[0073] Then, the video viewing unit 201 of the client terminal 20 plays a 2D video in 2D viewing mode using the overall reduced tile and 2D video tile sent from the video distribution server 10, similar to step S112 in Figure 5 (step S206). As a result, the display transitions from the standby screen to the 2D viewing screen.

[0074] Here, Figure 9 shows an example of the screen transitions of the client terminal 20 when switching from VR viewing mode to 2D viewing mode. As shown in Figure 9, the VR viewing screen 1000 displays a panoramic video 1001 and thumbnail videos 1002 and 1003. For example, if the user selects thumbnail video 1002, the standby screen 1500 is displayed in step S203 above. This standby screen 1500 displays a low-resolution 2D video 1501 and a standby indicator 1502. Note that the background of the low-resolution 2D video 1501 may be an image of the panoramic video area included in the overall reduced tile. Here, the 2D video 1501 is an enlarged version of the 2D video thumbnail tile corresponding to the thumbnail video 1002 selected by the user, scaled to the same resolution as the 2D video tile. When the overall reduced tile and 2D video tile are received from the video distribution server 10, the 2D viewing screen 2000 is displayed. This 2D viewing screen 2000 displays the 2D video 2001 corresponding to the thumbnail video 1002 selected by the user. In this way, while waiting for the 2D video selected by the user to be displayed, a lower-resolution version of that 2D video is shown as a thumbnail tile. This helps to prevent a decrease in the user's viewing quality when switching modes.

[0075] Figure 10 also shows another example of the screen transition of the client terminal 20 when switching from VR viewing mode to 2D viewing mode. As shown in Figure 10, a waiting screen 1600 may be displayed in step S203 above. On this waiting screen 1600, a waiting indicator and a string indicating that the 2D video is being loaded are displayed on area 1601. This reduces the user's viewing quality compared to Figure 9, but makes implementation easier. Note that instead of displaying both the waiting indicator and the string indicating that the 2D video is being loaded, only one of them may be displayed.

[0076] Switching from 2D viewing mode to VR viewing mode The video viewing unit 201 of the client terminal 20 sends a VR viewing mode distribution request to the video distribution server 10 (step S207).

[0077] Next, the video viewing unit 201 of the client terminal 20 displays a waiting screen until the panoramic video of the thumbnail video selected in the mode switching operation is displayed (step S208).

[0078] The compositing unit 105 of the video distribution server 10 composites each 2D video thumbnail tile for the corresponding frame time into the overall reduced tile for that frame time, similar to step S106 in Figure 5 (step S209).

[0079] Next, the distribution unit 106 of the video distribution server 10 transmits the composite overall reduced tile from step S209 and a partial region tile at the same frame time to the client terminal 20, similar to step S107 in Figure 5 (step S210). At this time, the distribution unit 106 uses the center coordinates of the field of view included in the VR viewing mode distribution request or information for identifying said center coordinates to identify the partial region tile that includes said field of view, and then transmits this identified partial region tile to the client terminal 20. The overall reduced tile and the partial region tile are encoded, and this encoded data is then transmitted to the client terminal 20.

[0080] Then, the video viewing unit 201 of the client terminal 20 plays the panoramic video in VR viewing mode using the overall reduced tile and partial area tile sent from the video distribution server 10, similar to step S108 in Figure 5 (step S211). As a result, the display transitions from the standby screen to the VR viewing screen.

[0081] Here, Figure 11 shows an example of the screen transition of the client terminal 20 when switching from 2D viewing mode to VR viewing mode. As shown in Figure 11, the 2D viewing screen 2000 displays a 2D video 2001, a thumbnail video 2002 of a panoramic video, and a thumbnail video 2003 of another 2D video different from the 2D video currently being viewed. For example, if the user selects thumbnail video 2002, the waiting screen 1700 is displayed in step S208 above. On this waiting screen 1700, a waiting indicator 1701 is displayed on a low-quality panoramic video. Here, the low-quality panoramic video is a panoramic video displayed using a full-size reduced tile. Then, when the full-size reduced tile and a partial-area tile are received from the video distribution server 10, the VR viewing screen 1000 is displayed. In this way, the low-quality panoramic video is displayed from the full-size reduced tile until the panoramic video using the partial-area tile is displayed. This makes it possible to suppress the deterioration of the user's viewing quality that occurs when switching modes.

[0082] In Figure 8, the case of switching from VR viewing mode to 2D viewing mode or from 2D viewing mode to VR viewing mode was explained. However, for example, if a thumbnail video of a different 2D video than the one currently being viewed is selected on the 2D viewing screen, a similar waiting screen may be displayed. In other words, in this case, the viewing mode remains 2D, and no viewing mode switch occurs. However, since it is necessary to receive the 2D video tile of the new 2D video selected by the user from the video distribution server 10, a waiting screen similar to that in step S206 or step S208 may be displayed from the time the distribution request for the new 2D video is made until the 2D video tile of the new 2D video is received and displayed.

[0083] [Second Embodiment] Next, a second embodiment will be described. In the first embodiment, a panoramic video represented by the orthographic cylindrical projection method with a vertical angle θ of 0 to π and a horizontal angle φ of 0 to 2π was the distribution target, and the overall reduced tiles and partial region tiles were created from this panoramic video. However, in this embodiment, instead, a region called the region of interest (ROI) in the panoramic video will be the distribution target, and the case of creating the overall reduced tiles and partial region tiles from the region of interest will be described. This is because generally, in a 360°×180° panoramic video, what viewers are interested in is often a partial region where the subject is shown. Hereinafter, the panoramic video representing the region of interest will be referred to as the "ROI panoramic video".

[0084] Note that in the second embodiment, mainly, the differences from the first embodiment will be described, and the description of the components that are the same or similar to those in the first embodiment will be omitted.

[0085] <ROI panoramic video> As shown in FIG. 12, the ROI panoramic video (more precisely, its frame image) can be considered as a partial region of a 360°×180° virtual panoramic video (more precisely, its frame image). FIG. 12 shows an example where the ROI panoramic video is a 240°×120° region (that is, when expressed by the orthographic cylindrical projection method, the region with a vertical angle of 30° to 150° and a horizontal angle of 60° to 300° with respect to the virtual panoramic video) in the 360°×180° virtual panoramic video.

[0086] In the following, the resolution of a 360°×180° virtual panoramic video (hereinafter also referred to as the virtual panoramic video) is defined as ROIFULLEQWIDTH×ROIFULLEQHEIGHT(pixels), the resolution of the ROI panoramic video is defined as RESOLUTIONWIDTH×RESOLUTIONHEIGHT(pixels), and the resolution of a partial region tile is defined as VIEWWIDTH×VIEWHEIGHT(pixels). Furthermore, the offsets of the top-left vertex coordinates of the ROI panoramic video to the top-left vertex coordinates of the virtual panoramic video are defined as ROILEFTINFULLEQ and ROITOPINFULLEQ. ROILEFTINFULLEQ is the offset of the horizontal component value of the top-left vertex coordinates of the ROI panoramic video to the horizontal component value of the top-left vertex coordinates of the virtual panoramic video, and ROITOPINFULLEQ is the offset of the vertical component value of the top-left vertex coordinates of the ROI panoramic video to the vertical component value of the top-left vertex coordinates of the virtual panoramic video. However, the horizontal rightward and vertical downward directions are considered positive directions for the coordinates.

[0087] At this time, when the video distribution server 10 sends a reduced overall tile, it also sends four parameters (ROIFULLEQWIDTH, ROIFULLEQHEIGHT, ROILEFTINFULLEQ, and ROITOPINFULLEQ) that represent the width and height of the ROI panoramic video, and the coordinates of the left edge and top edge of the ROI panoramic video in the virtual panoramic video, respectively. However, when sending a reduced overall tile of a normal 360°×180° panoramic video instead of an ROI panoramic video, the above four parameters are sent with default values ​​set. Here, the default values ​​for ROIFULLEQWIDTH and ROIFULLEQHEIGHT are the maximum resolution, and the default values ​​for ROILEFTINFULLEQ and ROITOPINFULLEQ are 0.

[0088] More specifically, when sending a full-size reduced tile of a normal 360° x 180° panoramic video, the following settings are used: ROIFULLEQWIDTH=maximum resolution width, ROIFULLEQHEIGHT=maximum resolution height, ROILEFTINFULLEQ=0, ROITOPINFULLEQ=0. On the other hand, when sending a full-size reduced tile of an ROI panoramic video, the following settings are used: ROIFULLEQWIDTH=maximum resolution width, ROIFULLEQHEIGHT=maximum resolution height, ROILEFTINFULLEQ=coordinates of the left edge of the full-size reduced tile of the ROI panoramic video, and ROITOPINFULLEQ=coordinates of the top edge of the full-size reduced tile of the ROI panoramic video.

[0089] This ensures backward compatibility if the program implementing the video viewing unit 201 of the client terminal 20 supports playback of ROI panoramic videos.

[0090] Alternatively, instead of sending the RESOLUTIONWIDTH and ROIFULLEQHEIGHT of the ROI panoramic video, you may send, for example, the ratio of ROIFULLEQWIDTH to VIEWWIDTH, and the ratio of ROIFULLEQHEIGHT to VIEWHEIGHT instead of RESOLUTIONHEIGHT. This is because the extent to which sub-region tiles of each resolution overlap is predetermined, so RESOLUTIONWIDTH can be expressed as the ratio of ROIFULLEQWIDTH to VIEWWIDTH, and ROIFULLEQHEIGHT as the ratio of ROIFULLEQHEIGHT to VIEWHEIGHT. This ratio may also be called, for example, the "reduction ratio" of the ROI panoramic video relative to the virtual panoramic video.

[0091] <Video streaming processing> Figure 13 shows a sequence diagram of the video distribution process according to this embodiment. Here, steps S301 to S302 are repeatedly executed for each frame time width of the panoramic video image, and steps S303 to S304 are repeatedly executed for each frame time width of the 2D video image. Below, steps S301 to S304 at a certain frame time t will be described.

[0092] The panoramic video creation unit 101 of the video distribution server 10 creates an ROI panoramic video (more precisely, its frame image) using each panoramic video capture image at frame time t (step S301). Here, the area to be used as the region of interest can be arbitrarily determined, but for example, it is conceivable to create a 240°×120° ROI panoramic video by selecting the 240°×120° region as the region of interest within a 360°×180° panoramic video. This is because the upper and lower parts (0°~30° and 150°~180°) and the rear parts (0°~60° and 300°~360°) are generally not viewed by users. However, this is just one example, and it is possible to select any region within 360°×180° as the region of interest and create an ROI panoramic video representing that region of interest.

[0093] Next, the panoramic video tile creation unit 102 of the video distribution server 10 creates a full-scaled tile, a partial-region tile, and a panoramic video thumbnail tile for frame time t using the ROI panoramic video created in step S301 (step S302). Here, this full-scaled tile is an image obtained by compressing the entire ROI panoramic video (more precisely, the frame images of the panoramic video) represented in equirectangular projection to a predetermined resolution. Otherwise, it is the same as in the first embodiment, and the full-scaled tile contains a panoramic video region and a thumbnail region.

[0094] Furthermore, the partial region tile is an image with the same resolution as the overall reduced tile, and represents a partial region of the ROI panoramic video (more precisely, its frame images). Aside from representing a partial region of the ROI panoramic video, it is the same as in the first embodiment.

[0095] The subsequent steps S303 to S304 and the subsequent processing are the same as in the first embodiment.

[0096] As described above, the video distribution server 10 according to this embodiment creates an ROI panoramic video instead of a 360° x 180° panoramic video, and then creates overall reduced tiles and partial region tiles from this ROI panoramic video. This makes it possible to create an ROI panoramic video with higher image quality than a 360° x 180° panoramic video, resulting in higher image quality for the overall reduced tiles and further improving the viewing quality for the user.

[0097] Furthermore, for example, when creating an ROI panoramic video with the same or comparable image quality as a 360° x 180° panoramic video, it is possible to reduce the overall data size of the reduced tiles and to reduce the number of cameras 30 used for panoramic video, thereby reducing camera introduction or installation costs.

[0098] In this embodiment, the ROI panoramic video was created based on the first embodiment, but it is not necessary to assume the first embodiment. In other words, this embodiment can also be applied when distributing and viewing panoramic videos without assuming the existence of 2D videos.

[0099] [Differentiation] The following describes some modified examples of each of the embodiments described above.

[0100] • Variation 1 When creating an ROI panoramic video in step S301 of Figure 13, the panoramic video creation unit 101 may embed some content (for example, CG (computer graphics), artist or team logos, etc.) in areas other than the area of ​​interest. Specifically, if the area of ​​interest is 240° x 120° within a 360° x 180° panoramic video, the panoramic video creation unit 101 may embed some content in areas other than the area of ​​interest within the 360° x 180° panoramic video. This is expected to prevent situations where nothing is displayed when the user views areas other than the area of ​​interest, thereby suppressing a decline in the user's viewing quality.

[0101] • Variation 2 In the above modification 1, some content was embedded in an area outside the area of ​​interest, but it is also acceptable to embed some content within the area of ​​interest, rather than outside it. Specifically, if the area of ​​interest is defined as 240° x 120° in a 360° x 180° panoramic video, some content may be embedded within that area of ​​interest. In particular, in this case, the content may be embedded within a sub-area that forms a sub-area tile within the area of ​​interest.

[0102] Furthermore, by combining Modification 2 and Modification 1, some kind of content may be embedded in both the area of ​​interest and the other areas.

[0103] • Modification example 3 While partial region tiles are distributed to the client terminal 20 during VR viewing mode, for example, if there is sufficient bandwidth available, sufficient hardware resources on the client terminal 20 side, and sufficient image quality can be expected even when playing panoramic videos using the overall reduced-size tiles, then it is not necessary to distribute partial region tiles to the client terminal 20. In other words, the resolution of the overall reduced-size tiles may be higher than the resolution of the partial region tiles. Thus, the resolution of the overall reduced-size tiles (especially the resolution of the panoramic video area contained within them) may be set to the same resolution as the partial region tiles, or to a higher resolution, or changed as appropriate.

[0104] • Modification 4 The video played when a thumbnail video is selected is assumed to be a video shot with the 2D video camera 40, but is not limited to this. For example, it may be non-live-action such as animation or CG, and its content may be an advertisement or content related to panoramic video (for example, a video explaining the rules of a sport).

[0105] • Variation 5 In this embodiment, different overall reduction tiles were combined depending on whether the client terminal 20 was in VR viewing mode or 2D viewing mode. However, if there is only one panoramic video that the user can view, an overall reduction tile like the one shown in Figure 6 may be created as a common overall reduction tile for both VR viewing mode and 2D viewing mode. This is because when displaying a thumbnail video of the panoramic video, it is sufficient to use the image of the area for the panoramic video.

[0106] • Modification 6 In this embodiment, the thumbnail tiles for the panoramic video correspond to the user's field of view, but they do not necessarily have to correspond to the user's field of view. For example, thumbnail tiles for the panoramic video that correspond to the field of view of a virtual agent other than the user may be created.

[0107] · Modification example 7 In this embodiment, when multiple thumbnail videos are displayed, all of them are 2D video thumbnails, or one is a panoramic video thumbnail and the rest are 2D video thumbnails. However, it is not limited to these, and for example, all of the multiple thumbnail videos may be panoramic video thumbnails.

[0108] The present invention is not limited to the embodiments specifically disclosed above, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims. [Explanation of symbols]

[0109] 1. Video distribution system 10 Video streaming servers 20 client terminals 30 Panoramic video cameras 40 2D video cameras 101 Panoramic Video Creation Department 102 Panoramic Video Tile Creation Section 103 2D Video Creation Department 104 2D Video Tile Creation Section 105 Synthesis part 106 Distribution Department N1 Communication Network N2 Communication Network

Claims

1. A first creation unit creates a first overall image by converting a first frame image constituting a first video to a first width and a first height that are smaller than the width and height of the first frame image, respectively. A second creation unit creates a thumbnail image by converting a second frame image that constitutes a second video which will be a thumbnail video for the first video to a second width and second height that are smaller than the width and height of the second frame image and smaller than the width and height of the first video, respectively. A compositing unit creates a second overall image by combining the aforementioned thumbnail image with the first overall image, The distribution unit distributes the second overall image to the user terminal, A video distribution system that has the following features.

2. A plurality of sub-region images representing each of a plurality of pre-set sub-regions for the first frame image, and having a third creation unit that creates a plurality of sub-region images with a first width and a first height, The aforementioned distribution unit, The video distribution system according to claim 1, wherein when the first video is played on the user terminal, a partial region image from among the plurality of partial region images corresponding to the display range of the user terminal when the first video is played is further distributed to the user terminal.

3. The second video mentioned above includes a 2D video. It has a fourth creation unit that creates a 2D image in which the second frame image constituting the 2D video has a predetermined third width and third height, The aforementioned distribution unit, The video distribution system according to claim 1 or 2, wherein when the 2D video is played on the user terminal, the 2D image is further distributed to the user terminal.

4. The second creation unit described above is: One or more thumbnail images are created by converting one or more second frame images constituting each of the one or more second videos to the second width and the second height, respectively. The aforementioned synthesis section is A video distribution system according to any one of claims 1 to 3, comprising creating a second overall image by combining one or more thumbnail images with the first overall image.

5. The first or second video includes at least one panoramic video and zero or more 2D videos. The second creation unit described above is: The video distribution system according to claim 4, wherein one or more thumbnail images are created by converting one or more second frame images constituting each of the videos other than the video played on the user terminal from the one or more second videos to the second width and second height, respectively.

6. The second width and the second height are both smaller than the first width and the first height, The aforementioned synthesis section is A video distribution system according to any one of claims 1 to 5, wherein the thumbnail image is embedded in the lower, upper, left, or right edge of the first overall image to composite the thumbnail image with the first overall image.

7. The video distribution system according to any one of claims 1 to 6, wherein the first video includes a panoramic video.

8. A first creation unit creates a first overall image by converting a first frame image constituting a first video to a first width and a first height that are smaller than the width and height of the first frame image, respectively. A second creation unit creates a thumbnail image by converting a second frame image that constitutes a second video which will be a thumbnail video for the first video to a second width and second height that are smaller than the width and height of the second frame image and smaller than the width and height of the first video, respectively. A compositing unit creates a second overall image by combining the aforementioned thumbnail image with the first overall image, The distribution unit distributes the second overall image to the user terminal, A video distribution device having the following features.

9. A first video playback device, When a display unit receives a second overall image obtained by combining a first overall image obtained by converting a first frame image constituting the first video to a predetermined first width and first height, and a thumbnail image obtained by converting a second frame image constituting a second video that will be a thumbnail video for the first video to a second width and second height smaller than the width and height of the second frame image, and smaller than the first width and first height, the display unit displays the video composed of the first video and the thumbnail image. A terminal.

10. The aforementioned display unit is If the first video is a panoramic video, the video composed of the thumbnail images is displayed in an area corresponding to one or more parts of the top, bottom, left, and right sides of the first video. The terminal according to claim 9, wherein, if the first video is a 2D video, the terminal displays the first video on a portion of the background with the panoramic video as the background, and displays the video composed of the thumbnail images in areas corresponding to one or more portions of the top, bottom, left, and right sides of the panoramic video.

11. A first creation procedure for creating a first overall image by converting a first frame image constituting a first video to a first width and a first height that are smaller than the width and height of the first frame image, respectively, A second creation procedure for creating a thumbnail image by converting a second frame image that constitutes a second video which will be a thumbnail video for the first video to a second width and second height that are smaller than the width and height of the second frame image and smaller than the width and height of the first video, respectively, A synthesis procedure for creating a second overall image by combining the aforementioned thumbnail image with the first overall image, The distribution procedure for delivering the second overall image to the user terminal, How a computer executes it.

12. The first device capable of playing the video is A display procedure that, upon receiving a second overall image obtained by combining a first overall image obtained by converting a first frame image constituting the first video to a predetermined first width and first height, and a thumbnail image obtained by converting a second frame image constituting a second video that will be a thumbnail video for the first video to a second width and second height smaller than the width and height of the second frame image, and smaller than the first width and first height, respectively, displays the video composed of the first video and the thumbnail image. How to do it.

13. A first creation procedure for creating a first overall image by converting a first frame image constituting a first video to a first width and a first height that are smaller than the width and height of the first frame image, respectively, A second creation procedure for creating a thumbnail image by converting a second frame image that constitutes a second video which will be a thumbnail video for the first video to a second width and second height that are smaller than the width and height of the second frame image and smaller than the width and height of the first video, respectively, A synthesis procedure for creating a second overall image by combining the aforementioned thumbnail image with the first overall image, The distribution procedure for delivering the second overall image to the user terminal, A program that causes a computer to execute something.

14. On a device capable of playing the first video, A display procedure that, upon receiving a second overall image obtained by combining a first overall image obtained by converting a first frame image constituting the first video to a predetermined first width and first height, and a thumbnail image obtained by converting a second frame image constituting a second video that will be a thumbnail video for the first video to a second width and second height smaller than the width and height of the second frame image, and smaller than the first width and first height, respectively, displays the video composed of the first video and the thumbnail image. A program that executes the command.

Images