Video distribution system, video distribution device, method, and program

Abstract

To create a panoramic moving image in an interest area.SOLUTION: A moving image distribution system according to an embodiment comprises: a first creation unit which creates a frame image constituting a moving image indicative of a predetermined interest area by using one or more photographic images photographed with each of one or more cameras; a second creation unit which creates an entire image obtained by converting the frame image to a predetermined resolution; and a distribution unit which distributes the entire image to a user terminal.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to a video distribution system, a video distribution device, a method, and a program. [Background technology]

[0002] In recent years, videos that can be viewed in all directions (360 degrees) have become known, and such videos are called panoramic videos (also called "panoramic videos," "360-degree panoramic videos," "VR (Virtual Reality) videos," "VR videos," "3D videos," etc.). On the other hand, regular videos that are not panoramic videos are also called "2D videos" in contrast to panoramic videos.

[0003] In addition, for example, a video distribution server may stream a panoramic video to a client terminal, allowing the client terminal to view the panoramic video. However, since panoramic videos generally have large data sizes, it may take time to distribute the panoramic video, and it may take time for the panoramic video to be displayed on the client terminal.

[0004] In response to this, a technique is known in which the range displayed on the client terminal (hereinafter also referred to as the "field of view range") is delivered in normal image quality, and the range not displayed on the client terminal is delivered in low image quality (for example, Non-Patent Document 1). By using this technique, when watching a panoramic video via streaming delivery or the like, it is possible to reduce the time required to deliver the panoramic video while suppressing a decrease in the user's viewing quality. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Daisuke Ochi and Shinnosuke Iwaki, "Real-time spherical video distribution system (Special feature: Dwango x NTT R&D collaboration)," NTT Technical Journal 27(4), 51-54, April 2015 Summary of the Invention [Problem to be solved by the invention]

[0006] However, content that is often viewed as panoramic video, such as sports, concerts, and plays, does not necessarily require 360° x 180° panoramic video. For example, in content such as concerts and plays, viewers are generally interested in the stage and its surrounding area, and are not interested in the area behind the audience seats. Therefore, by excluding areas that do not interest the viewer and creating a panoramic video of the area that is of interest, it is possible to reduce the data size of the panoramic video and achieve higher image quality without degrading the user's viewing quality.

[0007] An embodiment of the present invention has been made in view of the above points, and aims to create a panoramic video of a region of interest. [Means for solving the problem]

[0008] In order to achieve the above object, a video distribution system according to one embodiment includes a first creation unit that creates frame images constituting a video representing a predetermined region of interest using one or more captured images taken by each of one or more cameras, a second creation unit that creates an overall image by converting the frame images to a predetermined resolution, and a distribution unit that distributes the overall image to a user terminal. [Effects of the Invention]

[0009] You can create panoramic videos of areas of interest. [Brief explanation of the drawings]

[0010] [Figure 1] 10A and 10B are diagrams illustrating an example of a partial region tile and an entire reduced tile. [Figure 2] 1 is a diagram illustrating an example of the overall configuration of a video distribution system according to a first embodiment. [Figure 3]FIG. 1 is a diagram illustrating an example of a VR viewing mode and a 2D viewing mode. [Figure 4] FIG. 1 is a diagram illustrating an example of a functional configuration of a video distribution system according to a first embodiment. [Figure 5] FIG. 10 is a sequence diagram illustrating an example of a video distribution process according to the first embodiment. [Figure 6] FIG. 10 is a diagram (part 1) for explaining an example of combining a reduced-size overall tile for a panoramic video with a thumbnail tile. [Figure 7] FIG. 10 is a diagram (part 2) for explaining an example of combining reduced-size tiles for a panoramic video with thumbnail tiles. [Figure 8] FIG. 10 is a sequence diagram illustrating an example of a mode switching process according to the first embodiment. [Figure 9] FIG. 10 is a diagram (part 1) for explaining an example of switching from VR viewing mode to 2D viewing mode. [Figure 10] FIG. 10 is a diagram (part 2) for explaining an example of switching from the VR viewing mode to the 2D viewing mode. [Figure 11] FIG. 10 is a diagram illustrating an example of switching from a 2D viewing mode to a VR viewing mode. [Figure 12] FIG. 10 is a diagram illustrating an example of a ROI panoramic video. [Figure 13] FIG. 10 is a sequence diagram illustrating an example of a video distribution process according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, a first embodiment and a second embodiment will be described as embodiments of the present invention.

[0012] [First embodiment] First, a first embodiment will be described. In this embodiment, a video distribution system 1 that allows viewing of a panoramic video and a 2D video while switching between them will be described.

[0013] <How to distribute panoramic videos> As a prerequisite technology for this embodiment, a method for delivering panoramic videos will be described, which is described in Non-Patent Document 1. In the technology described in Non-Patent Document 1, the field of view displayed on the client terminal is delivered in normal image quality, and the other ranges are delivered in low image quality.

[0014] For example, as shown in FIG. 1, in a panoramic video displayed on a full-dome projection sphere, the observation point is the center of the full-dome projection sphere, and the field of view is the range captured by a pseudo camera device at this observation point at a predetermined angle of view. Furthermore, panoramic videos displayed on a full-dome projection sphere can be expressed using equirectangular projection with a vertical angle θ of 0 to π and a horizontal angle φ of 0 to 2π. In this case, the technology described in Non-Patent Document 1 delivers a partial area including the field of view of a panoramic video (more precisely, a frame image of the panoramic video) expressed using equirectangular projection to a client terminal as a "partial area tile," and a compressed image of the entire panoramic video as a "total reduced tile." Using this technology reduces the time required to deliver a panoramic video, enables high-quality video viewing within the field of view, and ensures uninterrupted video viewing even when the field of view is moved. Note that "compression" here refers to reducing the image size.

[0015] Although the term "field of view range" is used above, this may also be called, for example, "visible range," "display range," or "viewing range."

[0016] <Overall configuration of video distribution system 1> The overall configuration of a video distribution system 1 according to this embodiment is shown in Fig. 2. As shown in Fig. 2, the video distribution system 1 according to this embodiment includes a video distribution server 10, one or more client terminals 20, one or more panoramic video cameras 30, and one or more 2D video cameras 40. The video distribution server 10 and each client terminal 20 are communicatively connected via a communications network N1 such as the Internet. The video distribution server 10, each panoramic video camera 30, and each 2D video camera 40 are communicatively connected via a communications network N2 such as a LAN (Local Area Network).

[0017] The video distribution server 10 is a computer or computer system that streams panoramic videos and 2D videos in response to requests from the client terminal 20. At this time, the video distribution server 10 distributes the panoramic videos and 2D videos to the client terminal 20 so that the panoramic videos and 2D videos can be viewed while switching between them.

[0018] The client terminal 20 is a terminal capable of viewing panoramic videos and 2D images. For example, a smartphone, a tablet terminal, a head-mounted display, a wearable device, a PC (personal computer), etc. can be used as the client terminal 20. A user can use the client terminal 20 to view panoramic videos and 2D videos while switching between them.

[0019] The user can move or change the field of view range in the panoramic video by operating the client terminal 20. Such operations may be, for example, a swipe or flick operation on a touch panel provided on the client terminal 20, or pressing a button to move the field of view range, or pressing a button to move the field of view range using a pointing device such as a mouse. Furthermore, if the client terminal 20 is provided with a sensor that can detect orientation in all directions (360 degrees) up, down, left, and right, the field of view range may be moved or changed by changing the orientation of the client terminal 20.

[0020] The panoramic video camera 30 is a photographing device such as a camera that generates photographed images for creating a panoramic video (hereinafter also referred to as photographed images for a panoramic video). The panoramic video camera 30 may be a normal camera, a camera equipped with a wide-angle lens such as a fisheye lens, or a camera dedicated to photographing panoramic videos (for example, a camera equipped with multiple wide-angle lenses such as fisheye lenses).

[0021] The 2D moving image camera 40 is a photographing device such as a camera that generates photographed images for creating a 2D moving image (hereinafter also referred to as photographed images for a 2D moving image).

[0022] 1 is an example, and other configurations may be used. For example, if sound, audio, etc. are also to be distributed to the client terminal 20, a sound collection device such as a microphone may be included in the video distribution system 1.

[0023] Furthermore, when panoramic videos or 2D videos are distributed on demand rather than through streaming, a storage device for storing panoramic videos or 2D videos may be included in the video distribution system 1. The following description will be primarily based on the assumption that live content such as sports, concerts, and plays is distributed through streaming, but the following description is equally applicable to cases where the content is recorded and distributed on demand.

[0024] <Viewing mode> Here, the client terminal 20 has a "VR viewing mode" for the user to view panoramic videos and a "2D viewing mode" for the user to view 2D videos. The "VR viewing mode" may also be called, for example, a "panoramic viewing mode" or a "3D viewing mode."

[0025] 3, in VR viewing mode, a VR viewing screen 1000 is displayed on the client terminal 20. On this VR viewing screen 1000, a panoramic video 1001 is displayed from a partial region tile (or a reduced-size entire tile), and thumbnail videos 1002 to 1003 of 2D videos are displayed on the panoramic video 1001. When the thumbnail video 1002 or 1003 of 2D videos is selected by the user, the mode transitions from VR viewing mode to 2D viewing mode.

[0026] 3, in 2D viewing mode, a 2D viewing screen 2000 is displayed on the client terminal 20. This 2D viewing screen 2000 displays a 2D video 2001 selected by the user, as well as a thumbnail video 2002 of a panoramic video and a thumbnail video 2003 of another 2D video different from the currently displayed 2D video 2001. If the thumbnail video 2002 of the panoramic video is selected by the user, the viewing mode transitions from 2D to VR viewing mode, and if the thumbnail video 2003 of another 2D video is selected, the other 2D video is displayed while remaining in 2D viewing mode. The 2D video 2001, thumbnail videos 2002, and 2003 are displayed on the panoramic video represented by the overall reduced tile.

[0027] In this way, the client terminal 20 according to this embodiment displays thumbnail videos of 2D videos in VR viewing mode, and displays thumbnail videos of panoramic videos and other 2D videos in 2D viewing mode. When a thumbnail video is selected by the user, the video corresponding to that thumbnail video (panoramic video or 2D video) is displayed. This allows the user to easily switch between viewing panoramic videos and 2D videos. Note that thumbnail videos may also be called, for example, "auxiliary videos" or "auxiliary images."

[0028] Therefore, for example, when a sports broadcast provides panoramic video that overlooks the entire stadium, and 2D video, such as video from a player's point of view or video from a camera fixedly installed at the stadium, users can watch the game by switching between the panoramic video and the 2D video.

[0029] 3, there are two thumbnail videos in the VR viewing screen 1000 and the 2D viewing screen 2000, but this is just an example, and the number of thumbnail videos can be any number. Also, the number of thumbnail videos may be set by the user, or the number of thumbnail videos may be different between the VR viewing screen 1000 and the 2D viewing screen 2000.

[0030] 3, thumbnail videos are displayed at the bottom of the VR viewing screen 1000 and the 2D viewing screen 2000, but this is just one example and they may be displayed at the top, left, or right. Also, thumbnail videos may be displayed dispersedly in two or more areas: the top, bottom, left, and right. However, according to the human visual field characteristics that allow a wider range to be seen in the left-right direction than in the up-down direction, it is preferable to display thumbnail videos at the bottom or top (particularly the bottom).

[0031] In addition, thumbnail videos do not need to be displayed all the time on the VR viewing screen 1000 and the 2D viewing screen 2000. For example, thumbnail videos may be normally hidden and displayed when some user operation is detected or when a new thumbnail video is additionally distributed. Furthermore, a thumbnail video that has been displayed once may be hidden again, for example, if no user operation is detected for a certain period of time.

[0032] In the following, we will mainly assume that a user can view one panoramic video, but the user may also be able to view multiple panoramic videos. For example, if a first panoramic video and a second panoramic video are viewable, there may be a first VR viewing mode for viewing the first panoramic video and a second VR viewing mode for viewing the second panoramic video, and the user may be able to switch between the first VR viewing mode, the second VR viewing mode, and the 2D viewing mode. In this case, a thumbnail video of the second panoramic video is also displayed in the first VR viewing mode, and similarly, a thumbnail video of the first panoramic video is also displayed in the second VR viewing mode.

[0033] <Functional configuration of video distribution system 1> FIG. 4 shows the functional configuration of the video distribution system 1 according to this embodiment.

[0034] <Video Streaming Server 10> 4, the video distribution server 10 according to this embodiment has, as functional units, a panoramic video creation unit 101, a panoramic video tile creation unit 102, a 2D video creation unit 103, a 2D video tile creation unit 104, a composition unit 105, and a distribution unit 106. Each of these functional units is realized, for example, by processing executed by a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) by one or more programs installed in the video distribution server 10.

[0035] The panoramic video creation unit 101 creates a panoramic video using each captured image for the panoramic video acquired from the panoramic video camera 30. That is, the panoramic video creation unit 101 decodes each captured image for the panoramic video for each frame time width, and then creates the panoramic video by stitching together the captured images for the panoramic video using a known image processing technique called stitching. Note that the frame time width refers to the generation interval between captured images, and is the reciprocal of the frame rate.

[0036] The panoramic video tile creation unit 102 uses the panoramic video created by the panoramic video creation unit 101 to create whole reduced tiles, partial area tiles, and thumbnail tiles (hereinafter also referred to as thumbnail tiles for panoramic video) for displaying thumbnail videos of the panoramic video in 2D viewing mode.

[0037] Here, the overall reduced tile refers to an image obtained by compressing the entire panoramic video (more precisely, frame images of the panoramic video) expressed in equirectangular projection to a predetermined resolution (i.e., a predetermined width and height). However, unlike the overall reduced tile described in Non-Patent Document 1, in this embodiment, the overall reduced tile has an area for the panoramic video and an area for thumbnails, and an image obtained by compressing the panoramic video expressed in equirectangular projection at a predetermined resolution corresponds to the area for the panoramic video. Note that if an effective area is defined in the overall reduced tile, the effective area may be used as the area for the panoramic video, and the other ineffective area may be used as the area for thumbnails. Furthermore, in the following, the term "reducing" will be used consistently to refer to compressing the resolution of an image. However, since the overall reduced tile can also be considered an image obtained by converting the entire panoramic video to a predetermined resolution, the term "converting" may be used instead of "reducing."

[0038] A partial area tile is an image with the same resolution as the entire reduced tile, and represents a partial area of ​​the panoramic video (more precisely, its frame image). For each frame image of the panoramic video, a partial area tile is created so as to cover the entire frame image.

[0039] For example, if the resolution of each frame image of a panoramic video is 11,520 x 6,480 (pixels), the resolution of the partial region tiles is 1,920 x 1,080 (pixels), and partial region tiles are created with an overlap of 960 (pixels) horizontally and 540 (pixels) vertically, 11 x 11 = 121 partial region tiles are created for each frame image of the panoramic video. Note that this is just one example, and the degree of overlap of partial region tiles with each resolution created for each frame image can be set as appropriate. In general, creating partial region tiles with a greater overlap can provide users with higher viewing quality, but this requires more computational resources, memory, etc.

[0040] A thumbnail tile for a panoramic video is an image obtained by compressing a partial region tile to a predetermined resolution for a thumbnail.

[0041] The 2D moving image creation unit 103 creates each 2D moving image (more precisely, its frame images) using each captured image for the 2D moving image acquired from the 2D moving image camera 40. That is, the 2D moving image creation unit 103 creates each 2D moving image by decoding each captured image for the 2D moving image for each frame time width.

[0042] The 2D video tile creation unit 104 uses the 2D video created by the 2D video creation unit 103 to create 2D video tiles for displaying the 2D video and thumbnail tiles (hereinafter also referred to as 2D video thumbnail tiles) for displaying thumbnail videos of the 2D video in VR viewing mode or 2D viewing mode.

[0043] Here, a 2D video tile is an image with a resolution equal to or lower than that of a partial region tile. 2D video tiles are created for each 2D video. A thumbnail tile for a 2D video is an image with the same resolution as a thumbnail tile for a panoramic video, compressed to a predetermined resolution for thumbnails.

[0044] When the client terminal 20 is in VR viewing mode, the compositing unit 105 composites 2D video thumbnail tiles for each 2D video into the thumbnail area of ​​the reduced-size tile. When the client terminal 20 is in 2D viewing mode, the compositing unit 105 composites 2D video thumbnail tiles for 2D videos other than the 2D video currently being viewed and thumbnail tiles for the panoramic video into the thumbnail area of ​​the reduced-size tile.

[0045] When the client terminal 20 is in the VR viewing mode, the distribution unit 106 distributes the entire reduced-size tile and a partial area tile according to the field of view of the client terminal 20. When the client terminal 20 is in the 2D viewing mode, the distribution unit 106 distributes the entire reduced-size tile and a 2D video tile of the 2D video selected by the user (i.e., the 2D video currently being viewed by the user).

[0046] <Client terminal 20> 4, the client terminal 20 according to this embodiment has, as a functional unit, a video viewing unit 201. The functional unit is realized, for example, by a process in which one or more programs installed in the client terminal 20 are executed by a processor such as a CPU or a GPU.

[0047] The video viewing unit 201 transmits a delivery request to the video distribution server 10 to display a panoramic video or a 2D video and a thumbnail video according to the current viewing mode. Here, the delivery request includes, for example, information indicating the current viewing mode and the center coordinates of the field of view range or information for specifying the center coordinates. Examples of information for specifying the center coordinates of the field of view range include the coordinates of each vertex of the field of view range and the coordinates of the diagonal vertices.

[0048] Furthermore, the video viewing unit 201 displays a panoramic video or a 2D video and a thumbnail video using tiles (reduced-size tiles and partial area tiles or 2D video tiles) distributed from the video distribution server 10.

[0049] <Video distribution processing> A sequence diagram of the video distribution process according to this embodiment is shown in Fig. 5. Here, the following steps S101 to S102 are repeatedly executed for each frame time width of images shot for the panoramic video, and steps S103 to S104 are repeatedly executed for each frame time width of images shot for the 2D video. Below, steps S101 to S104 for a certain frame time t will be explained. Note that frame time t refers to the time expressed as t = t0 + Δt, t0 + 2Δt, t0 + 3Δt, ... where t0 is the shooting start time and Δt is the frame time width.

[0050] The panoramic moving image creating unit 101 of the moving image distribution server 10 creates a panoramic moving image (more precisely, its frame images) using each captured image for the panoramic moving image at frame time t (step S101).

[0051] Next, the panoramic video tile creation unit 102 of the video distribution server 10 uses the panoramic video created in step S101 above to create a whole reduced tile, a partial region tile, and a thumbnail tile for the panoramic video at frame time t (step S102).

[0052] The 2D moving image creating unit 103 of the moving image distribution server 10 creates each 2D moving image (more precisely, its frame images) using each 2D moving image shot image at frame time t (step S103).

[0053] Next, the 2D video tile creation unit 104 of the video distribution server 10 creates a 2D video tile for frame time t and a 2D video thumbnail tile using each of the 2D videos created in step S103 (step S104). For example, if there are M 2D video cameras 40 and 2D video captured images are obtained from each of these M 2D video cameras 40, M 2D video tiles and M 2D video thumbnail tiles will be created.

[0054] Next, steps S105 to S108 are executed if the viewing mode of the client terminal 20 is VR viewing mode, and steps S109 to S112 are executed if the viewing mode is 2D viewing mode. The following steps S105 to S108 and steps S109 to S112 are executed repeatedly every n×frame time width, where n is a preset integer of 1 or greater, while the viewing mode does not switch.

[0055] - When in VR viewing mode The video viewing unit 201 of the client terminal 20 transmits a VR viewing mode distribution request to the video distribution server 10 (step S105). Here, the VR viewing mode distribution request includes information representing n or more frame times for which tile distribution is requested. However, this information representing n or more frame times is not essential and does not have to be included in the VR viewing mode distribution request.

[0056] The compositing unit 105 of the video distribution server 10 composites each thumbnail tile for the 2D video at the corresponding frame time (i.e., the frame time at which tile distribution is requested) with the entire reduced-size tile for that frame time (step S106). Note that, for example, if the user can view multiple panoramic videos, the compositing unit 105 further composites thumbnail tiles for panoramic videos other than the panoramic video corresponding to the entire reduced-size tile with the entire reduced-size tile.

[0057] As an example, Fig. 6 shows a compositing example in which two 2D video cameras 40 are used and two 2D video thumbnail tiles are created at a certain frame time. As shown in Fig. 6, in this case, the 2D video thumbnail tiles are combined by embedding them in the thumbnail area of ​​the overall reduced tile.

[0058] Next, the distribution unit 106 of the video distribution server 10 transmits the overall reduced tile after composition in step S106 above and the partial area tile of the same frame time to the client terminal 20 (step S107). At this time, the distribution unit 106 uses the center coordinates of the field of view range included in the VR viewing mode distribution request or information for identifying the center coordinates to identify the partial area tile that includes the field of view, and then transmits this identified partial area tile to the client terminal 20. Note that the overall reduced tile and the partial area tile are encoded, and this encoded data is transmitted to the client terminal 20.

[0059] Then, the video viewing unit 201 of the client terminal 20 plays back the panoramic video in VR viewing mode using the entire reduced tile and the partial area tile transmitted from the video distribution server 10 (step S108). That is, the video viewing unit 201 decodes the data received from the video distribution server 10, and displays the partial area tiles as a panoramic video, and also displays the 2D video thumbnail included in the entire reduced tile as a thumbnail video on the panoramic video. As a result, a VR viewing screen 1000 such as that shown in FIG. 3 is displayed on the client terminal 20.

[0060] When in 2D viewing mode Video viewing unit 201 of client terminal 20 sends a 2D viewing mode delivery request to video delivery server 10 (step S109). The 2D viewing mode delivery request includes information indicating the 2D video currently being viewed and information indicating n or more frame times for which tile delivery is requested. However, this information indicating n or more frame times is not essential and does not have to be included in the 2D viewing mode delivery request.

[0061] The compositing unit 105 of the video distribution server 10 composites, into the overall reduced tile for that frame time, the thumbnail tiles for 2D videos other than the 2D video currently being viewed among the 2D video thumbnail tiles for that frame time (i.e., the frame time at which tile distribution is requested), and the thumbnail tile for the panoramic video for that frame time (step S110). As an example, FIG. 7 shows a composite example in which there are two 2D video cameras 40, and two 2D video thumbnail tiles have been created at a certain frame time, but the user is currently viewing 2D video from one of the 2D video cameras 40. As shown in FIG. 7, in this case, the 2D video thumbnail tile for the 2D video that the user is not viewing ("2D video 1" in the example shown in FIG. 7) and the thumbnail tile for the panoramic video are embedded in the thumbnail area of ​​the overall reduced tile to be composited. However, at this time, the synthesis unit 105 uses the center coordinates of the field of view range included in the 2D viewing mode distribution request or information for identifying the center coordinates to identify the partial area tile that includes the field of view, and then embeds the thumbnail tile for the panoramic video that corresponds to this identified partial area tile into the overall reduced tile.

[0062] However, in the above step S110, the thumbnail for the panoramic video does not have to be composited with the entire reduced tile. In this case, an image of the area for the panoramic video may be used for the thumbnail video of the panoramic video, or a partial area tile including the user's field of view may be sent to the client terminal 20 and used for the thumbnail video of the panoramic video.

[0063] 6 and 7, the thumbnail area is located at the bottom of the entire reduced-size tile, but this is just an example; the thumbnail area may be located at the top, left, or right of the entire reduced-size tile. There may also be multiple thumbnail areas, such as at both the top and bottom, or both the left and right. In particular, if there are a large number of thumbnails, multiple thumbnail areas may be set.

[0064] Next, distribution unit 106 of video distribution server 10 transmits the overall reduced-size tile resulting from the synthesis in step S105 above and 2D video tiles with the same frame times to client terminal 20 (step S111). At this time, distribution unit 106 identifies the 2D video currently being viewed by the user from information indicating the currently viewed 2D video included in the 2D viewing mode distribution request, and transmits the 2D video tiles of this identified 2D video to client terminal 20. Note that the overall reduced-size tile and 2D video tiles are encoded, and this encoded data is transmitted to client terminal 20.

[0065] Then, the video viewing unit 201 of the client terminal 20 plays back the 2D video in 2D viewing mode using the full reduced tile and the 2D video tiles transmitted from the video distribution server 10 (step S112). That is, the video viewing unit 201 decodes the data received from the video distribution server 10 and displays the 2D video tiles as a 2D video, and also displays the 2D video thumbnail tiles and panoramic video thumbnail tiles included in the full reduced tile as thumbnail videos. As a result, a 2D viewing screen 2000 such as that shown in FIG. 3 is displayed on the client terminal 20. Note that at this time, if the resolution of the 2D video tiles is lower than the resolution of the partial area tiles (that is, for example, if the resolution is lower than that of the panoramic video 1001, as in the case of 2D video 2001 shown in FIG. 3), an image of the panoramic video area included in the full reduced tile may be displayed as the background.

[0066] <Mode switching process> A sequence diagram of the mode switching process according to this embodiment is shown in Fig. 8. Hereinafter, it is assumed that the user of the client terminal 20 is viewing a panoramic video or a 2D video in either the VR viewing mode or the 2D viewing mode.

[0067] The video viewing unit 201 of the client terminal accepts a mode switching operation by the user (step S201). A mode switching operation is an operation for switching the viewing mode from VR viewing mode to 2D viewing mode or from 2D viewing mode to VR viewing mode, and in this embodiment, it is assumed to be an operation for selecting a thumbnail video of a 2D video in VR viewing mode and an operation for selecting a thumbnail video of a panoramic video in 2D viewing mode. However, any operation other than these that can switch the viewing mode may also be used as a mode switching operation.

[0068] Next, if a mode switching operation from VR viewing mode to 2D viewing mode is accepted, steps S202 to S206 are executed, and if a mode switching operation from 2D viewing mode to VR viewing mode is accepted, steps S207 to S211 are executed.

[0069] Switching from VR viewing mode to 2D viewing mode The video viewing unit 201 of the client terminal 20 transmits a 2D viewing mode distribution request to the video distribution server 10 (step S202). Here, it is assumed that the 2D viewing mode distribution request also includes information indicating the 2D video selected by the mode switching operation.

[0070] Next, the video viewing unit 201 of the client terminal 20 displays a standby screen until the 2D video of the thumbnail video selected by the mode switching operation is displayed (step S203).

[0071] As in step S110 of FIG. 5, the compositing unit 105 of the video distribution server 10 composites, among the thumbnail tiles for 2D videos at the relevant frame time, thumbnail tiles for 2D videos other than the 2D video selected by the mode switching operation, and the thumbnail tile for the panoramic video at that frame time into an overall reduced tile for that frame time (step S204).

[0072] 5 , distribution unit 106 of video distribution server 10 then transmits the combined overall reduced tile from step S204 and 2D video tiles with the same frame times to client terminal 20 (step S205). At this time, distribution unit 106 identifies the 2D video selected by the user from information indicating the 2D video selected by the mode switching operation included in the 2D viewing mode distribution request, and transmits the 2D video tiles of this identified 2D video to client terminal 20. Note that the overall reduced tile and 2D video tiles are encoded, and the encoded data is transmitted to client terminal 20.

[0073] 5, the video viewing unit 201 of the client terminal 20 plays the 2D video in 2D viewing mode using the full reduced tile and the 2D video tile transmitted from the video distribution server 10 (step S206). This causes the display to transition from the standby screen to the 2D viewing screen.

[0074] FIG. 9 shows an example of screen transitions on the client terminal 20 when switching from VR viewing mode to 2D viewing mode. As shown in FIG. 9, a panoramic video 1001 and thumbnail videos 1002 and 1003 are displayed on a VR viewing screen 1000. For example, when the user selects the thumbnail video 1002, a standby screen 1500 is displayed in step S203. This standby screen 1500 displays a low-quality 2D video 1501 and a standby indicator 1502. Note that an image of the panoramic video area included in the full reduced tile may be displayed as the background of the low-quality 2D video 1501. Here, the 2D video 1501 is a 2D video thumbnail tile corresponding to the thumbnail video 1002 selected by the user, enlarged to the same resolution as the 2D video tile. Then, when the full reduced tile and the 2D video tile are received from the video distribution server 10, a 2D viewing screen 2000 is displayed. The 2D viewing screen 2000 displays a 2D video 2001 corresponding to the thumbnail video 1002 selected by the user. In this way, until the 2D video selected by the user is displayed, a lower quality version of the selected 2D video is displayed from the thumbnail tile, thereby preventing a decrease in the user's viewing quality due to mode switching.

[0075] 10 shows another example of the screen transition of the client terminal 20 when switching from the VR viewing mode to the 2D viewing mode. As shown in FIG. 10, a standby screen 1600 may be displayed in step S203 described above. This standby screen 1600 displays a standby indicator and a string of characters indicating that 2D video is being loaded in an area 1601. This reduces the user's viewing quality compared to FIG. 9, but allows for simple implementation. Note that instead of displaying both the standby indicator and the string of characters indicating that 2D video is being loaded, only one of them may be displayed.

[0076] Switching from 2D viewing mode to VR viewing mode The video viewing unit 201 of the client terminal 20 transmits a VR viewing mode distribution request to the video distribution server 10 (step S207).

[0077] Next, the video viewing unit 201 of the client terminal 20 displays a standby screen until the panoramic video of the thumbnail video selected by the mode switching operation is displayed (step S208).

[0078] The combining unit 105 of the moving image distribution server 10 combines each 2D moving image thumbnail tile for the corresponding frame time with the entire reduced tile for that frame time, similar to step S106 in FIG. 5 (step S209).

[0079] 5 , the distribution unit 106 of the video distribution server 10 transmits the overall reduced tile after composition in step S209 above and the partial area tile of the same frame time to the client terminal 20 (step S210). At this time, the distribution unit 106 uses the center coordinates of the field of view range included in the VR viewing mode distribution request or information for identifying the center coordinates to identify the partial area tile that includes the field of view, and then transmits this identified partial area tile to the client terminal 20. Note that the overall reduced tile and the partial area tile are encoded, and this encoded data is transmitted to the client terminal 20.

[0080] 5, the video viewing unit 201 of the client terminal 20 plays back the panoramic video in VR viewing mode using the entire reduced tile and the partial area tile transmitted from the video distribution server 10 (step S211). As a result, the display transitions from the standby screen to the VR viewing screen.

[0081] FIG. 11 shows an example of screen transitions on the client terminal 20 when switching from the 2D viewing mode to the VR viewing mode. As shown in FIG. 11, a 2D viewing screen 2000 displays a 2D video 2001, a thumbnail video 2002 of a panoramic video, and a thumbnail video 2003 of another 2D video different from the 2D video currently being viewed. For example, when the user selects the thumbnail video 2002, a standby screen 1700 is displayed in step S208. On this standby screen 1700, a standby indicator 1701 is displayed on the low-quality panoramic video. Here, the low-quality panoramic video is a panoramic video displayed using full-scale reduced tiles. Then, when the full-scale reduced tiles and the partial-area tiles are received from the video distribution server 10, the VR viewing screen 1000 is displayed. In this way, the low-quality panoramic video is displayed using the full-scale reduced tiles until the panoramic video using the partial-area tiles is displayed. This makes it possible to prevent a decrease in the user's viewing quality due to the mode switch.

[0082] 8 has described a case where switching from VR viewing mode to 2D viewing mode or from 2D viewing mode to VR viewing mode, but a standby screen may also be displayed in the same way when, for example, a thumbnail video of a 2D video other than the 2D video currently being viewed is selected on the 2D viewing screen. That is, in this case, the 2D viewing mode remains the same and no switching of viewing mode occurs, but because 2D video tiles of the new 2D video selected by the user need to be received from video distribution server 10, a standby screen similar to that in step S206 or S208 above may be displayed from the time a distribution request for this new 2D video is made until the 2D video tiles of the new 2D video are received and displayed.

[0083] [Second embodiment] Next, a second embodiment will be described. In the first embodiment, a panoramic video represented by the orthographic cylindrical projection method with a vertical angle θ of 0 to π and a horizontal angle φ of 0 to 2π was the distribution target, and the overall reduced tiles and partial region tiles were created from this panoramic video. However, in this embodiment, instead, a region called the region of interest (ROI) in the panoramic video will be used as the distribution target, and the case of creating the overall reduced tiles and partial region tiles from the region of interest will be described. This is because generally, in a 360° × 180° panoramic video, what viewers are interested in is often a partial region where the subject is shown. Hereinafter, the panoramic video representing the region of interest will be referred to as the "ROI panoramic video".

[0084] Note that in the second embodiment, mainly, the differences from the first embodiment will be described, and the description of the components that are the same or similar to those in the first embodiment will be omitted.

[0085] <ROI panoramic video> As shown in FIG. 12, the ROI panoramic video (more precisely, its frame image) can be considered as a partial region of a 360° × 180° virtual panoramic video (more precisely, its frame image). FIG. 12 shows an example where the ROI panoramic video is a 240° × 120° region in a 360° × 180° virtual panoramic video (that is, when expressed by the orthographic cylindrical projection method, a region with a vertical angle of 30° to 150° and a horizontal angle of 60° to 300° with respect to the virtual panoramic video).

[0086] Hereinafter, the resolution of a 360° x 180° virtual panoramic video (hereinafter also referred to as a virtual panoramic video) is defined as ROIFULLEQWIDTH x ROIFULLEQHEIGHT (pixels), the resolution of an ROI panoramic video is defined as RESOLUTIONWIDTH x RESOLUTIONHEIGHT (pixels), and the resolution of a partial area tile is defined as VIEWWIDTH x VIEWHEIGHT (pixels). Also, the offset of the top-left vertex coordinate of the ROI panoramic video relative to the top-left vertex coordinate of the virtual panoramic video is defined as ROILEFTINFULLEQ and ROITOPINFULLEQ. ROILEFTINFULLEQ is the offset of the horizontal component value of the top-left vertex coordinate of the ROI panoramic video relative to the horizontal component value of the top-left vertex coordinate of the virtual panoramic video, and ROITOPINFULLEQ is the offset of the vertical component value of the top-left vertex coordinate of the ROI panoramic video relative to the vertical component value of the top-left vertex coordinate of the virtual panoramic video. Note that the horizontal rightward and vertical downward directions of the coordinates are defined as positive directions.

[0087] At this time, when transmitting the entire reduced tile, the video distribution server 10 also transmits four parameters (ROIFULLEQWIDTH, ROIFULLEQHEIGHT, ROILEFTINFULLEQ, and ROITOPINFULLEQ) that respectively represent the width and height of the ROI panoramic video and the coordinates of the left edge and top edge of the ROI panoramic video in the virtual panoramic video. However, when transmitting the entire reduced tile of a normal 360° x 180° panoramic video rather than an ROI panoramic video, the above four parameters are set to default values ​​before transmission. Here, the default values ​​of ROIFULLEQWIDTH and ROIFULLEQHEIGHT are the maximum resolution values, and the default values ​​of ROILEFTINFULLEQ and ROITOPINFULLEQ are 0.

[0088] More specifically, when transmitting the entire reduced tile of a normal 360° x 180° panoramic video, the following settings are used: ROIFULLEQWIDTH = maximum resolution width, ROIFULLEQHEIGHT = maximum resolution height, ROILEFTINFULLEQ = 0, ROITOPINFULLEQ = 0. On the other hand, when transmitting the entire reduced tile of an ROI panoramic video, the following settings are used: ROIFULLEQWIDTH = maximum resolution width, ROIFULLEQHEIGHT = maximum resolution height, ROILEFTINFULLEQ = left coordinate of the entire reduced tile of the ROI panoramic video, ROITOPINFULLEQ = top coordinate of the entire reduced tile of the ROI panoramic video.

[0089] As a result, if the program that realizes the video viewing unit 201 of the client terminal 20 supports playback of ROI panoramic videos, backward compatibility is realized.

[0090] Note that instead of transmitting the RESOLUTIONWIDTH and ROIFULLEQHEIGHT of the ROI panoramic video, for example, the ratio of ROIFULLEQWIDTH to VIEWWIDTH, and the ratio of ROIFULLEQHEIGHT to VIEWHEIGHT may be transmitted for the RESOLUTIONHEIGHT. This is because the extent to which partial area tiles of each resolution are overlapped to create them is set in advance, so the RESOLUTIONWIDTH can be expressed as the ratio of ROIFULLEQWIDTH to VIEWWIDTH, and the ROIFULLEQHEIGHT can be expressed as the ratio of ROIFULLEQHEIGHT to VIEWHEIGHT. Furthermore, this ratio may be referred to as, for example, the "reduction ratio" of the ROI panoramic video relative to the virtual panoramic video.

[0091] <Video distribution processing> A sequence diagram of the video distribution process according to this embodiment is shown in Fig. 13. Here, the following steps S301 to S302 are repeatedly executed for each frame time width of images shot for the panoramic video, and steps S303 to S304 are repeatedly executed for each frame time width of images shot for the 2D video. Steps S301 to S304 for a certain frame time t will be described below.

[0092] The panoramic video creation unit 101 of the video distribution server 10 creates an ROI panoramic video (more precisely, its frame images) using each panoramic video shot image at frame time t (step S301). The region of interest can be determined arbitrarily. For example, a 240°×120° region of a 360°×180° panoramic video can be selected as the region of interest to create a 240°×120° ROI panoramic video. This is because the upper and lower portions (0° to 30° and 150° to 180°) and the rear portion (0° to 60° and 300° to 360°) are generally not often viewed by users. However, this is just an example; any region within the 360°×180° frame can be selected as the region of interest to create an ROI panoramic video representing that region of interest.

[0093] Next, the panoramic video tile creation unit 102 of the video distribution server 10 creates an overall reduced tile, a partial region tile, and a thumbnail tile for the panoramic video at frame time t using the ROI panoramic video created in step S301 (step S302). Here, this overall reduced tile is an image obtained by compressing the entire ROI panoramic video (more precisely, frame images of the panoramic video) expressed using equirectangular projection to a predetermined resolution. Other than that, the overall reduced tile is the same as in the first embodiment, and includes an area for the panoramic video and an area for the thumbnail.

[0094] The partial area tile is an image with the same resolution as the entire reduced tile, and represents a partial area of ​​the ROI panoramic video (more precisely, its frame image). Note that the partial area tile is the same as the first embodiment except that it is a partial area of ​​the ROI panoramic video.

[0095] The subsequent steps S303 to S304 and subsequent processes are the same as those in the first embodiment.

[0096] In this way, the video distribution server 10 according to the present embodiment creates an ROI panoramic video instead of a 360°×180° panoramic video, and then creates a full-reduced tile and partial region tiles from this ROI panoramic video. This allows the ROI panoramic video to be created with higher image quality than the 360°×180° panoramic video, and therefore the full-reduced tile also has high image quality, thereby further improving the user's viewing quality.

[0097] Furthermore, for example, when creating an ROI panoramic video with the same or similar image quality as a 360° x 180° panoramic video, it is possible to reduce the data size of the entire reduced tile and also reduce the number of cameras 30 for the panoramic video, thereby reducing the costs of introducing or installing the cameras, etc.

[0098] In this embodiment, an ROI panoramic video is created based on the first embodiment, but it is not necessarily based on the first embodiment. That is, this embodiment can be applied to the distribution and viewing of panoramic videos without assuming the existence of 2D videos.

[0099] [Variations] Modifications of the above embodiments will now be described.

[0100] Variation 1 When creating an ROI panoramic video in step S301 of Fig. 13 , the panoramic video creation unit 101 may embed some content (e.g., CG (computer graphics), artist or team logos, etc.) in areas other than the region of interest. Specifically, when a 240° × 120° area in a 360° × 180° panoramic video is set as the region of interest, the panoramic video creation unit 101 may embed some content in areas other than the region of interest in the 360° × 180° panoramic video. This prevents a situation in which nothing is displayed when the user views an area other than the region of interest, and is thought to suppress a deterioration in the user's viewing quality.

[0101] Variation 2 In the first modification, some content is embedded in an area other than the region of interest. However, some content may be embedded within the region of interest instead of outside the region of interest. Specifically, if a 240°×120° area in a 360°×180° panoramic video is set as the region of interest, some content may be embedded within the region of interest. In this case, the content may be embedded in a partial region of the region of interest that will become a partial region tile.

[0102] Note that the second modification and the first modification may be combined to embed some content in both the region of interest and other regions.

[0103] Variation 3 Although partial area tiles are delivered to the client terminal 20 during VR viewing mode, for example, if there is sufficient communication bandwidth and sufficient hardware resources on the client terminal 20 side, and sufficient image quality can be expected even when a panoramic video is played using the entire reduced-size tile, the partial area tiles do not need to be delivered to the client terminal 20. In other words, the resolution of the entire reduced-size tile may be higher than the resolution of the partial area tile. In this way, the resolution of the entire reduced-size tile (particularly the resolution of the panoramic video area included therein) may be set to the same as the resolution of the partial area tile, or may be set to a higher resolution than that, or may be changed as appropriate.

[0104] Variation 4 The video that is played when a thumbnail video is selected is assumed to be a video shot with the 2D video camera 40, but is not limited to this and may be, for example, a video other than live action such as animation or CG, and the content may be an advertisement or content related to the panoramic video (for example, a video explaining the rules of a sport).

[0105] Variation 5 In this embodiment, different overall reduced tiles are synthesized depending on whether the client terminal 20 is in VR viewing mode or 2D viewing mode, but if the user can view only one panoramic video, an overall reduced tile such as that shown in Fig. 6 may be created as a common overall reduced tile for both VR viewing mode and 2D viewing mode. This is because when displaying a thumbnail video of a panoramic video, it is sufficient to use the image in the panoramic video area.

[0106] Variation 6 In this embodiment, the thumbnail tile for the panoramic video corresponds to the user's field of view, but it does not necessarily have to correspond to the user's field of view. For example, a thumbnail tile for the panoramic video may be created that corresponds to the field of view of a virtual agent other than the user.

[0107] Variation 7 In this embodiment, when multiple thumbnail videos are displayed, all of them are thumbnail videos of 2D videos, or one is a thumbnail video of a panoramic video and the rest are thumbnail videos of 2D videos, but this is not limited to this, and for example, all of the multiple thumbnail videos may be thumbnail videos of panoramic videos.

[0108] The present invention is not limited to the above-described embodiments specifically disclosed, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims. [Explanation of symbols]

[0109] 1. Video distribution system 10 Video distribution server 20 Client Terminals 30 Panoramic video camera 40 2D video camera 101 Panorama Video Creation Department 102 Panoramic video tile creation section 103 2D Video Creation Department 104 2D video tile creation section 105 Synthesis Section 106 Distribution Department N1 Communications Network N2 communication network

Claims

1. a first creation unit that creates first frame images constituting a moving image that represents a predetermined region of interest using one or more captured images captured by each of the one or more cameras; a second generation unit that generates an entire image by converting the first frame image to a predetermined resolution; a third generation unit that generates a plurality of partial region images to be combined with each of a plurality of partial regions preset for the first frame image, the partial region images having the resolution; a distribution unit that distributes the entire image and a partial region image, among the plurality of partial region images, that corresponds to a display range of the user terminal when the video is played back, to the user terminal; A video distribution system having the above.

2. the video is a panoramic video, The video distribution system of claim 1, wherein the region of interest is a region of a 360° x 180° panoramic video expressed in equirectangular projection, the region having a vertical angle greater than 0° and less than 180°, and a horizontal angle greater than 0° and less than 360°.

3. The region of interest is 3. The video distribution system according to claim 2, wherein the vertical angle is in the range of 30° to 150° and the horizontal angle is in the range of 60° to 300°.

4. The first creation unit A video distribution system as described in any one of claims 1 to 3, which creates either or both of a first frame image in which a predetermined first content is embedded in the region of interest and a second frame image in which a predetermined second content is embedded in a region other than the region of interest.

5. a first creation unit that creates first frame images constituting a moving image that represents a predetermined region of interest using one or more captured images captured by each of the one or more cameras; a second generation unit that generates an entire image by converting the first frame image to a predetermined resolution; a third generation unit that generates a plurality of partial region images to be combined with each of a plurality of partial regions preset for the first frame image, the partial region images having the resolution; a distribution unit that distributes the entire image and a partial region image, among the plurality of partial region images, that corresponds to a display range of the user terminal when the video is played back, to the user terminal; A video distribution device having the above.

6. a first creation procedure for creating first frame images constituting a moving image representing a predetermined region of interest using one or more captured images captured by each of one or more cameras; a second creation procedure for creating an entire image by converting the first frame image to a predetermined resolution; a third creation step of creating a plurality of partial region images to be combined with each of a plurality of partial regions preset for the first frame image, the partial region images having the resolution; a delivery step of delivering, to the user terminal, the entire image and a partial region image among the plurality of partial region images that corresponds to a display range of the user terminal when the video is played; How a computer runs.

7. a first creation procedure for creating first frame images constituting a moving image representing a predetermined region of interest using one or more captured images captured by each of one or more cameras; a second creation procedure for creating an entire image by converting the first frame image to a predetermined resolution; a third creation step of creating a plurality of partial region images to be combined with each of a plurality of partial regions preset for the first frame image, the partial region images having the resolution; a delivery step of delivering, to the user terminal, the entire image and a partial region image among the plurality of partial region images that corresponds to a display range of the user terminal when the video is played; A program that causes a computer to execute the following.

Images