Receiver

Abstract

To allow a video signal required for viewpoint interpolation to be easily identified when a user changes a viewpoint position.SOLUTION: A receiving device (40b) receives video signals grouped according to shooting positions from a transmitting device (30b) together with position information of an imaging device (10). The receiving device includes a communication interface (42) that simultaneously receives all the video signals with assigned group IDs from the transmitting device, and a control unit (43, 45, 47b, 48b, 50, 52, 53) that decodes the video signals and acquires a user's viewpoint position. When there is a change in the viewpoint position that does not involve a change in the ID, the control unit identifies position information of the imaging device that has captured a plurality of video signals required to generate a video corresponding to the user's viewpoint position, and performs viewpoint interpolation using a decoded video corresponding to the position information. When there is a change in the viewpoint position that involves a change in the ID, the control unit acquires a default decoded video from among decoded videos of a video group that corresponds to a new viewpoint position.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to a receiving device for receiving video signals.

Background Art

[0002] As a development of 360-degree video, a video system that allows users to view 360-degree video from a preferred viewpoint has been studied. For example, in Non-Patent Document 1, use cases such as "3D image message" and "immersive 6DoF streaming" are cited, which involve viewpoint movement within the range where the user moves their head while sitting as 3DoF+ (Degree of Freedom), or viewpoint movement within the range where the user moves freely as 6DoF.

[0003] Generally, one camera captures video from one viewpoint. By using multiple videos (multi-viewpoint videos) captured by multiple cameras, it is possible to generate a video from an intermediate position between the positions where they were captured. This is a known technique called viewpoint interpolation, intermediate viewpoint image generation, etc. (see, for example, Patent Document 1).

[0004] Also, as a file format for storing 360-degree video signals of predetermined viewpoints, etc., OMAF (Omnidirectional Media Format) is defined in Non-Patent Document 2.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] Viewpoint interpolation generates an image from a desired viewpoint using multiple video signals captured from different locations. However, for services that allow users to view images from viewpoints they can move freely, such as 6DoF, it is necessary to capture multiple images from many points and perform viewpoint interpolation. However, the captured video signals are independent of each other, and there is no information to relate them. OMAF can store video signals from a fixed viewpoint, but it does not support processing multiple video signals when the viewpoint is freely moved. Therefore, the receiving terminal has no way to identify what video signals exist and how they relate to each other, nor does it have a way to determine how to relate multiple video signals when they are received.

[0008] In view of these circumstances, the object of the present invention is to provide a receiving device that can easily identify the video signal necessary for viewpoint interpolation when the user changes their viewpoint position. [Means for solving the problem]

[0009] A receiving device according to one embodiment is a receiving device that receives multiple video signals transmitted by a transmitting device and taken from different locations, along with location information of an imaging device that took the video signals, wherein the video signals are grouped into multiple groups according to the shooting location, and comprises a communication interface that simultaneously receives all video signals from the transmitting device to which an ID for identifying the group has been assigned, and a control unit that decodes the video signals to generate decoded video and acquires the user's viewpoint position, wherein if there is a change in viewpoint position without a change in the ID, the control unit identifies the location information of the imaging device that took the multiple video signals necessary to generate video corresponding to the user's viewpoint position, and performs viewpoint interpolation using the decoded video corresponding to the location information, and if there is a change in viewpoint position with a change in the ID, it acquires the default decoded video from the decoded video group corresponding to the new viewpoint position.

[0010] In one embodiment, the first-level group among the plurality of groups may be a group to which a plurality of video signals used in the receiving device to generate images from a viewpoint different from the shooting position belong.

[0011] In one embodiment, the second-tier group among the plurality of groups may be one or more first-tier groups to which video signals captured in the same location belong.

[0012] In one embodiment, the video signals belonging to the first hierarchical group may be compressed by predictive coding using correlations between video signals.

[0013] In one embodiment, the ID and the video signal may be stored in a file in ISO Base Media File Format. [Effects of the Invention]

[0014] According to the present invention, when a user changes their viewpoint position, it becomes possible to easily identify the video signals necessary for viewpoint interpolation. [Brief Description of the Drawings]

[0015] [Figure 1] It is a block diagram showing an example of a video transmission system according to the first embodiment. [Figure 2] It is a diagram for explaining the first group ID according to the first embodiment. [Figure 3] It is a diagram for explaining the second group ID according to the first embodiment. [Figure 4] It is a diagram for explaining the hierarchical structure of a video signal according to the first embodiment. [Figure 5A] It is a diagram showing a configuration example of a file for storing a video signal according to the first embodiment. [Figure 5B] It is a diagram showing a configuration example of a file for storing a video signal according to the first embodiment. [Figure 6] It is a diagram showing a display example of a receiving device according to the first embodiment. [Figure 7] It is a sequence diagram showing an operation example of a video transmission system according to the first embodiment. [Figure 8] It is a block diagram showing an example of a video transmission system according to the second embodiment. [Figure 9] It is a sequence diagram showing an operation example of a video transmission system according to the second embodiment. [Figure 10] It is a block diagram showing an example of a video transmission system according to the third embodiment. [Figure 11] It is a sequence diagram showing an operation example of a video transmission system according to the third embodiment. [Figure 12] It is a block diagram showing a schematic configuration of a computer. [Modes for Carrying Out the Invention] [[ID=*]] [[ID=*]] [[ID=*]]

[0016] [[ID=*]] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0017] [First Embodiment] [Video Transmission System] First, a video transmission system according to the first embodiment will be described. Figure 1 is a diagram showing a video transmission system 1 according to the first embodiment, illustrating the overview of the transmission of multiple video signals captured at multiple locations. The video transmission system 1 comprises multiple imaging devices (cameras) 10, multiple first transmitting devices 20, a second transmitting device 30, and a receiving device 40. In this specification, "location" refers to a predetermined area established for a specific purpose, such as a venue, stadium, facility, or park. In this embodiment, the video transmission system 1 comprises three first transmitting devices 20, with the first transmitting device 20A transmitting a video signal captured at location A, the first transmitting device 20B transmitting a video signal captured at location B, and the first transmitting device 20C transmitting a video signal captured at location C, as will be described below.

[0018] The imaging device 10 adds metadata to the video signal captured at a predetermined position and transmits it to the first transmitting device 20. Here, the metadata transmitted by the imaging device 10 is position information indicating the position of the imaging device 10, and may also include information indicating the direction of shooting. The metadata may also include distance information indicating the distance from the imaging device 10 to the captured object.

[0019] In other words, multiple imaging devices 10A installed at location A each attach metadata to the video signal captured at a predetermined location in location A and transmit it to the first transmitting device 20A. Similarly, multiple imaging devices 10B installed at location B each attach metadata to the video signal captured at a predetermined location in location B and transmit it to the first transmitting device 20B. Similarly, multiple imaging devices 10C installed at location C each attach metadata to the video signal captured at a predetermined location in location C and transmit it to the first transmitting device 20C.

[0020] The first transmitter 20A receives video signals and metadata from multiple imaging devices 10A, assigns an ID to them, and transmits them to the second transmitter 30. Similarly, the first transmitter 20B receives video signals and metadata from multiple imaging devices 10B, assigns an ID to them, and transmits them to the second transmitter 30. Similarly, the first transmitter 20C receives video signals and metadata from multiple imaging devices 10C, assigns an ID to them, and transmits them to the second transmitter 30.

[0021] For transmission from the imaging device 10 to the first transmitting device 20, and from the first transmitting device 20 to the second transmitting device 30, SDI (Serial Digital Interface), IP (Internet Protocol), etc., can be used. More specifically, when using SDI, transmission can be done using 12G-SDI, 3G-SDI, HD-SDI, etc., and when using IP, transmission can be done using RTP (Real-Time Transport Protocol) based on the SMPTE (Society of Motion Picture and Television Engineers) ST 2110 standard. For example, when transmitting the video signal without encoding it, transmission can be done using SDI or IP, and when transmitting the video signal after encoding it, transmission can be done using IP. In this embodiment, the imaging device 10 transmits the video signal to the first transmitting device 20 without encoding it, and the first transmitting device 20 encodes the video signal and then transmits it to the second transmitting device 30. However, encoding may be performed in the imaging device 10, or encoding may not be performed in the first transmitting device 20.

[0022] The second transmitting device 30 aggregates the video signals transmitted from multiple first transmitting devices 20 and transmits the video signals to the receiving device 40 via a transmission path (for example, an IP network). However, it is not necessarily required that the video signals be aggregated by the second transmitting device 30; the first transmitting devices 20 may directly transmit the video signals to the receiving device 40 via various transmission paths such as broadcast transmission paths, terrestrial IMT (International Mobile Telecommunication) networks, optical fibers, or Wi-Fi® lines.

[0023] The receiving device 40 can be any device capable of receiving video transmission from the first transmitting device 20 or the second transmitting device 30. Examples of the receiving device 40 include a head-mounted display, VR (Virtual Reality) goggles, tablet terminal, smartphone, and PC (Personal Computer).

[0024] (First Transmitter) Next, the first transmitting device 20 according to the first embodiment will be described.

[0025] As shown in Figure 1, the first transmitting device 20 comprises a video receiving unit 21, a video encoding unit 22, a first ID assignment unit 23, a storage unit 24, and a video transmitting unit 25. The video encoding unit 22 and the first ID assignment unit 23 constitute a control unit (controller). The control unit may be composed of dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), or it may be composed of a processor, or it may include both. The video receiving unit 21 and the video transmitting unit 25 constitute a communication interface.

[0026] The video receiving unit 21 acquires video signals and metadata from multiple imaging devices 10 and outputs them to the video encoding unit 22.

[0027] The video encoding unit 22 encodes the video signal input from the video receiving unit 21 using an arbitrary video encoding scheme such as H.265 / HEVC (High Efficiency Video Coding) to generate an encoded video signal. For efficient transmission, the video encoding unit 22 may perform compression using predictive encoding based on the correlation between each video signal. The video encoding unit 22 then outputs the encoded video signal to the storage unit 24.

[0028] In this invention, video signals are grouped into multiple groups according to the shooting position. Multiple video signals captured by multiple imaging devices 10 can be grouped into units of video signal groups used to generate video from a viewpoint different from the shooting position (used for viewpoint interpolation). This grouping is referred to as the "first-level grouping" to distinguish it from the grouping described later. In other words, the first-level group is the group to which the multiple video signals used by the receiving device 40 to generate video from a viewpoint different from the shooting position belong.

[0029] The first ID assignment unit 23 receives an ID (first group ID) for identifying the first-level group based on the operation of the operator of the first transmission device 20, and stores the first group ID in the storage unit 24 in association with the video signal. In other words, the first ID assignment unit 23 assigns the first group ID to the video signal.

[0030] Referring to Figure 2, the first group ID will be explained. Figure 2 shows an example of filming at a baseball field, where a total of 15 imaging devices 10 are installed behind the catcher's position, 5 horizontally and 3 vertically. For the sake of explanation, the position information p of these 15 imaging devices 10 will be denoted as 1 to 15. Also, a total of 15 imaging devices 10 are installed on the first base side of the baseball field, 5 horizontally and 3 vertically. For the sake of explanation, the position information p of these 15 imaging devices 10 will be denoted as 16 to 30. Furthermore, a total of 15 imaging devices 10 are installed on the third base side of the baseball field, 5 horizontally and 3 vertically. For the sake of explanation, the position information p of these 15 imaging devices 10 will be denoted as 31 to 45. Note that in the example shown in Figure 2, the imaging devices 10 only capture images in front of the filming position, but they may also capture 360-degree images of the entire surroundings from the filming position.

[0031] The receiving device 40 can receive video signals captured by the imaging device 10 at location information p=1 to 15 and, by interpolating the viewpoint, can present the user with video footage of a certain range of viewpoints from behind the catcher. For example, if the receiving device 40 is displaying video signals captured by the imaging device 10 at location information p=8, and the user then moves within a predetermined range (for example, a radius of several meters), the receiving device 40 can present the user with video footage corresponding to the user's viewpoint. Similarly, by receiving video signals captured by the imaging device 10 at location information p=16 to 30, the receiving device 40 can present the user with video footage of a certain range of viewpoints from the first base side. Similarly, by receiving video signals captured by the imaging device 10 at location information p=31 to 45, the receiving device 40 can present the user with video footage of a certain range of viewpoints from the third base side.

[0032] For example, as shown in Figure 2, when images are captured by the imaging device 10 with position information p=1 to 45, the first ID assignment unit 23 assigns "I" as the first group ID to 15 video signals captured from behind the catcher, "II" as the first group ID to 15 video signals captured from the first base side, and "III" as the first group ID to 15 video signals captured from the third base side. The first ID assignment unit 23 may assign the first group ID to all video signals. Furthermore, if the video encoding unit 22 performs compression using the correlation between each video signal, the first ID assignment unit 23 may assign the first group ID only to the reference video signal (for example, the video signal captured by the imaging device 10 with position information p=8, 23, 38).

[0033] Referring again to Figure 1, the storage unit 24 stores the encoded video signal and metadata (location information) input from the video encoding unit 22, as well as the metadata (information indicating the first group ID) input from the first ID assignment unit 23.

[0034] The video transmission unit 25 transmits the encoded video signal and metadata (location information and information indicating the first group ID) stored in the storage unit 24 to the second transmission device 30.

[0035] The imaging device 10 and part or all of the first transmitting device 20 may be integrated. For example, if the imaging device 10 has the function of an image encoding unit 22, the image receiving unit 21 of the first transmitting device 20 receives an encoded video signal from the imaging device 10 and outputs it to the storage unit 24.

[0036] (Second Transmitter) Next, the second transmitting device 30 according to the first embodiment will be described.

[0037] As shown in Figure 1, the second transmitting device 30 includes a video receiving unit 31, a second ID assignment unit 32, a storage unit 33, a viewing request receiving unit 34, a viewpoint information receiving unit 35, a position determination unit 36, and a video transmitting unit 37. The second ID assignment unit 32 and the position determination unit 36 ​​constitute a control unit. The control unit may be composed of dedicated hardware such as an ASIC or FPGA, a processor, or a combination of both. The video receiving unit 31, the viewing request receiving unit 34, the viewpoint information receiving unit 35, and the video transmitting unit 37 constitute a communication interface.

[0038] The video receiving unit 31 receives encoded video signals and metadata transmitted from multiple first transmitting devices 20, and outputs the received encoded video signals and metadata to the storage unit 33.

[0039] Video signals captured by multiple imaging devices 10 can be grouped into units of video signals captured within the same location. This grouping is referred to as the "second-tier grouping" to distinguish it from the first-tier grouping described above. In other words, a second-tier group is one or more first-tier groups to which multiple video signals captured within the same location belong.

[0040] The second ID assignment unit 32 receives an ID (second group ID) for identifying the second-level group based on the operation of the operator of the second transmission device 30, and stores the second group ID in the storage unit 33 in association with the video signal. In other words, the second ID assignment unit 32 assigns the second group ID to the video signal.

[0041] Referring to Figure 3, the second group ID will be explained. In the example shown in Figure 3, the second transmitter 30 receives video signals captured at location A from the first transmitter 20A. Specifically, the second transmitter 30 receives 15 video signals of first group ID=I captured by the imaging device 10 with location information p=1 to 15, 15 video signals of first group ID=II captured by the imaging device 10 with location information p=16 to 30, and 15 video signals of first group ID=III captured by the imaging device 10 with location information p=31 to 45. The second transmitter 30 also receives video signals captured at location B from the first transmitter 20B. Specifically, the second transmitter 30 receives from the first transmitter 20B 15 video signals of the first group ID=I captured by the imaging device 10 at location p=46~60, 15 video signals of the first group ID=II captured by the imaging device 10 at location p=61~75, and 15 video signals of the first group ID=III captured by the imaging device 10 at location p=76~90. The second transmitter 30 also receives video signals captured at location C from the first transmitter 20C. Specifically, the second transmitter 30 receives from the first transmitter 20C 15 video signals of the first group ID=I captured by the imaging device 10 at position information p=91~105, 15 video signals of the first group ID=II captured by the imaging device 10 at position information p=106~120, and 15 video signals of the first group ID=III captured by the imaging device 10 at position information p=121~135.

[0042] Furthermore, the second ID assignment unit 32 assigns, for example, "A" as a second group ID to the 45 video signals captured at location A, for example, "B" as a second group ID to the 45 video signals captured at location B, and for example, "C" as a second group ID to the 45 video signals captured at location C.

[0043] Figure 4 shows a hierarchical structure in which video signals are grouped in two layers in this way. As shown in Figure 3, 45 video signals are taken at each of locations A, B, and C. In this case, the 45 video signals taken at location A are classified into Group A of the second group ID=A, and within Group A they are further classified into 15 video signals belonging to the first group ID=I, 15 video signals belonging to the first group ID=II, and 15 video signals belonging to the first group ID=III. Similarly, the 45 video signals taken at location B are classified into Group B of the second group ID=B, and within Group B they are further classified into 15 video signals belonging to the first group ID=I, 15 video signals belonging to the first group ID=II, and 15 video signals belonging to the first group ID=III. Furthermore, the 45 video signals captured at location C are classified into Group C of Group 2 ID=C, and within Group C, they are further classified into 15 video signals belonging to Group 1 ID=I, 15 video signals belonging to Group 1 ID=II, and 15 video signals belonging to Group 1 ID=III.

[0044] Referring again to Figure 1, the storage unit 33 stores the encoded video signal and metadata input from the video receiving unit 31, as well as the metadata input from the second ID assignment unit 32.

[0045] When the viewing request receiving unit 34 receives a viewing request from the receiving device 40 indicating the content to be viewed, it outputs the viewing request to the video transmission unit 37.

[0046] When the viewpoint information receiving unit 35 receives viewpoint information from the receiving device 40 indicating the viewpoint position of the user of the receiving device 40 (for example, the position of the receiving device 40), it outputs the viewpoint information to the position determination unit 36. For example, the viewpoint position information is a three-dimensional coordinate in the global coordinate system. Alternatively, the viewpoint position information may be a three-dimensional coordinate in the local coordinate system based on the default user position.

[0047] When the position determination unit 36 ​​receives viewpoint information from the viewpoint information receiving unit 35, it determines whether the change in the user's viewpoint position is accompanied by a change in IDs (first group ID and second group ID). Specifically, the position determination unit 36 ​​identifies the request ID, which is the ID of the video signal corresponding to the change in the user's viewpoint position, and determines whether the request ID is the same as the ID of the video signal transmitted immediately before.

[0048] If the position determination unit 36 ​​finds that the request ID is the same as the ID of the video signal transmitted immediately before, it outputs to the video transmission unit 37 the position information of multiple imaging devices 10 that captured multiple video signals necessary to generate an image corresponding to the user's viewpoint position from the group of video signals to which the request ID is assigned. For example, the position determination unit 36 ​​finds the shooting position corresponding to the user's viewpoint position and outputs to the video transmission unit 37 the position information of imaging devices 10 located adjacent to or surrounding that shooting position.

[0049] Furthermore, if the position determination unit 36 ​​finds that the request ID is different from the ID of the video signal transmitted immediately before, it outputs the position information of the imaging device 10 that captured the default video signal from the group of video signals to which the request ID was assigned to the video transmission unit 37.

[0050] The video transmission unit 37 transmits a video signal to the receiving device 40. In this specification, "transmitting a video signal" means that metadata attached to (multiplexed with) the video signal is also transmitted simultaneously. The video transmission unit 37 can also transmit only the metadata. The video signals belonging to the first-level group transmitted by the video transmission unit 37 may be compressed by the video encoding unit 22 of the first transmitting device 20 using predictive coding based on the correlation between video signals.

[0051] When a viewing request is received from the viewing request receiving unit 34, the video transmission unit 37 obtains the default video signal of the requested content, all ID information related to the requested content (information indicating the first group ID and the second group ID), and the location information of the imaging device 10 representing each first-level group from the storage unit 33 and transmits them to the receiving device 40. Based on the ID information and the location information of the imaging device 10, the receiving device 40 presents to the user what viewpoint positions are available.

[0052] Furthermore, when the video transmission unit 37 receives location information from the location determination unit 36, it retrieves a video signal (a video signal corresponding to the location information) captured by the imaging device 10 that possesses the location information from the storage unit 33 and transmits it to the receiving device 40. In other words, if the request ID is the same as the ID of the video signal transmitted immediately before, the video transmission unit 37 transmits multiple video signals from the group of video signals to which the request ID is assigned that are necessary to generate a video corresponding to the user's viewpoint position. If the request ID is different from the ID of the video signal transmitted immediately before, the video transmission unit 37 transmits the default video signal from the group of video signals to which the request ID is assigned.

[0053] Figure 5 shows an example of file structure when ID and video signals are stored in an ISO Base Media File Format file such as OMAF. This format consists of basic units called boxes.

[0054] In the file structure shown in Figure 5A, when storing n video signals with a common second group ID and first group ID in a file, the group to which the video signals belong is indicated by storing the n video signals under the boxes indicating the second group ID and first group ID. In the file structure shown in Figure 5B, when storing n video signals with a common second group ID and first group ID in a file, the group to which the video signals belong is indicated by storing boxes indicating the second group ID and first group ID along with the track that stores each video signal.

[0055] In this embodiment, a two-tiered grouping system was used for explanation, but for user operation purposes, a three-tiered or higher grouping system may be used depending on the content. Even when a three-tiered or higher grouping system is used, the video signals can be identified in a similar tiered manner. For example, one or more second-tiered groups containing video signals shot within the same region may be grouped together as a third-tiered group.

[0056] (Receiving device) Next, the receiving device 40 according to the first embodiment will be described.

[0057] As shown in Figure 1, the receiving device 40 includes a viewing request transmission unit 41, a video receiving unit 42, a video decoding unit 43, a storage unit 44, a viewpoint position detection unit 45, a viewpoint information transmission unit 46, a viewpoint insertion unit 47, an operation information addition unit 48, and a display unit 49. The video decoding unit 43, the viewpoint position detection unit 45, the viewpoint insertion unit 47, and the operation information addition unit 48 constitute a control unit. The control unit may be composed of dedicated hardware such as an ASIC or FPGA, a processor, or a combination of both. The viewing request transmission unit 41, the video receiving unit 42, and the viewpoint information transmission unit 46 constitute a communication interface.

[0058] The viewing request transmission unit 41 transmits a viewing request to the second transmission unit 30, indicating the content selected by the user of the receiving device 40.

[0059] The video receiving unit 42 receives video signals and metadata (location information and ID information) from the second transmitting device 30. The video receiving unit 42 outputs the received video signals and metadata to the video decoding unit 43, associating them with each other.

[0060] The video decoding unit 43 decodes the video signal, generates a decoded video, and outputs it to the storage unit 44.

[0061] The memory unit 44 stores the decoded video generated by the video decoding unit 43.

[0062] The viewpoint position detection unit 45 detects the user's viewpoint position and outputs viewpoint information indicating the detected viewpoint position to the viewpoint information transmission unit 46. For example, the viewpoint position detection unit 45 acquires an image of the user's eyes captured by the imaging device provided in the receiving device 40 and detects the viewpoint position by analyzing the image using any known technique. Alternatively, the viewpoint position detection unit 45 may be a gaze sensor provided separately from the control unit in the receiving device 40. The control unit acquires the user's viewpoint position using the viewpoint position detection unit 45.

[0063] The viewpoint information transmission unit 46 transmits the viewpoint information detected by the viewpoint position detection unit 45 to the second transmission device 30. The viewpoint information transmission unit 46 may transmit viewpoint information to the second transmission device 30 at predetermined intervals, or it may transmit viewpoint information to the second transmission device 30 only when the amount of change in viewpoint position exceeds a threshold.

[0064] The viewpoint interpolation unit 47 performs viewpoint interpolation using multiple decoded images stored in the memory unit 44 that share a common first group ID and second group ID (i.e., decoded images obtained by decoding video signals corresponding to position information determined by the position determination unit 36), and generates an image corresponding to the viewpoint position detected by the viewpoint position detection unit 45. Any known technique (see, for example, Patent Document 1) can be used for viewpoint interpolation. The viewpoint interpolation unit 47 outputs the viewpoint-interpolated decoded image to the operation information addition unit 48.

[0065] The operation information addition unit 48, in response to a user's viewing request, obtains a default decoded video from the storage unit 44, adds operation information to the decoded video, and outputs it to the display unit 49. After viewpoint interpolation is performed by the viewpoint interpolation unit 47, the operation information addition unit 48 obtains the viewpoint-interpolated decoded video from the viewpoint interpolation unit 47, adds operation information to the decoded video, and outputs it to the display unit 49.

[0066] Here, operation information refers to information that indicates to the user how to change their viewpoint position. For example, operation information may indicate how to change to a different viewpoint position in the same location, or to a different viewpoint position in a different location. Operation information may also include icons. The user can request a change in viewpoint position by performing operations based on the operation information. For example, if the receiving device 40 is a head-mounted display, the user may perform operations based on the operation information by changing the direction of their gaze. In this case, the direction of the gaze is detected by the viewpoint position detection unit 45. If the receiving device 40 is a tablet terminal, the user may perform operations based on the operation information by using screen operations such as flicking or swiping. If the receiving device 40 is a non-mobile or non-portable device with a flat panel display, the user may perform operations based on the operation information using a remote control.

[0067] The display unit 49 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 49 displays the decoded video and operation information input from the operation information addition unit 48.

[0068] Figure 6 shows an example of the display of video and operation information on the display unit 49. In the example shown in Figure 6, video 481 and operation information 482 to 486 are displayed. Operation information 482 indicates that the displayed video 481 is based on the video group of the first group ID=II at location C. Operation information 483 indicates that the viewpoint position at location C can be changed by a predetermined operation (the displayed video 481 can be changed to video based on the video group of the first group ID=I at location C). Operation information 484 indicates that the viewpoint position at location C can be changed by a predetermined operation (the displayed video 481 can be changed to video based on the video group of the first group ID=III at location C). Operation information 485 indicates that the viewpoint position can be changed to a different location by a predetermined operation (the displayed video 481 can be changed to video taken at location A). Operation information 486 indicates that the viewpoint position can be changed to a different location by a predetermined operation (the displayed video 481 can be changed to video taken at location B).

[0069] When a user performs an operation based on the operation information displayed on the display unit 49, the viewpoint position detection unit 45 detects the viewpoint position corresponding to the operation. This allows the user to request a change in viewpoint position from the second transmission device 30.

[0070] (Action sequence) Next, the operation sequence of the video transmission system 1 according to this embodiment will be described with reference to Figure 7. Figure 7 is a diagram showing the operation sequence between the second transmitting device 30 and the receiving device 40.

[0071] In step S101, the receiving device 40 transmits a viewing request indicating the content selected by the user to the second transmitting device 30 via the viewing request transmission unit 41. The second transmitting device 30 receives the viewing request transmitted from the receiving device 40 via the viewing request receiving unit 34.

[0072] In step S102, the second transmitting device 30 transmits all ID information and location information of the imaging device 10 representing each first-level group to the receiving device 40 via the video transmitting unit 37 for the content requested by the receiving device 40 for viewing. The receiving device 40 receives the above information regarding the content requested by the second transmitting device 30 via the video receiving unit 42. For example, if the installation positions of the imaging devices 10 are as shown in Figure 2, the location information of the imaging devices 10 representing each first-level group will be the location information of the imaging devices 10 located in the center, p=8,23,38. The video transmitting unit 37 may transmit multiple locations to the receiving device 40 for each first-level group division.

[0073] In step S103, the second transmitting device 30 transmits a default video signal to the receiving device 40 for the content requested to be viewed by the receiving device 40 via the video transmission unit 37. The second transmitting device 30 pre-determines a default video signal for each piece of content. For example, if the video signals for the requested content are grouped as shown in Figure 4, the default video is set to video signal 8 for the second group ID=A and the first group ID=I (the video signal captured by the imaging device 10 with position information p=8, located in the center behind the catcher in Figure 2).

[0074] In step S104, the receiving device 40 receives the default video signal of the requested content from the second transmitting device 30 via the video receiving unit 42. The receiving device 40 then decodes the video signal using the video decoding unit 43 and displays the default video using the display unit 49.

[0075] In step S105, the receiving device 40 transmits viewpoint information to the second transmitting device 30 via the viewpoint information transmitting unit 46.

[0076] In step S106, the second transmitting device 30 uses the position determination unit 36 ​​to identify a request ID, which is the ID of the video signal corresponding to the change in the user's viewpoint position. If the request ID is the same as the ID of the video signal transmitted immediately before, the second transmitting device 30 proceeds to step S107; otherwise, it proceeds to step S109.

[0077] In step S107, the second transmitting device 30 transmits a video signal necessary to generate an image corresponding to the user's viewpoint position to the receiving device 40 via the video transmitting unit 37.

[0078] In step S108, the receiving device 40 performs viewpoint interpolation processing using the viewpoint interpolation unit 47 and the multiple video signals necessary for viewpoint interpolation that it has received from the second transmitting device 30. The receiving device 40 then displays the viewpoint-interpolated video on the display unit 49. The receiving device 40 then returns to step S105.

[0079] In step S109, the second transmitting device 30 transmits the default video signal from the group of video signals to which the request ID has been assigned to the receiving device 40 via the video transmission unit 37.

[0080] In step S110, the receiving device 40 receives a default video signal with a changed viewpoint position from the second transmitting device 30 using the video receiving unit 42. The receiving device 40 then decodes the video signal using the video decoding unit 43 and displays the default video using the display unit 49. The receiving device 40 then returns to step S105.

[0081] As explained above, the video transmission system 1, which transmits multiple video signals captured at different locations, groups the video signals and assigns them IDs. By grouping multiple video signals as a first-level group, it becomes easy to identify multiple video signals to be used for viewpoint interpolation. Furthermore, by grouping multiple video signals as a second-level group, it becomes possible to show the user what viewpoint position changes are possible. The user can then view videos with the viewpoint position changed within the same location, and videos with the viewpoint position changed to a different location, by performing operations according to the operation information displayed on the display unit 49.

[0082] <Second Embodiment> Next, a video transmission system according to the second embodiment will be described. Figure 8 shows a video transmission system 2 according to the second embodiment. The video transmission system 2 comprises a plurality of imaging devices 10, a plurality of first transmitting devices 20, a second transmitting device 30a, and a receiving device 40a. The imaging devices 10 and first transmitting devices 20 in this embodiment are the same as the imaging devices 10 and first transmitting devices 20 in the first embodiment, so their description will be omitted. The second transmitting device 30a and receiving device 40a in this embodiment are partially the same as the second transmitting device 30 and receiving device 40 in the first embodiment, so the same reference numerals are used for common components, and their descriptions will be omitted as appropriate.

[0083] The receiving device 40a of this embodiment differs from the receiving device 40 of the first embodiment in that it further includes an ID information detection unit 50 and replaces the viewpoint information transmission unit 46 with a viewpoint information transmission unit 46a. The ID information detection unit 50 also constitutes the control unit.

[0084] The ID information detection unit 50 detects an ID (request ID) assigned to the group of video signals corresponding to the new viewpoint position requested by the user when the user requests a change in viewpoint position by performing an operation based on the operation information displayed on the display unit 49. The ID information detection unit 50 then outputs the detected request ID as viewpoint information to the viewpoint information transmission unit 46a. For example, in Figure 6, if the user performs a swipe operation along the arrow icon indicated by the operation information 483, the ID information detection unit 50 detects the first group ID as I and the second group ID as C as the request ID. Also in Figure 6, if the user performs a swipe operation along the arrow icon indicated by the operation information 485, the ID information detection unit 50 detects the first group ID as the default value and the second group ID as A as the request ID. Note that if the first group ID is the default value, the request ID may only include the second group ID.

[0085] If the user does not perform an operation based on the operation information displayed on the display unit 49, the viewpoint information transmission unit 46a transmits the viewpoint information detected by the viewpoint position detection unit 45 to the second transmission device 30a. If the user performs an operation based on the operation information displayed on the display unit 49, the viewpoint information (request ID) detected by the ID information detection unit 50 transmits the viewpoint information (request ID) to the second transmission device 30a.

[0086] The second transmitting device 30a of this embodiment differs from the second transmitting device 30 of the first embodiment in that it includes a viewpoint information receiving unit 35a, a position determination unit 36a, and a video transmitting unit 37a, instead of the viewpoint information receiving unit 35, a position determination unit 36, and a video transmitting unit 37.

[0087] When the viewpoint information receiving unit 35a receives viewpoint information that does not include a request ID, it outputs the viewpoint information to the position determination unit 36a, similar to the first embodiment. Furthermore, when the viewpoint information receiving unit 35a receives viewpoint information that includes a request ID, it identifies the position information corresponding to the default video signal among the group of video signals to which the request ID is attached, and transmits the identified position information to the video transmission unit 37a.

[0088] When the position determination unit 36a receives viewpoint information that does not include a request ID from the viewpoint information receiving unit 35a, it outputs the position information of the multiple imaging devices 10 that have captured video signals necessary to generate video corresponding to the user's viewpoint position to the video transmission unit 37a. Also, when the position determination unit 36a receives viewpoint information that includes a request ID from the viewpoint information receiving unit 35a, it outputs the position information of the default video signal from the group of video signals to which the request ID has been assigned to the video transmission unit 37a.

[0089] When position information is input from the viewpoint information receiving unit 35a or the position determination unit 36a, the video transmission unit 37a acquires a video signal (a video signal corresponding to the position information) captured by the imaging device 10 having the position information from the storage unit 33 and transmits it to the receiving device 40a.

[0090] (Action sequence) Next, the operation sequence of the video transmission system 2 according to this embodiment will be described with reference to Figure 9. Figure 9 is a diagram showing the operation sequence between the second transmitting device 30a and the receiving device 40a. Steps S201 to S204 are the same as steps S101 to S104 of the first embodiment described with reference to Figure 7, so their description will be omitted.

[0091] In step S205, the receiving device 40a transmits viewpoint information to the second transmitting device 30a via the viewpoint information transmitting unit 46a.

[0092] In step S206, the second transmitting device 30a uses the position determination unit 36a to determine whether the viewpoint information includes a request ID. If the viewpoint information does not include a request ID, the second transmitting device 30a proceeds to step S207; if the viewpoint information includes a request ID, it proceeds to step S209.

[0093] Steps S207 to S210 are identical to steps S107 to S110 of the first embodiment described with reference to Figure 7, so their description is omitted.

[0094] As explained above, in this embodiment, the detection of the request ID is performed by the receiving device 40a instead of the transmitting device 30a, but even in this case, the same effects as in the first embodiment can be obtained.

[0095] <Third Embodiment> Next, a video transmission system according to the third embodiment will be described. Figure 10 shows a video transmission system 3 according to the third embodiment. The video transmission system 3 comprises a plurality of imaging devices 10, a plurality of first transmitting devices 20, a second transmitting device 30b, and a receiving device 40b. The imaging devices 10 and first transmitting devices 20 in this embodiment are the same as the imaging devices 10 and first transmitting devices 20 in the first embodiment, so their description will be omitted. The second transmitting device 30b and receiving device 40b in this embodiment are partially the same as the second transmitting device 30 and receiving device 40 in the first embodiment, so the same reference numerals are used for common components, and their descriptions will be omitted as appropriate.

[0096] In the first and second embodiments described above, a transmission path that transmits bidirectionally is assumed as the transmission path between the second transmitting devices 30, 30a and the receiving devices 40, 40a. On the other hand, in this embodiment, a transmission path that transmits unidirectionally is assumed as the transmission path between the second transmitting device 30b and the receiving device 40b. This transmission path may be a transmission path that supports IP multicast, or a transmission path that supports broadcast waves. Note that IP multicast on a communication line may also be transmitted using broadcast waves. Broadcast systems that support the transmission of IP packets, such as ISDB (Integrated Services Digital Broadcasting)-S3 and ATSC (Advanced Television Systems Committee) 3.0, can be considered to use a transmission path similar to that of IP multicast on a communication line.

[0097] The second transmitting device 30b of this embodiment differs from the second transmitting device 30 of the first embodiment in that it does not include a viewpoint information receiving unit 35 and a position determination unit 36, and it includes a video transmitting unit 37b instead of a video transmitting unit 37.

[0098] When IP multicast transmission is performed over a communication line, the video transmission unit 37b receives a viewing request from the viewing request receiving unit 34 and retrieves all video signals related to the requested content from the storage unit 33. The video transmission unit 37b then simultaneously transmits all video signals to the receiving device 40b. Metadata (ID information and location information) is multiplexed into the video signals, as in other embodiments. For example, if the video signals of the requested content are grouped as shown in Figure 4, the video transmission unit 37b simultaneously transmits 145 video signals to the receiving device 40b.

[0099] On the other hand, when broadcasting, there is no viewing request from the user, and the video transmission unit 37b always acquires all video signals related to all content from the storage unit 33 without any viewing request from the receiving device 40b. The video transmission unit 37b then simultaneously transmits all video signals related to all content to the receiving device 40b. The selection of content is then performed within the receiving device 40b. In other words, in the case of a broadcast transmission line, there is no need to have a viewing request transmission unit 41 and a viewing request reception unit 34.

[0100] The receiving device 40b of this embodiment differs from the receiving device 40 of the first embodiment in that it does not have a viewpoint information transmission unit 46, it further includes an ID information detection unit 50, a position determination unit 52, and a viewpoint change unit 53, and it includes a viewpoint insertion unit 47b and an operation information addition unit 48b instead of a viewpoint insertion unit 47 and an operation information addition unit 48. The ID information detection unit 50, the position determination unit 52, and the viewpoint change unit 53 also constitute a control unit.

[0101] When the position determination unit 52 receives viewpoint information from the viewpoint position detection unit 45, it determines whether or not there has been a change in the viewpoint position. For example, the position determination unit 52 determines that there has been a change in the viewpoint position if the amount of change in the viewpoint position exceeds a threshold. If the position determination unit 52 determines that there has been a change in the viewpoint position, it identifies the position information of the multiple imaging devices 10 that have captured multiple video signals necessary to generate an image corresponding to the user's viewpoint position, and outputs the identified position information to the viewpoint interpolation unit 47b.

[0102] When the viewpoint interpolation unit 47b receives location information from the location determination unit 52, it obtains the decoded video corresponding to the location information from the storage unit 44. The viewpoint interpolation unit 47b then performs viewpoint interpolation using the obtained decoded video and outputs the viewpoint-interpolated decoded video to the operation information addition unit 48b.

[0103] When a user requests a change in viewpoint position by performing an operation based on the operation information displayed on the display unit 49, the ID information detection unit 50 detects the ID (request ID) assigned to the group of images corresponding to the new viewpoint position requested by the user. The ID information detection unit 50 then outputs the detected request ID to the viewpoint change unit 53.

[0104] When the viewpoint change unit 53 receives a request ID from the ID information detection unit 50, it retrieves the default decoded video from the decoded video of the video signal group to which the request ID has been assigned, from the storage unit 44, and outputs it to the operation information addition unit 48b.

[0105] When content is played back, the operation information adding unit 48b first obtains the default decoded video from the storage unit 44, adds operation information to the decoded video, and outputs it to the display unit 49. After viewpoint interpolation is performed by the viewpoint interpolation unit 47b, the operation information adding unit 48b obtains the viewpoint-interpolated decoded video from the viewpoint interpolation unit 47b, adds operation information to the decoded video, and outputs it to the display unit 49. After the viewpoint change unit 53 detects the request ID, the operation information adding unit 48b obtains the default decoded video with the viewpoint position changed from the viewpoint change unit 53, adds operation information to the decoded video, and outputs it to the display unit 49.

[0106] (Action sequence) Next, the operation sequence of the video transmission system 3 according to this embodiment will be described with reference to Figure 11. Figure 11 is a diagram showing the operation sequence between the second transmitting device 30b and the receiving device 40b when transmitting a video signal via IP multicast.

[0107] In step S301, the receiving device 40b transmits a viewing request indicating the content selected by the user to the second transmitting device 30b via the viewing request transmission unit 41. The second transmitting device 30b receives the viewing request transmitted from the receiving device 40b via the viewing request receiving unit 34.

[0108] In step S302, the second transmitting device 30b transmits all video signals related to the content requested by the receiving device 40b, along with metadata (ID information and location information), via the video transmitting unit 37b. The receiving device 40b receives all video signals related to the requested content, along with metadata, from the second transmitting device 30b via the video receiving unit 42.

[0109] In step S303, the receiving device 40b decodes the video signal using the video decoding unit 43. Then, the receiving device 40b displays the default video of the requested content using the display unit 49.

[0110] In step S304, the receiving device 40b uses the position determination unit 52 to determine whether there is a change in viewpoint position (a minor change) that does not involve a change in ID. If there is a minor change in viewpoint position, the receiving device 40b proceeds to step S305; otherwise, it proceeds to step S306.

[0111] In step S305, the receiving device 40b uses the viewpoint interpolation unit 47b to perform viewpoint interpolation processing using multiple decoded images and generate an image corresponding to the user's viewpoint position. The receiving device 40b then displays the viewpoint-interpolated image on the display unit 49. The receiving device 40b then returns to step S304.

[0112] In step S306, the receiving device 40b uses the ID information detection unit 50 to determine whether there is a change in viewpoint position (a major change) that involves a change in ID. If there is a major change in viewpoint position, the receiving device 40b detects the request ID and proceeds to step S307; if there is no major change in viewpoint position, it returns to step S304.

[0113] In step S307, the receiving device 40b uses the viewpoint changing unit 53 to obtain the default decoded video from the decoded video of the video signal group to which the request ID has been assigned. The receiving device 40b then displays the default video on the display unit 49.

[0114] In the case of broadcast transmission, the operation in step S301 is not performed. In step S302, the second transmitting device 30b transmits all video signals related to all content, along with metadata (ID information and location information), to the receiving device 40b via the video transmitting unit 37b. Then, in step S303, the receiving device 40b displays the default video of the content selected by the user. The operation from step S304 onward is the same as in the case of IP multicast transmission.

[0115] As described above, the second transmitting device 30b transmits all video signals to the receiving device 40b along with metadata (ID information indicating the grouping of the first and second layers, and location information). Therefore, when the viewpoint position changes, the receiving device 40b can identify the decoded video required in accordance with the change in viewpoint position. Thus, the present invention can also be applied when broadcasting or IP multicast distributing video signals.

[0116] <Program> To enable the first transmitting device 20, the second transmitting devices 30, 30a, 30b, and the receiving devices 40, 40a, 40b described above, it is also possible to use a computer capable of executing program instructions for each of them. Figure 12 is a block diagram illustrating the schematic configuration of a computer that functions as the first transmitting device 20, the second transmitting devices 30, 30a, 30b, or the receiving devices 40, 40a, 40b. Here, the computer 100 may be a general-purpose computer, a dedicated computer, a workstation, a PC, an electronic notepad, etc. Program instructions may be program code, code segments, etc., for executing the required tasks.

[0117] As shown in Figure 12, the computer 100 comprises a processor 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, storage 140, an input unit 150, an output unit 160, and a communication interface (I / F) 170. Each component is connected to the others via a bus 180 so as to be able to communicate with each other. The processor 110 is specifically a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), SoC (System on a Chip), etc., and may be composed of multiple processors of the same or different types.

[0118] The processor 110 controls each configuration and performs various arithmetic operations. Specifically, the processor 110 reads a program from the ROM 120 or storage 140 and executes the program using the RAM 130 as a working area. The processor 110 controls each configuration and performs various arithmetic operations according to the program stored in the ROM 120 or storage 140. In this embodiment, the program according to the disclosure is stored in the ROM 120 or storage 140.

[0119] The program may be recorded on a recording medium readable by the computer 100. Using such a recording medium, the program can be installed on the computer 100. The recording medium on which the program is recorded may be a non-transitory recording medium. A non-transitory recording medium is not particularly limited, but may include, for example, a CD-ROM, DVD-ROM, or USB (Universal Serial Bus) memory. Alternatively, the program may be downloaded from an external device via a network.

[0120] ROM120 stores various programs and data. RAM130 temporarily stores programs or data as a working area. Storage140 consists of an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and data, including the operating system.

[0121] The input unit 150 includes one or more input interfaces that receive user input operations and acquire information based on the user operations. For example, the input unit 150 is a pointing device, a keyboard, a mouse, etc., but is not limited to these.

[0122] The output unit 160 includes one or more output interfaces for outputting information. For example, the output unit 160 is a display that outputs information as an image, or a speaker that outputs information as sound, but is not limited to these. If the output unit 160 is a touch panel display, it also functions as an input unit 150.

[0123] The communication interface 170 is an interface for communicating with external devices.

[0124] Although the embodiments described above are representative examples, it will be apparent to those skilled in the art that many modifications and substitutions are possible within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the embodiments described above, and various modifications and changes are possible without departing from the claims. For example, multiple component blocks or processing steps described in the embodiments can be combined into one, or one can be divided into multiple. [Explanation of Symbols]

[0125] 1,2,3 Video transmission system 10 Imaging device 20. First Transmitter 21 Video receiving unit 22 Video Encoding Section 23. First ID Assignment Section 24 Memory section 25 Video transmission unit 30, 30a, 30b Second Transmitter 31 Video receiving unit 32 2nd ID assignment section 33 Storage section 34 Viewing Request Receiving Unit 35,35a Viewpoint information receiving unit 36 Position determination section 37 Video transmission unit 40, 40a, 40b receiving device 41 Viewing Request Transmission Unit 42 Video receiving unit 43 Video Decoding Unit 44 Storage section 45 Viewpoint position detection unit 46,46a Viewpoint information transmission unit 47,47b Viewpoint interpolation 48,48b Operation information addition section 49 Display section 50 ID Information Detection Unit 52 Position determination section 53. Viewpoint Change Section 100 Computers 110 processors 120 ROM 130 RAM 140 storage 150 Input section 160 Output section 170 Communication Interfaces 180 bus 481 Video 482~486 Operation information

Claims

1. A receiving device that receives multiple video signals transmitted by a transmitting device and captured from different locations, along with location information of the imaging device that captured the video signals, The aforementioned video signals are grouped into multiple groups according to the shooting position. A communication interface that simultaneously receives all video signals from the transmitting device that are assigned an ID for identifying the group, The system includes a control unit that decodes the aforementioned video signal to generate a decoded video and acquires the user's viewpoint position, If there is a change in viewpoint position that does not involve a change in the ID, the control unit identifies the location information of the imaging device that captured multiple video signals necessary to generate an image corresponding to the user's viewpoint position, and performs viewpoint interpolation using the decoded image corresponding to the location information. A receiving device that, in the event of a change in viewpoint position accompanied by a change in the aforementioned ID, acquires the default decoded video from the decoded video of the video group corresponding to the new viewpoint position.

2. The receiving device according to claim 1, wherein the first-level group among the plurality of groups is a group to which a plurality of video signals used in the receiving device to generate images from a viewpoint different from the shooting position belong.

3. The receiving device according to claim 2, wherein the second-tier group among the plurality of groups is one or more first-tier groups to which video signals captured in the same location belong.

4. The receiving device according to claim 2 or 3, wherein the video signals belonging to the first hierarchical group are compressed by predictive coding using correlations between video signals.

5. The receiving device according to any one of claims 1 to 4, wherein the ID and the video signal are stored in a file in ISO Base Media File Format.

Images