Receiving device

Abstract

To allow a video signal required for viewpoint interpolation to be easily identified when a user changes a viewpoint position.SOLUTION: A receiving device (40b) receives video signals grouped according to shooting positions from a transmitting device (30b) together with position information of an imaging device (10). The receiving device includes a communication interface (42) that simultaneously receives all the video signals with assigned group IDs from the transmitting device, and a control unit (43, 45, 47b, 48b, 50, 52, 53) that decodes the video signals and acquires a user's viewpoint position. When there is a change in the viewpoint position that does not involve a change in the ID, the control unit identifies position information of the imaging device that has captured a plurality of video signals required to generate a video corresponding to the user's viewpoint position, and performs viewpoint interpolation using a decoded video corresponding to the position information. When there is a change in the viewpoint position that involves a change in the ID, the control unit acquires a default decoded video from among decoded videos of a video group that corresponds to a new viewpoint position.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to a receiving apparatus for receiving a video signal.

Background Art

[0002] As a development of 360-degree video, a video system that allows a user to view 360-degree video from a preferred perspective has been studied. For example, Non-Patent Document 1 lists use cases such as "3D image message" and "immersive 6DoF streaming" that involve viewpoint movement within the range where the user moves their head while sitting as 3DoF+ (Degree of Freedom), or viewpoint movement within the range where the user moves freely as 6DoF.

[0003] Generally, a single camera captures video from a single viewpoint. By using multiple videos (multi-viewpoint videos) captured by multiple cameras, it is possible to generate a video from an intermediate position among the positions where they were captured. This is a known technique called viewpoint interpolation, intermediate viewpoint image generation, etc. (see, for example, Patent Document 1).

[0004] In addition, Non-Patent Document 2 defines OMAF (Omnidirectional Media Format) as a file format for storing 360-degree video signals of predetermined viewpoints.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] In viewpoint interpolation, a video from a desired viewpoint is generated using a plurality of video signals captured from different positions. On the other hand, for a service that allows a user to view a video from a freely moving viewpoint such as 6DoF, it is necessary to capture a plurality of videos centered on many points and perform viewpoint interpolation. However, the captured video signals are independent of each other, and there is no information for associating them. Although OMAF can store video signals with a fixed viewpoint, the processing of a plurality of video signals when the viewpoint is freely moved is not supported. Therefore, the receiving terminal has no way to identify what video signals exist and how they are related, and also has no way to identify how to associate the received plurality of video signals with each other.

[0008] In view of such circumstances, an object of the present invention is to provide a receiving device capable of easily identifying video signals necessary for performing viewpoint interpolation when a user changes the viewpoint position. [Means for Solving the Problems]

[0009] A receiving apparatus according to an embodiment is a receiving apparatus that receives a plurality of video signals captured from different positions, which are transmitted by a transmitting apparatus, together with position information of an imaging apparatus that captured the video signals. The video signals are grouped into a plurality of groups according to the shooting positions, and the communication interface simultaneously receives all the video signals to which an ID for identifying the group is assigned from the transmitting apparatus. And a control unit that decodes the video signal to generate a decoded video and acquires the viewpoint position of the user. When there is a change in the viewpoint position without a change in the ID, the control unit specifies the position information of the imaging apparatuses that captured the plurality of video signals necessary for generating the video corresponding to the viewpoint position of the user, and performs viewpoint interpolation using the decoded video corresponding to the position information. When there is a change in the viewpoint position accompanied by a change in the ID, a default decoded video is acquired from among the decoded videos of the video group corresponding to the new viewpoint position.

[0010] In one embodiment, the first - layer group among the plurality of groups may be a group to which a plurality of video signals used for generating a video from a viewpoint different from the shooting position in the receiving apparatus belong.

[0011] In one embodiment, the second - layer group among the plurality of groups may be one or more of the first - layer groups to which video signals captured within the same location belong.

[0012] In one embodiment, the video signals belonging to the first - layer group may be compressed by predictive coding using the correlation between video signals.

[0013] In one embodiment, the ID and the video signal may be stored in a file in the ISO Base Media File Format.

Advantages of the Invention

[0014] According to the present invention, when a user changes the viewpoint position, it becomes possible to easily specify the video signals necessary for performing viewpoint interpolation.

Brief Description of the Drawings

[0015]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0017] <First Embodiment> (Video Transmission System) First, the video transmission system according to the first embodiment will be described. FIG. 1 is a diagram showing a video transmission system 1 according to the first embodiment, and shows an overview of the transmission of a plurality of video signals captured at a plurality of locations. The video transmission system 1 includes a plurality of imaging devices (cameras) 10, a plurality of first transmission devices 20, a second transmission device 30, and a reception device 40. In this specification, the "location" refers to a predetermined area provided for a specific purpose, such as a venue, stadium, facility, park, etc. In this embodiment, the video transmission system 1 includes three first transmission devices 20. The first transmission device 20A transmits a video signal captured at location A, the first transmission device 20B transmits a video signal captured at location B, and the first transmission device 20C transmits a video signal captured at location C. The following description will be made based on this assumption.

[0018] The imaging device 10 attaches metadata to the video signal captured at a predetermined position and transmits it to the first transmission device 20. Here, the metadata transmitted by the imaging device 10 is position information indicating the shooting position of the imaging device 10, and may include information indicating the shooting direction. Further, the metadata may include distance information indicating the distance from the imaging device 10 to the object being photographed.

[0019] That is, the plurality of imaging devices 10A installed at location A each attach metadata to the video signal captured at a predetermined position in location A and transmit it to the first transmission device 20A. Similarly, the plurality of imaging devices 10B installed at location B each attach metadata to the video signal captured at a predetermined position in location B and transmit it to the first transmission device 20B. Similarly, the plurality of imaging devices 10C installed at location C each attach metadata to the video signal captured at a predetermined position in location C and transmit it to the first transmission device 20C.

[0020] The first transmission device 20A receives video signals and metadata from a plurality of imaging devices 10A, assigns an ID, and transmits them to the second transmission device 30. Similarly, the first transmission device 20B receives video signals and metadata from a plurality of imaging devices 10B, assigns an ID, and transmits them to the second transmission device 30. Similarly, the first transmission device 20C receives video signals and metadata from a plurality of imaging devices 10C, assigns an ID, and transmits them to the second transmission device 30.

[0021] For the transmission from the imaging device 10 to the first transmission device 20 and the transmission from the first transmission device 20 to the second transmission device 30, SDI (Serial Digital Interface), IP (Internet Protocol), etc. can be used. More specifically, when using SDI, it can be transmitted with 12G-SDI, 3G-SDI, HD-SDI, etc., and when using IP, it can be transmitted by RTP (Real-Time Transport Protocol) based on the SMPTE (Society of Motion Picture and Television Engineers) ST 2110 standard. For example, when transmitting the video signal without encoding, it can be transmitted using SDI or IP, and when transmitting the video signal after encoding, it can be transmitted using IP. In this embodiment, the imaging device 10 transmits the video signal to the first transmission device 20 without encoding, and the first transmission device 20 transmits the video signal to the second transmission device 30 after encoding. However, the imaging device 10 may perform encoding, or the first transmission device 20 may not perform encoding.

[0022] The second transmission device 30 aggregates the video signals transmitted from a plurality of first transmission devices 20 and transmits the video signals to the receiving device 40 via a transmission path (e.g., an IP network). Note that it is not always necessary for the video signals to be aggregated at the second transmission device 30. The first transmission device 20 may directly transmit the video signals to the receiving device 40 via various transmission paths such as a broadcast transmission path, a terrestrial IMT (International Mobile Telecommunication) network, an optical fiber, or a Wi-Fi (registered trademark) line.

[0023] The receiving device 40 may be any device capable of receiving video transmission from the first transmitting device 20 or the second transmitting device 30. The receiving device 40 is, for example, a head-mounted display, VR (Virtual Reality) goggles, a tablet terminal, a smartphone, a PC (Personal Computer), or the like.

[0024] (First transmitting device) Next, the first transmitting device 20 according to the first embodiment will be described.

[0025] As shown in FIG. 1, the first transmitting device 20 includes a video receiving unit 21, a video encoding unit 22, a first ID adding unit 23, a storage unit 24, and a video transmitting unit 25. The video encoding unit 22 and the first ID adding unit 23 constitute a control unit (controller). The control unit may be constituted by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be constituted by a processor, or may be constituted by including both. The video receiving unit 21 and the video transmitting unit 25 constitute a communication interface.

[0026] The video receiving unit 21 acquires a video signal and metadata from a plurality of imaging devices 10 and outputs them to the video encoding unit 22.

[0027] The video encoding unit 22 encodes the video signal input from the video receiving unit 21 by an arbitrary video encoding method such as H.265 / HEVC (High Efficiency Video Coding) to generate an encoded video signal. The video encoding unit 22 may perform compression by predictive encoding using the correlation between the respective video signals for efficient transmission. Then, the video encoding unit 22 outputs the encoded video signal to the storage unit 24.

[0028] In the present invention, video signals are grouped into a plurality of groups according to the shooting positions. The plurality of video signals captured by the plurality of imaging devices 10 can be grouped in units of a group of video signals (used for viewpoint interpolation) that are used to generate a video from a viewpoint position different from the shooting position. In order to distinguish this grouping from the grouping described later, it is referred to as "first-layer grouping". That is, the first-layer group is a group to which a plurality of video signals used to generate a video from a viewpoint different from the shooting position in the receiving device 40 belong.

[0029] The first ID assigning unit 23 inputs an ID (first group ID) for identifying the first-layer group by the operation of the operator of the first transmitting device 20, and associates the first group ID with the video signal and stores it in the storage unit 24. That is, the first ID assigning unit 23 assigns the first group ID to the video signal.

[0030] Referring to FIG. 2, the first group ID will be described. FIG. 2 shows an example of shooting at a baseball field. A total of 15 imaging devices 10, 5 in the horizontal direction and 3 in the vertical direction, are installed behind the catcher of the baseball field. For convenience of explanation, the position information p of these 15 imaging devices 10 is set to 1 to 15. In addition, on the first base side of the baseball field, a total of 15 imaging devices 10, 5 in the horizontal direction and 3 in the vertical direction, are installed. For convenience of explanation, the position information p of these 15 imaging devices 10 is set to 16 to 30. In addition, on the third base side of the baseball field, a total of 15 imaging devices 10, 5 in the horizontal direction and 3 in the vertical direction, are installed. For convenience of explanation, the position information p of these 15 imaging devices 10 is set to 31 to 45. Note that in the example shown in FIG. 2, the imaging device 10 captures only the front video from the shooting position, but may capture the 360-degree full-peripheral video from the shooting position.

[0031] The receiving device 40 can receive the video signals captured by the imaging devices 10 with position information p = 1 to 15, and by performing viewpoint interpolation, present the user with videos from viewpoints within a certain range from behind the catcher. For example, if the receiving device 40 is displaying the video signal captured by the imaging device 10 with position information p = 8, and then the user moves within a predetermined range (for example, a range with a radius of several meters), the video corresponding to the user's viewpoint can be presented to the user. Similarly, the receiving device 40 can receive the video signals captured by the imaging devices 10 with position information p = 16 to 30, and present the user with videos from viewpoints within a certain range from the first base side. Similarly, the receiving device 40 can receive the video signals captured by the imaging devices 10 with position information p = 31 to 45, and present the user with videos from viewpoints within a certain range from the third base side.

[0032] For example, as shown in FIG. 2, when captured by the imaging devices 10 with position information p = 1 to 45, the first ID assigning unit 23 assigns "I" as the first group ID to the 15 video signals captured from behind the catcher, assigns "II" as the first group ID to the 15 video signals captured from the first base side, and assigns "III" as the first group ID to the 15 video signals captured from the third base side. The first ID assigning unit 23 may assign the first group ID to all of the video signals. Also, when the video encoding unit 22 performs compression using the correlation between the respective video signals, the first ID assigning unit 23 may assign the first group ID only to the reference video signals (for example, the video signals captured by the imaging devices 10 with position information p = 8, 23, 38).

[0033] Referring to FIG. 1 again, the storage unit 24 stores the encoded video signal and metadata (position information) input from the video encoding unit 22, and the metadata (information indicating the first group ID) input from the first ID assigning unit 23.

[0034] The video transmitting unit 25 transmits the encoded video signal and metadata (position information and information indicating the first group ID) stored in the storage unit 24 to the second transmitting device 30.

[0035] Note that the imaging device 10 and part or all of the first transmission device 20 may be integrated. For example, when the imaging device 10 has the function of the video encoding unit 22, the video receiving unit 21 of the first transmission device 20 receives the encoded video signal from the imaging device 10 and outputs it to the storage unit 24.

[0036] (Second transmission device) Next, the second transmission device 30 according to the first embodiment will be described.

[0037] As shown in FIG. 1, the second transmission device 30 includes a video receiving unit 31, a second ID adding unit 32, a storage unit 33, a viewing request receiving unit 34, a viewpoint information receiving unit 35, a position determination unit 36, and a video transmission unit 37. The second ID adding unit 32 and the position determination unit 36 constitute a control unit. The control unit may be constituted by dedicated hardware such as an ASIC or an FPGA, may be constituted by a processor, or may be constituted by including both. The video receiving unit 31, the viewing request receiving unit 34, the viewpoint information receiving unit 35, and the video transmission unit 37 constitute a communication interface.

[0038] The video receiving unit 31 receives the encoded video signals and metadata transmitted from a plurality of first transmission devices 20, and outputs the received encoded video signals and metadata to the storage unit 33.

[0039] The video signals captured by the plurality of imaging devices 10 can be grouped in units of a group of video signals captured within the same location. To distinguish this grouping from the above-described first-level grouping, it is referred to as "second-level grouping". That is, the second-level group is one or more first-level groups to which a plurality of video signals captured within the same location belong.

[0040] The second ID assigning unit 32 inputs an ID (second group ID) for identifying a group in the second layer by an operation of an operator of the second transmission device 30, and stores the second group ID in the storage unit 33 in association with the video signal. That is, the second ID assigning unit 32 assigns the second group ID to the video signal.

[0041] Referring to FIG. 3, the second group ID will be described. In the example shown in FIG. 3, the second transmission device 30 receives a video signal captured at location A from the first transmission device 20A. Specifically, the second transmission device 30 receives 15 video signals with the first group ID = I captured by the imaging device 10 with position information p = 1 to 15, 15 video signals with the first group ID = II captured by the imaging device 10 with position information p = 16 to 30, and 15 video signals with the first group ID = III captured by the imaging device 10 with position information p = 31 to 45. Further, the second transmission device 30 receives a video signal captured at location B from the first transmission device 20B. Specifically, the second transmission device 30 receives 15 video signals with the first group ID = I captured by the imaging device 10 with position information p = 46 to 60, 15 video signals with the first group ID = II captured by the imaging device 10 with position information p = 61 to 75, and 15 video signals with the first group ID = III captured by the imaging device 10 with position information p = 76 to 90 from the first transmission device 20B. Further, the second transmission device 30 receives a video signal captured at location C from the first transmission device 20C. Specifically, the second transmission device 30 receives 15 video signals with the first group ID = I captured by the imaging device 10 with position information p = 91 to 105, 15 video signals with the first group ID = II captured by the imaging device 10 with position information p = 106 to 120, and 15 video signals with the first group ID = III captured by the imaging device 10 with position information p = 121 to 135 from the first transmission device 20C.

[0042] Further, the second ID assigning unit 32 assigns, for example, "A" as the second group ID to 45 video signals captured at location A, assigns, for example, "B" as the second group ID to 45 video signals captured at location B, and assigns, for example, "C" as the second group ID to 45 video signals captured at location C.

[0043] FIG. 4 shows a hierarchical structure in which video signals are grouped in such a two - layer manner. As shown in FIG. 3, it is assumed that 45 video signals are captured at each of location A, location B, and location C. In this case, the 45 video signals captured at location A are classified into group A with the second group ID = A. Further within group A, they are classified into 15 video signals belonging to the first group ID = I, 15 video signals belonging to the first group ID = II, and 15 video signals belonging to the first group ID = III. Also, the 45 video signals captured at location B are classified into group B with the second group ID = B. Further within group B, they are classified into 15 video signals belonging to the first group ID = I, 15 video signals belonging to the first group ID = II, and 15 video signals belonging to the first group ID = III. Also, the 45 video signals captured at location C are classified into group C with the second group ID = C. Further within group C, they are classified into 15 video signals belonging to the first group ID = I, 15 video signals belonging to the first group ID = II, and 15 video signals belonging to the first group ID = III.

[0044] Referring to FIG. 1 again, the storage unit 33 stores the encoded video signal and metadata input from the video receiving unit 31, and the metadata input from the second ID assigning unit 32.

[0045] When the viewing request receiving unit 34 receives a viewing request indicating the content to be viewed from the receiving device 40, it outputs the viewing request to the video transmitting unit 37.

[0046] When the viewpoint information receiving unit 35 receives viewpoint information indicating the viewpoint position of the user of the receiving device 40 (for example, the position of the receiving device 40) from the receiving device 40, it outputs the viewpoint information to the position determination unit 36. For example, the viewpoint position information is three-dimensional coordinates in a global coordinate system. Also, the viewpoint position information may be three-dimensional coordinates in a local coordinate system based on a default user position.

[0047] When the position determination unit 36 receives the viewpoint information from the viewpoint information receiving unit 35, it determines whether the change in the user's viewpoint position is accompanied by a change in the ID (the first group ID and the second group ID). Specifically, the position determination unit 36 identifies the request ID, which is the ID of the video signal corresponding to the change in the user's viewpoint position, and determines whether the request ID is the same as the ID of the video signal transmitted immediately before.

[0048] When the request ID is the same as the ID of the video signal transmitted immediately before, the position determination unit 36 outputs the position information of the plurality of imaging devices 10 that captured the plurality of video signals necessary for generating the video corresponding to the user's viewpoint position among the group of video signals to which the request ID is assigned to the video transmission unit 37. For example, the position determination unit 36 obtains the shooting position corresponding to the user's viewpoint position, and outputs the position information of the imaging device 10 arranged at a position adjacent to the shooting position or at a position surrounding the shooting position to the video transmission unit 37.

[0049] Also, when the request ID is different from the ID of the video signal transmitted immediately before, the position determination unit 36 outputs the position information of the imaging device 10 that captured the default video signal among the group of video signals to which the request ID is assigned to the video transmission unit 37.

[0050] The video transmission unit 37 transmits a video signal to the receiving device 40. In this specification, when it is said that "a video signal is transmitted", it means that the metadata (multiplexed) attached to the video signal is also transmitted simultaneously. Also, the video transmission unit 37 can transmit only the metadata. The video signals belonging to the first-layer group transmitted by the video transmission unit 37 may be compressed by predictive coding using the correlation between video signals by the video encoding unit 22 of the first transmission device 20.

[0051] When a viewing request is input from the viewing request receiving unit 34 to the video transmission unit 37, the video transmission unit 37 acquires from the storage unit 33 the default video signal of the content requested for viewing, all ID information (information indicating the first group ID and the second group ID) regarding the content requested for viewing, and the position information of the imaging device 10 representing each first-layer group, and transmits them to the receiving device 40. The receiving device 40 presents to the user what viewpoint positions can be selected based on the ID information and the position information of the imaging device 10.

[0052] Also, when position information is input to the video transmission unit 37 from the position determination unit 36, the video transmission unit 37 acquires from the storage unit 33 the video signal captured by the imaging device 10 having the position information (the video signal corresponding to the position information), and transmits it to the receiving device 40. That is, when the request ID is the same as the ID of the video signal transmitted immediately before, the video transmission unit 37 transmits a plurality of video signals necessary for generating the video corresponding to the user's viewpoint position among the group of video signals to which the request ID is assigned, and when the request ID is different from the ID of the video signal transmitted immediately before, the video transmission unit 37 transmits the default video signal among the group of video signals to which the request ID is assigned.

[0053] FIG. 5 is a diagram showing a configuration example of a file when IDs and video signals are stored in a file in the ISO Base Media File Format such as OMAF. This format is composed of basic units called boxes.

[0054] In the file configuration shown in FIG. 5A, when the second group ID and the first group ID store n common video signals in a file, by storing the n video signals under the boxes indicating the second group ID and the first group ID, the group to which the video signal belongs is indicated. In the file configuration shown in FIG. 5B, when the second group ID and the first group ID store n common video signals in a file, by storing the boxes indicating the second group ID and the first group ID together with the tracks storing the respective video signals, the group to which the video signal belongs is indicated.

[0055] Note that, in this embodiment, a two - layer grouping has been described, but for user operations, three - layer or more grouping may be performed according to the content. Even when performing three - layer or more grouping, the video signals can be identified hierarchically in the same way. For example, one or more second - layer groups to which video signals captured within the same region belong may be grouped as a third - layer group.

[0056] (Receiving device) Next, the receiving device 40 according to the first embodiment will be described.

[0057] As shown in FIG. 1, the receiving device 40 includes a viewing request transmission unit 41, a video receiving unit 42, a video decoding unit 43, a storage unit 44, a viewpoint position detection unit 45, a viewpoint information transmission unit 46, a viewpoint interpolation unit 47, an operation information addition unit 48, and a display unit 49. The video decoding unit 43, the viewpoint position detection unit 45, the viewpoint interpolation unit 47, and the operation information addition unit 48 constitute a control unit. The control unit may be constituted by dedicated hardware such as an ASIC or an FPGA, may be constituted by a processor, or may be constituted by including both. The viewing request transmission unit 41, the video receiving unit 42, and the viewpoint information transmission unit 46 constitute a communication interface.

[0058] The viewing request transmission unit 41 transmits a viewing request indicating the content selected by the user of the receiving device 40 to the second transmission device 30.

[0059] The video receiving unit 42 receives a video signal and metadata (position information and ID information) from the second transmission device 30. The video receiving unit 42 outputs the received video signal and metadata to the video decoding unit 43 after associating them with each other.

[0060] The video decoding unit 43 decodes the video signal to generate a decoded video and outputs it to the storage unit 44.

[0061] The storage unit 44 stores the decoded video generated by the video decoding unit 43.

[0062] The viewpoint position detection unit 45 detects the viewpoint position of the user and outputs viewpoint information indicating the detected viewpoint position to the viewpoint information transmission unit 46. For example, the viewpoint position detection unit 45 acquires an image of the user's eyes captured by an imaging device included in the receiving device 40, and analyzes the image using any known technique to detect the viewpoint position. Also, the viewpoint position detection unit 45 may be a line-of-sight sensor provided separately from the control unit in the receiving device 40. The control unit acquires the viewpoint position of the user from the viewpoint position detection unit 45.

[0063] The viewpoint information transmission unit 46 transmits the viewpoint information detected by the viewpoint position detection unit 45 to the second transmission device 30. The viewpoint information transmission unit 46 may transmit the viewpoint information to the second transmission device 30 at a predetermined interval, or may transmit the viewpoint information to the second transmission device 30 only when the amount of change in the viewpoint position exceeds a threshold value.

[0064] The viewpoint interpolation unit 47 performs viewpoint interpolation using a plurality of decoded videos (that is, decoded videos obtained by decoding video signals corresponding to the position information obtained by the position determination unit 36) stored in the storage unit 44 and having the same first group ID and second group ID, and generates a video corresponding to the viewpoint position detected by the viewpoint position detection unit 45. Any known technique (for example, see Patent Document 1) can be used for the viewpoint interpolation. The viewpoint interpolation unit 47 outputs the viewpoint-interpolated decoded video to the operation information addition unit 48.

[0065] The operation information adding unit 48 acquires a default decoded video from the storage unit 44 in response to the user's viewing request, adds operation information to the decoded video, and outputs it to the display unit 49. After the viewpoint interpolation by the viewpoint interpolation unit 47 is performed, the operation information adding unit 48 acquires the decoded video interpolated in viewpoint from the viewpoint interpolation unit 47, adds operation information to the decoded video, and outputs it to the display unit 49.

[0066] Here, the operation information is information indicating an operation method for changing the viewpoint position for the user. For example, the operation information is information indicating an operation method for changing to different viewpoint positions at the same location or for changing to viewpoint positions at different locations. The operation information may include an icon. By operating based on the operation information, the user can request a change in the viewpoint position. For example, when the receiving device 40 is a head-mounted display, the user may perform an operation based on the operation information according to the direction of the line of sight. At this time, the line-of-sight direction is detected by the viewpoint position detection unit 45. Also, when the receiving device 40 is a tablet terminal, an operation based on the operation information may be performed by a screen operation such as a flick or a swipe. Further, when the receiving device 40 is a non-mobile or non-portable device having a flat panel display, an operation based on the operation information may be performed using a remote control.

[0067] The display unit 49 is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 49 displays the decoded video and the operation information input from the operation information adding unit 48.

[0068] Fig. 6 shows an example of the display of video and operation information on the display unit 49. In the example shown in Fig. 6, video 481 and operation information 482 to 486 are being displayed. Operation information 482 indicates that the video 481 being displayed is based on the video group of the first group with ID = II at location C. Operation information 483 indicates that the viewpoint position at location C can be changed by a predetermined operation (the video 481 being displayed can be changed to a video based on the video group of the first group with ID = I at location C). Operation information 484 indicates that the viewpoint position at location C can be changed by a predetermined operation (the video 481 being displayed can be changed to a video based on the video group of the first group with ID = III at location C). Operation information 485 indicates that the viewpoint position can be changed to a different location by a predetermined operation (the video 481 being displayed can be changed to a video shot at location A). Operation information 486 indicates that the viewpoint position can be changed to a different location by a predetermined operation (the video 481 being displayed can be changed to a video shot at location B).

[0069] When the user operates based on the operation information displayed on the display unit 49, the viewpoint position detection unit 45 detects the viewpoint position corresponding to the operation. Thereby, the user can request the second transmission device 30 to change the viewpoint position.

[0070] (Operation sequence) Next, the operation sequence of the video transmission system 1 according to the present embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing the operation sequence between the second transmission device 30 and the reception device 40.

[0071] In step S101, the reception device 40 transmits, by the viewing request transmission unit 41, a viewing request indicating the content selected by the user to the second transmission device 30. The second transmission device 30 receives the viewing request transmitted from the reception device 40 by the viewing request reception unit 34.

[0072] In step S102, the second transmission device 30 transmits, by the video transmission unit 37, all the ID information and the position information of the imaging device 10 representing each first-layer group for the content requested for viewing from the receiving device 40 to the receiving device 40. The receiving device 40 receives, by the video receiving unit 42, the above information regarding the content requested for viewing from the second transmission device 30. For example, when the installation position of the imaging device 10 is as shown in FIG. 2, the position information of the imaging device 10 representing each first-layer group is set to the position information p = 8, 23, 38 of the imaging device 10 arranged at the center respectively. Note that the video transmission unit 37 may transmit a plurality of pieces of position information to the receiving device 40 for each first-layer grouping.

[0073] In step S103, the second transmission device 30 transmits, by the video transmission unit 37, a default video signal for the content requested for viewing from the receiving device 40 to the receiving device 40. It is assumed that the second transmission device 30 has predetermined a default video signal for each content. For example, when the video signal regarding the content requested for viewing is grouped as shown in FIG. 4, the default video is set to the video signal 8 of the second group ID = A and the first group ID = I (the video signal captured by the imaging device 10 with the position information p = 8 arranged at the center behind the catcher in FIG. 2).

[0074] In step S104, the receiving device 40 receives, by the video receiving unit 42, the default video signal of the content requested for viewing from the second transmission device 30. Then, the receiving device 40 decodes the video signal by the video decoding unit 43 and displays the default video by the display unit 49.

[0075] In step S105, the receiving device 40 transmits viewpoint information to the second transmission device 30 by the viewpoint information transmission unit 46.

[0076] In step S106, the second transmission device 30 specifies a request ID, which is the ID of the video signal corresponding to the change in the user's viewpoint position, by the position determination unit 36. When the request ID is the same as the ID of the video signal transmitted immediately before, the second transmission device 30 advances the process to step S107. When the request ID is different from the ID of the video signal transmitted immediately before, the second transmission device 30 advances the process to step S109.

[0077] In step S107, the second transmission device 30 transmits, by the video transmission unit 37, a video signal necessary for generating a video corresponding to the user's viewpoint position to the receiving device 40.

[0078] In step S108, the receiving device 40 performs viewpoint interpolation processing using a plurality of video signals necessary for viewpoint interpolation received from the second transmission device 30 by the viewpoint interpolation unit 47. Then, the receiving device 40 displays the viewpoint-interpolated video on the display unit 49. Then, the receiving device 40 returns the process to step S105.

[0079] In step S109, the second transmission device 30 transmits, by the video transmission unit 37, the default video signal among the video signal group to which the request ID is assigned to the receiving device 40.

[0080] In step S110, the receiving device 40 receives, by the video receiving unit 42, the default video signal with the changed viewpoint position from the second transmission device 30. Then, the receiving device 40 decodes the video signal by the video decoding unit 43 and displays the default video on the display unit 49. Then, the receiving device 40 returns the process to step S105.

[0081] As described above, the video transmission system 1 that transmits a plurality of video signals captured at different positions groups the video signals and assigns IDs. By grouping the plurality of video signals as a first-layer group, it becomes possible to easily identify the plurality of video signals for use in viewpoint interpolation. Further, by grouping the plurality of video signals as a second-layer group, it becomes possible to present to the user what kind of viewpoint position changes are possible. Then, the user can view videos with the viewpoint position changed within the same location and with the viewpoint position changed to different locations by operating according to the operation information displayed on the display unit 49.

[0082] <Second Embodiment> Next, the video transmission system according to the second embodiment will be described. FIG. 8 is a diagram showing a video transmission system 2 according to the second embodiment. The video transmission system 2 includes a plurality of imaging devices 10, a plurality of first transmission devices 20, a second transmission device 30a, and a reception device 40a. Since the imaging device 10 and the first transmission device 20 of the present embodiment are the same as the imaging device 10 and the first transmission device 20 of the first embodiment, the description thereof will be omitted. Since the second transmission device 30a and the reception device 40a of the present embodiment are partially common to the second transmission device 30 and the reception device 40 of the first embodiment, the same reference numerals are given to the common configurations, and the description thereof will be omitted as appropriate.

[0083] The reception device 40a of the present embodiment is different from the reception device 40 of the first embodiment in that it further includes an ID information detection unit 50 and that it includes a viewpoint information transmission unit 46a instead of the viewpoint information transmission unit 46. The ID information detection unit 50 also constitutes a control unit.

[0084] When the ID information detection unit 50 requests a change in the viewpoint position by the user performing an operation based on the operation information displayed on the display unit 49, it detects the ID (request ID) assigned to the video signal group corresponding to the new viewpoint position requested by the user. Then, the ID information detection unit 50 outputs the detected request ID as viewpoint information to the viewpoint information transmission unit 46a. For example, in FIG. 6, when the user swipes along the arrow icon indicated by the operation information 483, the ID information detection unit 50 detects the first group ID = I and the second group ID = C as the request ID. Also, in FIG. 6, when the user swipes along the arrow icon indicated by the operation information 485, the ID information detection unit 50 detects the first group ID = default value and the second group ID = A as the request ID. Note that when the first group ID is the default value, only the second group ID may be included as the request ID.

[0085] When the user does not perform an operation based on the operation information displayed on the display unit 49, the viewpoint information transmission unit 46a transmits the viewpoint information detected by the viewpoint position detection unit 45 to the second transmission device 30a. When the user performs an operation based on the operation information displayed on the display unit 49, the viewpoint information transmission unit 46a transmits the viewpoint information (request ID) detected by the ID information detection unit 50 to the second transmission device 30a.

[0086] The second transmission device 30a of the present embodiment is different from the second transmission device 30 of the first embodiment in that it includes a viewpoint information reception unit 35a, a position determination unit 36a, and a video transmission unit 37a instead of the viewpoint information reception unit 35, the position determination unit 36, and the video transmission unit 37.

[0087] When the viewpoint information reception unit 35a receives viewpoint information that does not include a request ID, it outputs the viewpoint information to the position determination unit 36a in the same manner as in the first embodiment. Also, when the viewpoint information reception unit 35a receives viewpoint information that includes a request ID, it specifies the position information corresponding to the default video signal among the video signal group to which the request ID is assigned, and transmits the specified position information to the video transmission unit 37a.

[0088] When the viewpoint information receiving unit 35a inputs viewpoint information not including the request ID, the position determination unit 36a outputs the position information of a plurality of imaging devices 10 that have captured the video signals necessary for generating the video corresponding to the user's viewpoint position to the video transmission unit 37a. Further, when the viewpoint information receiving unit 35a inputs viewpoint information including the request ID, the position determination unit 36a outputs the position information corresponding to the default video signal among the group of video signals to which the request ID is assigned to the video transmission unit 37a.

[0089] When the position information is input from the viewpoint information receiving unit 35a or the position determination unit 36a, the video transmission unit 37a acquires the video signal (the video signal corresponding to the position information) captured by the imaging device 10 having the position information from the storage unit 33 and transmits it to the receiving device 40a.

[0090] (Operation sequence) Next, the operation sequence of the video transmission system 2 according to the present embodiment will be described with reference to FIG. 9. FIG. 9 is a diagram showing the operation sequence between the second transmission device 30a and the receiving device 40a. Steps S201 to S204 are the same as steps S101 to S104 of the first embodiment described with reference to FIG. 7, and thus the description thereof will be omitted.

[0091] In step S205, the receiving device 40a transmits the viewpoint information to the second transmission device 30a by the viewpoint information transmission unit 46a.

[0092] In step S206, the second transmission device 30a determines whether the request ID is included in the viewpoint information by the position determination unit 36a. When the request ID is not included in the viewpoint information, the second transmission device 30a proceeds to step S207, and when the request ID is included in the viewpoint information, the second transmission device 30a proceeds to step S209.

[0093] Steps S207 to S210 are the same as steps S107 to S110 of the first embodiment described with reference to FIG. 7, and thus the description thereof will be omitted.

[0094] As described above, in this embodiment, the detection of the request ID is performed by the receiving device 40a instead of the transmitting device 30a. Even in this case, the same effects as those of the first embodiment can be obtained.

[0095] <Third Embodiment> Next, a video transmission system according to the third embodiment will be described. FIG. 10 is a diagram showing a video transmission system 3 according to the third embodiment. The video transmission system 3 includes a plurality of imaging devices 10, a plurality of first transmitting devices 20, a second transmitting device 30b, and a receiving device 40b. Since the imaging device 10 and the first transmitting device 20 of this embodiment are the same as those of the imaging device 10 and the first transmitting device 20 of the first embodiment, the description thereof will be omitted. Since the second transmitting device 30b and the receiving device 40b of this embodiment are partially common to the second transmitting device 30 and the receiving device 40 of the first embodiment, the same reference numerals are given to the common configurations, and the description thereof will be omitted as appropriate.

[0096] In the above-described first and second embodiments, a transmission path that transmits bidirectionally is assumed as the transmission path between the second transmitting devices 30 and 30a and the receiving devices 40 and 40a. On the other hand, in this embodiment, a transmission path that transmits unidirectionally is assumed as the transmission path between the second transmitting device 30b and the receiving device 40b. The transmission path may be a transmission path corresponding to IP multicast or a transmission path corresponding to a broadcast wave. Note that IP multicast over a communication line may be transmission over a broadcast wave. Broadcast systems corresponding to the transmission of IP packets such as ISDB (Integrated Services Digital Broadcasting)-S3 and ATSC (Advanced Television Systems Committee) 3.0 are considered to have the same transmission path as IP multicast over a communication line.

[0097] The second transmission device 30b of this embodiment is different from the second transmission device 30 of the first embodiment in that it does not include a viewpoint information receiving unit 35 and a position determination unit 36, and in that it includes a video transmission unit 37b instead of the video transmission unit 37.

[0098] When performing IP multicast transmission over a communication line, when a viewing request is input from the viewing request receiving unit 34 to the video transmission unit 37b, the video transmission unit 37b acquires all video signals related to the content requested for viewing from the storage unit 33. Then, the video transmission unit 37b simultaneously transmits all the video signals to the receiving device 40b. Similar to other embodiments, metadata (ID information and position information) is multiplexed in the video signal. For example, when the video signals of the content requested for viewing are grouped as shown in FIG. 4, the video transmission unit 37b simultaneously transmits 145 video signals to the receiving device 40b.

[0099] On the other hand, when performing broadcast transmission, no viewing request is made by the user, and without depending on the viewing request of the receiving device 40b, the video transmission unit 37b always acquires all video signals related to all content from the storage unit 33. Then, the video transmission unit 37b simultaneously transmits all video signals related to all content to the receiving device 40b. Then, the selection of the content is performed within the receiving device 40b. That is, in the case of a broadcast transmission path, it is not necessary to include a viewing request transmission unit 41 and a viewing request receiving unit 34.

[0100] The receiving device 40b of this embodiment is different from the receiving device 40 of the first embodiment in that it does not include a viewpoint information transmission unit 46, in that it further includes an ID information detection unit 50, a position determination unit 52, and a viewpoint change unit 53, and in that it includes a viewpoint interpolation unit 47b and an operation information addition unit 48b instead of the viewpoint interpolation unit 47 and the operation information addition unit 48. The ID information detection unit 50, the position determination unit 52, and the viewpoint change unit 53 also constitute a control unit.

[0101] When the viewpoint information is input from the viewpoint position detection unit 45, the position determination unit 52 determines whether there is a change in the viewpoint position. For example, when the amount of change in the viewpoint position exceeds a threshold value, the position determination unit 52 determines that there is a change in the viewpoint position. Then, when the position determination unit 52 determines that there is a change in the viewpoint position, it identifies the position information of a plurality of imaging devices 10 that have captured a plurality of video signals necessary for generating a video corresponding to the user's viewpoint position, and outputs the identified position information to the viewpoint interpolation unit 47b.

[0102] When the position information is input from the position determination unit 52, the viewpoint interpolation unit 47b acquires a decoded video corresponding to the position information from the storage unit 44. Then, the viewpoint interpolation unit 47b performs viewpoint interpolation using the acquired decoded video, and outputs the viewpoint-interpolated decoded video to the operation information addition unit 48b.

[0103] When the user requests a change in the viewpoint position by performing an operation based on the operation information displayed on the display unit 49, the ID information detection unit 50 detects an ID (request ID) assigned to the video group corresponding to the new viewpoint position requested by the user. Then, the ID information detection unit 50 outputs the detected request ID to the viewpoint change unit 53.

[0104] When the request ID is input from the ID information detection unit 50, the viewpoint change unit 53 acquires the default decoded video from the storage unit 44 among the decoded videos of the video signal group to which the request ID is assigned, and outputs it to the operation information addition unit 48b.

[0105] During content playback, the operation information addition unit 48b first acquires the default decoded video from the storage unit 44, adds operation information to the decoded video, and outputs it to the display unit 49. After the viewpoint interpolation by the viewpoint interpolation unit 47b, the operation information addition unit 48b acquires the viewpoint-interpolated decoded video from the viewpoint interpolation unit 47b, adds operation information to the decoded video, and outputs it to the display unit 49. After the detection of the request ID by the viewpoint change unit 53, the operation information addition unit 48b acquires the default decoded video with the viewpoint position changed from the viewpoint change unit 53, adds operation information to the decoded video, and outputs it to the display unit 49.

[0106] (Operation sequence) Next, the operation sequence of the video transmission system 3 according to the present embodiment will be described with reference to FIG. 11. FIG. 11 is a diagram showing the operation sequence between the second transmission device 30b and the reception device 40b in the case of transmitting a video signal by IP multicast transmission.

[0107] In step S301, the reception device 40b transmits a viewing request indicating the content selected by the user to the second transmission device 30b by the viewing request transmission unit 41. The second transmission device 30b receives the viewing request transmitted from the reception device 40b by the viewing request reception unit 34.

[0108] In step S302, the second transmission device 30b transmits all video signals related to the content requested for viewing from the reception device 40b to the reception device 40b together with metadata (ID information and position information) by the video transmission unit 37b. The reception device 40b receives all video signals related to the content requested for viewing from the second transmission device 30b together with the metadata by the video reception unit 42.

[0109] In step S303, the reception device 40b decodes the video signal by the video decoding unit 43. Then, the reception device 40b displays the default video of the content requested for viewing by the display unit 49.

[0110] In step S304, the reception device 40b determines whether there is a change in the viewpoint position (small change) without changing the ID by the position determination unit 52. If there is a small change in the viewpoint position, the reception device 40b proceeds to step S305, and if there is no small change in the viewpoint position, the reception device 40b proceeds to step S306.

[0111] In step S305, the receiving device 40b performs viewpoint interpolation processing using a plurality of decoded videos by the viewpoint interpolator 47b, and generates a video corresponding to the viewpoint position of the user. Then, the receiving device 40b displays the viewpoint-interpolated video on the display unit 49. Then, the receiving device 40b returns the process to step S304.

[0112] In step S306, the receiving device 40b determines, by the ID information detector 50, whether there is a change in the viewpoint position (a large change) involving a change in the ID. If there is a large change in the viewpoint position, the receiving device 40b detects the request ID and advances the process to step S307. If there is no large change in the viewpoint position, the receiving device 40b returns the process to step S304.

[0113] In step S307, the receiving device 40b acquires, by the viewpoint changer 53, the default decoded video among the decoded videos of the video signal group to which the request ID is assigned. Then, the receiving device 40b displays the default video on the display unit 49.

[0114] In the case of broadcast transmission, the operation of step S301 is not performed. In step S302, the second transmission device 30b transmits, by the video transmitter 37b, all video signals related to all contents to the receiving device 40b together with metadata (ID information and position information). Then, in step S303, the receiving device 40b displays the default video of the content selected by the user. The operations after step S304 are the same as those in the case of IP multicast transmission.

[0115] As described above, the second transmission device 30b transmits all video signals to the receiving device 40b together with metadata (ID information indicating the grouping of the first layer and the second layer, and position information). Therefore, in the receiving device 40b, when there is a change in the viewpoint position, the decoded video required according to the change in the viewpoint position can be specified. Thus, the present invention can also be applied to the case of broadcasting or IP multicast distribution of video signals.

[0116] <Program> In order to function as the above-described first transmission device 20, second transmission devices 30, 30a, 30b, and reception devices 40, 40a, 40b, it is also possible to use a computer capable of executing program instructions. FIG. 12 is a block diagram showing a schematic configuration of a computer that functions as the first transmission device 20, second transmission devices 30, 30a, 30b, or reception devices 40, 40a, 40b. Here, the computer 100 may be a general-purpose computer, a dedicated computer, a workstation, a PC, an electronic notebook, or the like. The program instructions may be program codes, code segments, etc. for executing necessary tasks.

[0117] As shown in FIG. 12, the computer 100 includes a processor 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, a storage 140, an input unit 150, an output unit 160, and a communication interface (I / F) 170. Each component is connected to be communicable with each other via a bus 180. Specifically, the processor 110 is a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), SoC (System on a Chip), etc., and may be composed of a plurality of processors of the same type or different types.

[0118] The processor 110 controls each component and executes various arithmetic processes. That is, the processor 110 reads a program from the ROM 120 or the storage 140 and executes the program using the RAM 130 as a work area. The processor 110 performs control of each of the above components and various arithmetic processes according to the program stored in the ROM 120 or the storage 140. In the present embodiment, the program according to the present disclosure is stored in the ROM 120 or the storage 140.

[0119] The program may be recorded on a computer-readable recording medium. By using such a recording medium, it is possible to install the program on the computer 100. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, a USB (Universal Serial Bus) memory, etc. Also, this program may be in a form downloaded from an external device via a network.

[0120] The ROM 120 stores various programs and various data. The RAM 130 temporarily stores programs or data as a working area. The storage 140 is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0121] The input unit 150 includes one or more input interfaces that receive a user's input operation and acquire information based on the user's operation. For example, the input unit 150 may be a pointing device, a keyboard, a mouse, etc., but is not limited thereto.

[0122] The output unit 160 includes one or more output interfaces that output information. For example, the output unit 160 may be a display that outputs information as an image, or a speaker that outputs information as sound, but is not limited thereto. Note that when the output unit 160 is a touch panel type display, it also functions as the input unit 150.

[0123] The communication interface 170 is an interface for communicating with an external device.

[0124] Although the above-described embodiments have been described as representative examples, it is obvious to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments, and various modifications and changes are possible without departing from the scope of the claims. For example, it is possible to combine a plurality of the constituent blocks or processing steps described in the embodiments into one, or to divide one into a plurality.

Explanation of Reference Numerals

[0125] 1, 2, 3 Video transmission system 10 Imaging device 20 First transmission device 21 Video reception unit 22 Video encoding unit 23 First ID assignment unit 24 Storage unit 25 Video transmission unit 30, 30a, 30b Second transmission device 31 Video reception unit 32 Second ID assignment unit 33 Storage unit 34 Viewing request reception unit 35, 35a Viewpoint information reception unit 36 Position determination unit 37 Video transmission unit 40, 40a, 40b Receiving device 41 Viewing request transmission unit 42 Video reception unit 43 Video decoding unit 44 Storage unit 45 Viewpoint position detection unit 46, 46a Viewpoint information transmission unit 47, 47b Viewpoint interpolation unit 48, 48b Operation information addition unit 49 Display unit 50 ID information detection unit 52 Position determination unit 53 Viewpoint change unit 100 Computer 110 Processor 120 ROM 130 RAM 140 Storage 150 Input section 160 Output section 170 Communication interface 180 Bus 481 Video 482 - 486 Operation information

Claims

1. A receiving device that receives a plurality of video signals captured from different positions, transmitted by a transmitting device, together with position information of an imaging device that captured the video signals, wherein the video signals are grouped into a plurality of groups according to the shooting positions, a communication interface that simultaneously receives all the video signals to which an ID for identifying the group is assigned from the transmitting device, a control unit that decodes the video signals to generate decoded video and acquires the viewpoint position of the user, and when there is a change in the viewpoint position without a change in the ID, the control unit identifies the position information of the imaging devices that captured a plurality of video signals necessary for generating the video corresponding to the viewpoint position of the user, and performs viewpoint interpolation using the decoded video corresponding to the position information, a receiving device that, when there is a change in the viewpoint position accompanied by a change in the ID, acquires a default decoded video among the decoded videos of the video group corresponding to the new viewpoint position.

2. The receiving device according to claim 1, wherein the first-layer group among the plurality of groups is a group to which a plurality of video signals used for generating video from a viewpoint different from the shooting position in the receiving device belong.

3. The receiving device according to claim 2, wherein the second-layer group among the plurality of groups is one or more of the first-layer groups to which video signals captured within the same location belong.

4. The receiving device according to claim 2 or 3, wherein the video signals belonging to the first-layer group are compressed by predictive coding using the correlation between the video signals.

5. The receiving device according to any one of claims 1 to 4, wherein the ID and the video signals are stored in a file in the ISO Base Media File Format.

Images