Recording and playback of 360-degree videos with object tracking

Abstract

Device comprising the following: a decoding device configured to receive a 360-degree video stream as input and to decodecode the 360-degree video stream; an object tracking device configured to track an object detected in the decoded 360-degree video stream, and providing one or more tracking angles associated with the detected object, the object being detected as being closest to the center point of the current view of the 360-degree video stream; a storage device configured to store the 360-degree video stream and viewing history data associated with the 360-degree video stream; and a rendering device configured to render the decoded stream from the viewing history data using one or more viewing angles.

Description

[0001] The present disclosure relates to the recording and playback of videos, and in particular the recording and playback of 360-degree videos with object tracking, or a device with a decoding device, and a method implemented by means of a computer.

[0002] 360-degree videos, also known as surround videos, full-sphere videos, and / or panoramic videos, are video recordings of a real-world panorama, where the view in every direction is recorded simultaneously using an omnidirectional camera or a collection of cameras. During playback, the viewer controls the angles of the field of view (FOV) and the viewing direction (a form of virtual reality).

[0003] Document US 2010 / 0157016 A1 discloses a multi-view camera system for video conferencing that uses scalable video encoding.

[0004] The publication EP 2 490 179 A1 describes a system for video transmission in which a panoramic image is encoded with low quality, and certain sections thereof with higher quality.

[0005] From publication US 2013 / 0202265 A1, a method is known in which several common objects are identified in several videos, each of the several videos having a different viewing angle of the objects.

[0006] Publication JP 2016-031576 A describes an object detection device in which a moving object is detected using compressed image data.

[0007] The invention aims to provide a novel device comprising a decoding device and a novel computer-implemented method, in particular a novel device and a novel computer-implemented method, in which the tracking of an object can be improved. The invention achieves this and further objectives through the subject matter of the independent claims. Advantageous embodiments of the invention are specified in particular in the dependent claims.

[0008] Conveniently, the 360-degree video stream is encoded with a variety of predetermined viewing angles from the viewing history data.

[0009] The decoding device is also conveniently configured for the following: Extracting the multitude of predetermined viewing angles from the 360-degree video stream. Conveniently, the multitude of predetermined viewing angles is extracted from one or more messages with additional enhancement information within the 360-degree video stream. Advantageously, the rendering device is also configured to receive the decoded 360-degree video stream separately from the extracted predetermined viewing angles. Advantageously, the rendering device is further configured to receive a set of user-selected viewing angles as input, wherein the set of user-selected viewing angles is provided by means of a user input device.

[0010] The rendering device is also conveniently configured for the following: Making a selection between the multitude of predefined viewing angles and the set of user-selected viewing angles to render the decoded 360-degree video stream. The rendering device is also conveniently configured for the following:

[0011] Rendering the decoded 360-degree video stream with one or more viewing angles from the set of user-selected viewing angles, wherein the one or more viewing angles from the set of user-selected viewing angles override corresponding viewing angles from the plurality of predetermined viewing angles.

[0012] The rendering device is also conveniently configured for the following: Re-rendering the decoded 360-degree video stream with the corresponding viewing angles from the multitude of predetermined viewing angles after a predetermined period without user activity. Conveniently, the 360-degree video stream is encoded with a variety of predetermined viewing angles.

[0013] The decoding device is also conveniently configured for the following: Extracting the multitude of predetermined viewing angles from the 360-degree video stream.

[0014] Conveniently, the viewing history data is inserted as user-defined image data into a video sequence of the 360-degree video bitstream.

[0015] It is advantageous for the viewing history data to be a separate data stream within a video sequence of the 360-degree video stream.

[0016] Conveniently, the viewing history data includes a variety of viewing angles for each frame in a video sequence of the 360-degree video stream.

[0017] Advantageously, the rendering device is further configured to receive as input a set of user-selected field-of-view angles, wherein the set of user-selected field-of-view angles is provided by means of a user input device.

[0018] Conveniently, one or more viewing angles correspond to a previous rendering of the decoded 360-degree video stream.

[0019] According to one interpretation, a procedure implemented by means of a computer comprises the following: Decoding a 360-degree video stream; Extracting a variety of predetermined viewing angles from the decoded 360-degree video stream; Rendering the decoded 360-degree video stream using the extracted predetermined viewing angles; and Providing the rendered 360-degree video stream for display.

[0020] The procedure should also expeditely include the following: Receiving a set of user-selected viewing angles as input; Making a choice between the set of user-selected viewing angles and the multitude of predetermined viewing angles; and Rendering one or more video sequences of the decoded 360-degree video stream with one or more viewing angles from the set of user-selected viewing angles, where the set of user-selected viewing angles takes precedence over the multitude of predetermined viewing angles during selection. where one or more video sequences are re-rendered after a predetermined period without user activity, using one or more viewing angles from a multitude of predetermined viewing angles.

[0021] According to one form of appearance, a system comprises the following: a decoding device configured to receive a 360-degree video stream as input and to decodecode the 360-degree video stream; an object tracking device configured to track one or more objects in the decoded 360-degree video stream and to provide one or more tracking angles associated with the one or more objects; and a rendering device configured to render the decoded 360-degree video stream using the one or more tracking angles from the object tracking device to retain at least one of the one or more objects for one or more rendered frames in a rendered view.

[0022] The rendering device is also conveniently configured for the following: Receiving one or more user-selected viewing angles as input; Rendering the decoded 360-degree video stream with one or more user-selected viewing angles; and Reverse rendering of the decoded 360-degree video stream with one or more tracking angles after a predetermined period of inactivity. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Certain features of the claimed technology are set forth in the accompanying claims. However, for illustrative purposes, one or more implementations of the claimed technology are shown in the following figures. Fig. Figure 1 illustrates an exemplary network environment in which the capture and playback of a 360-degree video can be implemented according to one or more implementations. Fig. Figure 2 schematically illustrates an example of an equirectangular projection format. Fig. Figure 3 schematically illustrates an example of an equirectangular projection using a map of the Earth. Fig. Figure 4 schematically illustrates an example of a 360-degree video with equirectangular projection. Fig. Figure 5 schematically illustrates examples of 360-degree images with equirectangular projection layout. Fig. Figure 6 schematically illustrates an exemplary definition of a six-sided cube. Fig. Figure 7 schematically illustrates an example of a cube projection format. Fig. Figure 8 schematically illustrates examples of 360-degree images in the cube projection layout. Fig. Figure 9 schematically illustrates an example of a normalized projection plane size determined using field of view angles. Fig. Figure 10 schematically illustrates an example of viewing direction angles. Fig. Figure 11 illustrates a schematic diagram of a coordinate mapping between an output rendering image and a 360-degree input video image. Fig. Figure 12 schematically illustrates an example of mapping a point in the normalized rendering coordinate system to the normalized projection coordinate system using the equirectangular projection format. Fig. Figure 13 schematically illustrates an example of mapping a point in the normalized rendering coordinate system to the normalized projection coordinate system using the cube projection format. Fig. Figure 14 schematically illustrates an example of a two-dimensional layout 1400 from samples of a 360-degree input video image, which was projected to render a 360-degree video. Fig. Figure 15 illustrates an exemplary network environment in which the recording and playback of a 360-degree video can be implemented according to one or more implementations. Fig. Figure 16 schematically illustrates examples of equirectangular and cube projections. Fig. Figure 17 schematically illustrates an example of a 360-degree video rendering. Fig. Figure 18 illustrates a block diagram of an example of encoding a specific two-dimensional view (2D view) from a 360-degree video stream. Fig. Figure 19 illustrates a block diagram of an example of recording a 360-degree video stream including a specific set of viewing angles. Fig. Figure 20 illustrates a block diagram of an example of playing back a 360-degree video stream with recorded viewing angles. Fig. Figure 21 illustrates a block diagram of an example of extracting viewing angles from a compressed bitstream. Fig. Figure 22 illustrates an exemplary network environment in which object tracking for 360-degree videos may be implemented according to one or more implementations. Fig. Figure 23 schematically illustrates an example of spherical distortion in an equirectangular projection. Fig. Figure 24 schematically illustrates an example of side discontinuities in a cube projection. Fig. Figure 25 illustrates a block diagram of an example of a rendering system for 360-degree videos with object tracking. Fig. Figure 26 schematically illustrates an electronic system with which one or more implementations of the claimed technology can be implemented.

[0024] The accompanying annex, which was included to provide a better understanding of the claimed technology and is incorporated into this patent application and forms a part thereof, illustrates manifestations of the claimed technology and, together with the description, serves to explain the principles of the claimed technology. DETAILED DESCRIPTION

[0025] The description set forth below is intended to represent various configurations of the claimed technology and is not meant to represent the only configurations in which the claimed technology can be implemented in practice. The accompanying drawings are incorporated into this document and form part of the detailed description. The detailed description contains specific details intended to facilitate a better understanding of the claimed technology. However, it is clear and obvious to those skilled in the art that the claimed technology is not limited to the specific details set forth in this document and that it can be implemented in practice using one or more of these configurations.In one or more cases, generally known structures and components are shown in the form of block diagrams to prevent the concepts of the claimed technology from becoming incomprehensible.

[0026] In a 360-degree video capture and playback system, 360-degree videos can be captured, stitched, encoded, decoded, rendered, and played back. In one or more implementations, a decoding device receives a 360-degree video stream as input and decodes it, while a storage device stores the 360-degree video stream and associated viewing history data. A rendering device can render the decoded stream using viewing angles derived from the viewing history data. In one or more implementations, an object tracking device tracks one or more objects in the decoded 360-degree video stream and provides one or more tracking angles associated with those objects.The rendering device can render the decoded 360-degree video stream using one or more tracking angles to maintain at least one object in the 360-degree video stream for one or more rendered frames.

[0027] Fig. Figure 1 illustrates an exemplary network environment 100 in which the capture and playback of 360-degree videos can be implemented according to one or more implementations. Not all components shown may be used; however, one or more implementations may include additional components not shown in the figure. Variations in the arrangement and type of components are possible without derogating from the essence or scope of protection of the claims set forth in this document. Additional components, different components, or fewer components may be provided.

[0028] The exemplary network environment 100 comprises a 360-degree video capture device 102, a 360-degree video stitching device 104, a video encoding device 106, a transmission link or storage media, a video decoding device 108, and a 360-degree video rendering device 110. In one or more implementations, one or more of the devices 102, 104, 106, 108, and 110 may be combined in the same physical device. For example, the 360-degree video capture device 102, the 360-degree video stitching device 104, and the video encoding device 106 may be combined in a single device, and the video decoding device 108 and the 360-degree video rendering device 110 may be combined in a single device.In some forms, the network environment 100 may include a storage device 114 in which the encoded 360-degree video (such as on DVDs, Blu-ray, DVR (Digital Video Recording) in the cloud or in a gateway / set-top box, etc.) is stored and then played back on a display device (for example, 112).

[0029] The network environment 100 may further include a device for converting the projection format for 360-degree videos (not shown), which can perform a conversion of the projection format for 360-degree videos before video encoding by the video encoding device 106 and / or after video decoding by the video decoding device 108. The network environment 100 may also include a device for converting the projection format for 360-degree videos (not shown) that is inserted between the video decoding device 108 and the 360-degree video rendering device 110. In one or more implementations, the video encoding device 106 may be communicatively coupled to the video decoding device 108 via a transmission link, such as a network.

[0030] In the claimed system, the 360-degree video stitching device 104 can use an additional coordinate system that allows more flexibility on the 360-degree video capture side when the captured 360-degree video is projected onto a coordinate system for 2D input images for storage or transmission. The 360-degree video stitching device 104 can also support multiple projection formats for storing, compressing, transmitting, decoding, rendering, etc., 360-degree videos. For example, the video stitching device 104 can remove overlapping areas captured by a camera mount and output, for example, six view sequences, each covering a 90° x 90° viewport.The device for converting the projection format for 360-degree videos (not shown) can convert an input projection format for a 360-degree video (for example, the cube projection format) into an output projection format for a 360-degree video (for example, the equirectangular format).

[0031] The Video Coding Device 106 can minimize spatial discontinuities (for example, the number of side boundaries) in the composite image to achieve better spatial prediction and thus improved compression efficiency during video compression. For cube projection, for example, a preferred layout should have a minimized number of side boundaries, such as four, within a composite 360-degree video image. To achieve better compression efficiency, the Video Coding Device 106 can implement Unrestricted Motion Compensation (UMC).

[0032] In the claimed system, the 360-degree video rendering device 110 can derive a chroma projection image from a luma prediction image. The 360-degree video rendering device 110 can also select the rendering image size to optimize the video's display quality. Furthermore, the 360-degree video rendering device 110 can select the horizontal field of view α along with the vertical field of view β to minimize rendering distortion. The 360-degree video rendering device 110 can also control the field of view to achieve real-time rendering of the 360-degree video within the limits of the available memory bandwidth budget.

[0033] In Fig. 1. The 360-degree video is captured using a camera mount and assembled into the equirectangular format. The video is then compressed into any suitable video compression format (for example, MPEG / ITU-T AVC / H.264, HEVC / H.265, VP9, ​​etc.) and transmitted via the transmission link (for example, cable, satellite, terrestrial transmission, internet streaming, etc.). On the receiving end, the video is decoded (for example, 108) and stored in the equirectangular format, then rendered (for example, 110) and displayed (for example, 112) according to the viewing angles and the field of view. In the claimed system, the end users have control over the field of view and the viewing angles to view the video from the desired angles. Coordinate systems

[0034] There are several coordinate systems that are applied to the claimed technology, including but not limited to the following: • (x, y, z) - 3D coordinate system for 360-degree video capture (camera coordinate system) • (x', y', z') - 3D coordinate system for viewing 360-degree videos • (x p , y p ) - Normalized 2D projection coordinate system with x p ∈ [0,0: 1,0] and y p ∈ [0,0: 1,0]. • (X p , Y p ) - Coordinate system for 2D input images with X p ∈ [0: inputPicWidth - 1] and Y p ∈ [0: inputPicHeight - 1], where inputPicWidth x inputPicHeight is the size of the input images of a color component (for example, Y, U, or V). • (x c , y c ) - Normalized 2D rendering coordinate system with x c ∈ [0,0: 1,0] and y c ∈ [0,0: 1,0]. • (X c , Yc ) - Coordinate system for 2D output rendering images with X c ∈ [0: renderingPicWidth - 1] and Y c ∈ [0: renderingPicHeight - 1], where picWidth x picHeight is the output rendering image size of a color component (for example, Y, U, or V). • (x r , y r , z r ) - 3D coordinate system for 360-degree video projection.

[0035] Fig. Figure 2 schematically illustrates an example of an equirectangular projection format 200. The equirectangular projection format 200 is a standard method for texture mapping a sphere in computer graphics. It is also known as equidistant cylindrical projection, geographic projection, plate map, or plate carrée. As shown in Fig. As shown in Figure 2, to project a point p(x, y, z) on the surface of a sphere (for example, 202) onto a sampling point p'(x) p , y p) in the normalized projection coordinate system (for example 204) both the geographic longitude ω and the geographic latitude φ for p (x, y, z) are calculated according to equation 1. { ω=arc tant2(x,z)φ=arcsin(yx2+y2+z2) , where ω ∈ [-π: π] and φ∈[− π2:π2] This applies. π is the ratio of a circle's circumference to its diameter, usually approximated as 3.1415926.

[0036] The equirectangular projection format 200 can be defined as in equation 2: {xp=ω2π+0,5yp=−φπ+0.5 , where x p ∈ [0,0: 1,0] and y p ∈ [0,0: 1,0] holds. (x p , y p ) is the coordinate in the normalized projection coordinate system.

[0037] Fig. Figure 3 schematically illustrates an example of an equirectangular projection layout 300 using a map of the Earth. In the equirectangular projection layout 300, the image only has a 1:1 mapping along the equator; everywhere else it is stretched. The greatest distortion occurs at the north and south poles of a sphere (for example, 302), where a single point is mapped onto a line of sampling points on the equirectangular projection image (for example, 304), resulting in a large amount of redundant data in the composite 360-degree video using the equirectangular projection layout 300.

[0038] Fig. Figure 4 schematically illustrates an example of a 360-degree video with an equirectangular projection layout (400). To utilize the existing video delivery infrastructure, which employs a single-layer video codec, the 360-degree video footage (for example, 402) captured by multiple cameras at different angles is normally stitched together and assembled into a single video sequence, which is stored in the equirectangular projection layout. As shown in Fig. As shown in Figure 4, in the equirectangular projection layout 400, the left, front, and right video footage of the 360-degree video is projected into the center of the image; the rear video footage is divided equally and placed on the right and left sides of the image; the top and bottom video footage are placed on the top and bottom parts of the image, respectively (for example, 404). All video footage is stretched, with the top and bottom footage being stretched the most. Fig. Figure 5 schematically illustrates examples of 360-degree video images with equirectangular projection layout. Cube projection

[0039] Fig. Figure 6 illustrates an exemplary definition of a six-sided cube 600. Another common projection format for storing the 360-degree view is to project the video footage onto the sides of a cube. As in Fig. Figure 6 shows the six sides of a cube labeled Front, Back, Left, Right, Top and Bottom.

[0040] Fig. Figure 7 schematically illustrates an example of a 700 cube projection format. Fig. 7 includes the cube projection format 700, which maps a surface point p(x, y, z) of a sphere onto one of six cube faces (for example, 702), where both the ID of the cube face and the coordinate (x) p , y p ) in the normalized coordinate system for cube projection (for example, 704).

[0041] Fig. Figure 8 schematically illustrates examples of 360-degree video images with cube projection layout 800. The projection rule for cube projection is described in Table 1, which provides pseudo-code for mapping a surface point p(x,y,z) of a sphere onto a cube face. Field of view and viewing direction angle

[0042] To display a 360-degree video, a portion of each 360-degree video frame must be projected and rendered. The field-of-view angles define how large the displayed portion of a 360-degree video frame is, while the viewing direction angles define which part of the 360-degree video frame is shown.

[0043] To display a 360-degree video, imagine that the video is projected onto the surface of a unit sphere. A viewer sitting at the center of the sphere can view a rectangular screen, and the screen has four corners located on the surface of the sphere. Here, (x', y', z') is called the 360-degree view viewing coordinate system, and (x c , y c ) is the normalized rendering coordinate system.

[0044] Fig. Figure 9 schematically illustrates an example of a normalized projection plane size of 900, determined using field-of-view angles. As in Fig. As shown in Figure 9, in the viewing coordinate system (x',y',z'), the center of the projection plane (i.e., the rectangular screen) lies on the z'-axis and is parallel to the x'y'-plane. Therefore, the projection plane size wxh and its distance to the center of the sphere d can be calculated as follows: {w=2tata2+tb2+1h=2tbta2+tb2+1d=1ta2+tb2+1 , where ta=tan(α2) and tb=tan(β2) and α ∈ (0: π] specify the horizontal field of view angle and β ∈ (0: π] specify the vertical field of view angle.

[0045] Fig. Figure 10 schematically illustrates an example for a viewing direction angle of 1000°. The viewing direction is defined by the rotation angle of the 3D viewing coordinate system (x', y', z') relative to the 3D acquisition coordinate system (x, y, z). As shown in Fig. As shown in Figure 10, the viewing direction is determined by the clockwise rotation angle θ around the y-axis (for example, 1002, yaw angle), the counterclockwise rotation angle γ around the x-axis (for example, 1004, pitch angle), and the counterclockwise rotation angle ∈ around the z-axis (for example, 1006, roll angle).

[0046] The coordinate mapping between the coordinate systems (x,y,z) and (x',y',z') is defined as follows: [xyz]=[cos εsin ε0−sin εcos ε0001][cos θ0sin θ010−sin θ0cos θ][1000cos γsin γ0−sin γcos γ][x'y'z']

[0047] Therefore: [xyz]=[cos ε cos θ−cos ε sin θ sin γ+sin ε cos γcos ε sin θ cos γ+sin ε sin γ−sin ε cos θsin ε sin θ sin γ+cos ε cos γ−sin ε sin θ cos γ+cos ε sin γ−sin θ−cos θ sin γcos θ cos γ][x'y'z']

[0048] Fig. Figure 11 illustrates a schematic diagram of a coordinate mapping 1100 between an output rendering image and an input image. Using the field of view and viewing direction angles defined above, the coordinate mapping between the coordinate system for output rendering images (X) can be determined. c , Y c ) (that is, the rendering image for display) and the coordinate system for input images (X p , Y p ) (that is, the input 360-degree video image). At a given sampling point (X) c , y c ) in the rendering image, the coordinate of the corresponding sampling point (X) can be displayed. p , y p ) in the input image, as in Fig. 11 shown, can be derived by means of the following steps: • Calculating the normalized projection plane size and the distance to the center of the sphere based on the field-of-view angles (α, β) (i.e., Equation 3); calculating the coordinate transformation matrix between the viewing and acquisition coordinate systems based on the viewing direction angles (ε, θ, γ) (i.e., Equation 4). • Normalizing (X c , Y c ) based on the rendering image size and the normalized projection plane size. • Mapping the coordinate (x c , y c ) in the normalized rendering coordinate system to the 3D viewing coordinate system (x', y', z'). • Convert the coordinate to the 3D acquisition coordinate system (x,y,z) • Differentiating the coordinate (x p , y p ) in the normalized projection coordinate system. • Converting the derived coordinate into an integer position in the input image based on the size of the input image and the format for the projection layout.

[0049] Fig. Figure 12 schematically illustrates an example of mapping a point in the normalized rendering coordinate system (for example, p(x)). c , y c )) to the normalized projection coordinate system (for example, p'(x) p , y p )) using the equirectangular projection format 1200.

[0050] In one or more implementations, the projection is performed starting from the equirectangular input format. For example, if the input image is in the equirectangular projection format, the following steps can be applied to determine a sampling point (X). c , Y c ) in the rendering image at a sampling point (X) p , Y p) to be displayed in the input image. • Calculating the projection plane size of a normalized display based on the field of view angles: {w=2tata2+tb2+1h=2tbta2+tb2+1d=1ta2+tb2+1, where the following applies: ta=tan(α2) and tb=tan(β2) • Mapping of (X c , Y c ) to the normalized rendering coordinate system: {xc=XcwrenderingPicWidthyc=YchrenderingPicHeight • Calculating the coordinate of p(x) c , y c ) in the coordinate system (x',y', z') {x'=xc−w2y'=−yc+h2 z'=d • Converting the coordinate (x',y',z') into the acquisition coordinate system (x, y, z) based on the viewing direction angles: [xyz]=[cos ε cos θ−cos ε sin θ sin γ+sin ε cos γcos ε sin θ cos γ+sin ε sin γ−sin ε cos θsin ε sin θ sin γ+cos ε cos γ−sin ε sin θ cos γ+cos ε sin γ−sin θ−cos θ sin γcos θ cos γ][x'y'z'] • Projecting p(x,y,z) onto the normalized projection coordinate system p'(x) p , y p ): { xp=arc tant2(x,z)2π+0.5yp=−arcsin(yx2+y2+z2)π+0.5 • Mapping p'(x p , y p ) onto the (equirectangular) input image coordinate system (X p , y p ) {Xp=(int)(xp∗inputPicWidth)Yp=(int)(yp∗inputPicHeight) , where the following applies: • α, β are field of view angles and e, θ, γ are viewing direction angles. • renderingPicWidth x renderingPicHeight is the rendering image size. • inputPicWidth x inputPicHeight is the size of the input image (in the equirectangular projection format).

[0051] Fig. Figure 13 schematically illustrates an example of mapping a point in the normalized rendering coordinate system (for example, p(x)). c , y c)) to the normalized projection coordinate system (for example, p'(x) p , y p )) using the cube projection format 1300.

[0052] In one or more implementations, the projection is performed starting from the input format for cube projection. For example, if the input image is in cube projection format, the following similar steps can be applied to determine a sampling point (X). c , Y c ) in the rendering image at a sampling point (X) p , Y p ) to be displayed in the input image. • Calculating the projection plane size of a normalized display based on the field of view angles: {w=2tata2+tb2+1h=2tbta2+tb2+1d=1ta2+tb2+1 where the following applies: ta=tan(α2) and tb=tan(β2) • Mapping of (X c , Y c ) to the normalized rendering coordinate system: {xc=xcwrenderingPicWidthyc=YchrendetringPicHeight • Calculating the coordinate of p(x) c , y c ) in the coordinate system (x',y', z') {x'=xc−w2y'=−yc+h2z'=d • Converting the coordinate (x',y',z') into the acquisition coordinate system (x, y, z) based on the viewing direction angles: [xyz]=[cos∈cos θ−cos∈sin θ sin γ+sin∈cos γcos∈sin θ cos γ+sin∈sin γ−sin∈cos θsin∈sin θ sin γ+cos∈cos γ−sin∈sin θ cos γ+cos∈sin γ−sin θ−cos θ sin γcos θ cos γ][x'y'z'] • Projecting p(x,y,z) onto the normalized cube coordinate system p'(x) p , y p ) based on the pseudo-code defined in Table 1. • Mapping p'(x p , y p ) on the input cube coordinate system (X p , Y p ) (assuming that all sides of the cube have an identical resolution) {Xp=(int)(xp∗inputPicWidth3)+Xoffset[faceID]Yp=(int)(yp∗inputPicHeight2)+Yoffset[faceID] , where the following applies: • α, β are field of view angles and ε, θ, γ are viewing direction angles. • renderingPicWidth x renderingPicHeight is the rendering image size. • inputPicWidth x inputPicHeight is the size of the input image (in the cube projection format). • {(Xoffset[facelD],Yof fset[faclD])| faceID (Side ID) = Front, Back, Left, Right, Top and Down} specifies the coordinate offsets of a cube face in the input cube projection coordinate system. {Xoffset[6]={0,inputPicWidth3,2inputPicWidth3,0,inputPicWidth3,2inputPicWidth3}Yoffset[6]={0,0,0,inputPicHeight2,inputPicHeight2,inputPicHeight2}

[0053] For the in Fig. In the 13 illustrated cube projection layout, the faceID uses the following sequence: Front, Back, Left, Right, Top and then Down to access the array of coordinate offsets. Rendering the sampling points for the display

[0054] When projecting 360-degree videos for display, multiple sampling points on an input image for a 360-degree video (for example, in the equirectangular format or in the cube projection format) can be placed at the same integer position (X). c , Y c ) are projected into the rendering image. To achieve smooth rendering, not only the integer pixel positions but also their subpixel positions are projected into the rendering image in order to find corresponding sampling points in the input image.

[0055] Fig. Figure 14 schematically illustrates an example of a two-dimensional layout 1400 from samples of a 360-degree input video image, which was projected to render a 360-degree video. If, as in Fig. Figure 14 shows the accuracy of the projection. 1n Subpixels in the horizontal direction and 1m If the subpixel in the vertical direction is, the sample value of the rendering image at position (X) can be c , Y c ) are rendered as follows: renderingImg[Xc,Yc]=∑i=0m−1∑j=0n−1inputImg[mapping_func(Xc+j+0.5n,Yc+i+0.5m)]+mn2mn , where the following applies: • (X p , Y p ) = mapping_func(X c ,Y c ) is the coordinate mapping function from the rendering image to the input image of the 360-degree video, as defined in the sections above (for example, using equirectangular projection or in the cube projection format). • inputImg[X p, Y p ] is the sample value at position (X p , Y p ) in the input image. • renderingImg [X c , Y c ] is the sample value at position (X c , Y c ) in the output rendering image.

[0056] Instead of the coordinate mapping between the coordinate system for output rendering images (X c , Y c ) and the coordinate system for input images (X p , Y p To avoid having to calculate the coordinate mapping during operation, the coordinate mapping can also be pre-calculated and saved as a projection for the entire rendered image. Since the viewing direction and field of view angles may not change from image to image, the pre-calculated projection can be used for rendering multiple images together.

[0057] Let projectMap[n * X c + j, m * Y c + i] the pre-calculated projection mapping with Xc = 0, 1, ..., renderingPicWidth - 1, Y c = 0,1, ..., renderingPicHeight - 1, j = 0,1, ..., n - 1 and i = 0, 1, ..., m - 1. Each entry of the projection mapping stores the pre-calculated coordinate value (X p , Y p ) of the coordinate system for input images for a subpixel position (Xc+j+0,5n,Yc+i+0,5m) in the rendering image. The rendering can be expressed as follows: renderingImg[Xc,Yc]=∑i=0m−1∑j=0n−1inputImg[projectMap[n∗Xc+j,m∗Yc+i]]+mn2mn

[0058] An image can contain multiple color components, such as YUV, YCbCr, and RGB. The rendering process described above can be applied to individual color components independently. Recording and playback

[0059] Several services, including YouTube and Facebook, have recently begun offering 360° video sequences. These services allow users to view the scene from all angles while the video is playing. Users can rotate the scene to focus on what they find interesting at any given time.

[0060] There are several formats used for 360-degree videos, but each requires some form of projection of a 3D surface (sphere, cube, octahedron, icosahedron, etc.) onto a 2D plane. The 2D projection is then encoded / decoded like any normal video sequence. In the decoder, a portion of this 360° view is rendered and displayed, depending on the user's viewing angle at any given time.

[0061] The end result is that the user is given the freedom to look around as they please, which significantly enhances the feeling of being "immersed" in the scene, making them feel as if they are actually there. Combined with spatial audio effects (rotating the surround sound to match the video), this effect can be quite captivating.

[0062] Fig. Figure 15 illustrates an exemplary network environment 1500 in which the recording and playback of a 360-degree video can be implemented according to one or more implementations. Not all of the components shown may be used; however, one or more implementations may include additional components not shown in the figure. Variations in the arrangement and type of components are possible without derogating from the essence or scope of protection of the claims set forth in this document. Additional components, different components, or fewer components may be provided.

[0063] The exemplary network environment 1500 comprises a 360-degree video capture device 1502, a 360-degree video stitching device 1504, a video encoding device 1506, a video decoding device 1508, and a 360-degree video rendering device 1510. In one or more implementations, one or more of the devices 1502, 1504, 1506, 1508, and 1510 may be combined in the same physical device. For example, the 360-degree video capture device 1502, the 360-degree video joining device 1504 and the video encoding device 1506 can be combined in a single device, and the video decoding device 1508 and the 360-degree video rendering device 1510 can be combined in a single device.In some embodiments, the video decoding device 1508 may include an audio decoding device (not shown), or in other embodiments, the video decoding device 1508 may be communicatively coupled to a separate audio decoding device to process an incoming or stored compressed 360-degree video bitstream.

[0064] On the playback side for 360-degree videos, the network environment 1500 may further include a demultiplexer device (not shown) that can demultiplex the incoming compressed 360-degree video bitstream and provide the demultiplexed bitstream to the video decoding device 1508, the audio decoding device, and a viewing angle extraction device (not shown), respectively. In some manifestations, the demultiplexer device may be configured to decompress the 360-degree video bitstream. The network environment 1500 may further include a device for converting the layout format for 360-degree videos (not shown), which can perform a conversion of the layout format for 360-degree videos before video encoding by means of the video encoding device 1506 and / or after video decoding by means of the video decoding device 1508.The network environment 1500 may also include a 360-degree video playback device (not shown) that plays back the content of the rendered 360-degree video. In one or more implementations, the video encoding device 1506 may be communicatively coupled to the video decoding device 1508 via a transmission link, such as a network.

[0065] The 360-degree video playback device can save the rendering settings for the 360-degree video (for example, the field of view angles, viewing direction angles, rendering image size, etc.) immediately before playback stops or when switching to another program channel. This allows the saved rendering settings to be used when playback resumes on the same channel. The 360-degree video playback device can include a preview mode in which the viewing angles can be automatically changed every N frames to make it easier for viewers to select their preferred viewing direction. The 360-degree video capture and playback device can calculate the projection image during processing (for example, block by block) to save memory bandwidth. In this case, the projection image cannot be loaded from the chip's external memory.In the claimed system, different information regarding fidelity to the reproduction can be assigned to different views.

[0066] In the claimed system, content providers can offer a "suggested view" for a given 360-degree video. The suggested view can be a specific set of viewing angles for each frame of the 360-degree video, designed to provide a recommended experience for the user. If the user is not particularly interested in controlling the view at any given time, they can view (or play back) the suggested view and experience the perspective recommended by the content provider.

[0067] To enable the system to efficiently store data for recording / playing back the decompressed 360-degree video bitstream in a specific view, as originally viewed by a user during one or more specific viewing sessions, the viewing angle data, such as field of view angle and viewing direction angles (yaw, pitch, and roll), can be stored for each frame in the storage device. Combined with the already recorded, complete original 360-degree view data, a previously stored view can be recreated.

[0068] The viewing angle data can be stored in any number of ways. For example, the viewing angle data can be stored as 1) a separate file that must be stored with the video sequence, 2) it can be inserted into the video stream as user image data (for example, AVC / HEVC SEI messages (Supplemental Enhancement Information)), and / or 3) it can be transported as a separate data stream within the video sequence (for example, another MPEG-2 TS-PID or MP4 data stream). Storing the field-of-view angles and the three viewing direction angles (that is, yaw, pitch, and roll) for each frame requires relatively low storage costs and processing overhead to enable the recording / playback of the 360-degree video content in any previously stored view.

[0069] In the claimed system, the recorded viewing angle data and also a specific set of viewing angles of the video rendering device 1510 can be provided for recording a 360-degree video stream. The viewing angle data can be provided by the storage device as recorded viewing angles of the video rendering device 1510 in order to reconstruct a 360-degree video stream in the specified recorded viewing angles.

[0070] Depending on how the view angles were stored, a corresponding process for extracting them can be initiated. In one or more implementations where the recorded view angles are stored within the compressed 360-degree video bitstream, the view angle extraction process can be used. For example, the video decoding device 1508 and / or the audio decoding device can extract view angles from the compressed 360-degree video bitstream (for example, from the SEI messages within an HEVC bitstream). In this respect, the view angles extracted by the video decoding device 1508 can be provided to the video rendering device 1510.If the viewing angles are stored in a separate data stream (for example, MPEG-2 TS-PID), the demultiplexer device can extract this information and send it to the video rendering device 1510 as suggested viewing angles. In some examples, the demultiplexer feeds a separate viewing angle extraction device (not shown) to extract the recorded viewing angles. In this respect, the claimed system would have the ability to switch at any time between the previously recorded view and the manually selected user view.

[0071] In one or more implementations, the 1510 video rendering device can receive a set of user-selected view angles as input. The 1510 video rendering device can choose between the user-selected set of view angles and predetermined view angles from viewing history data. The 1510 video rendering device can render one or more video sequences of the decoded 360-degree video stream using one or more view angles from the user-selected set, provided that the user-selected set takes precedence over the predetermined view angles. In some implementations, after a predetermined period of inactivity, the one or more video sequences are re-rendered using one or more recorded view angles from the predetermined view angles.

[0072] Switching between a pre-recorded view and the manual view may require the user to make a user-specific selection (for example, pressing a control key in the user interface) to enter or exit this mode. In one or more implementations, the system can perform the switching automatically. For example, if the user moves the view manually (with the mouse, a remote control, hand gestures, a headset, etc.), the view is updated to reflect the user's request. If the user has not made any manual adjustments for a certain period of time, the view can revert to the predefined view.

[0073] In one or more implementations, multiple suggested views can be provided where appropriate, and / or more than one suggested view can be rendered simultaneously. For example, in a soccer match, one view might track the goalkeeper, and other views might track the strikers. In the soccer example above, the user can have a split screen with four views running concurrently. Alternatively, different views can be used to track specific cars during a Formula 1 race. The user can select from these suggested views to personalize their experience without having to fully control the view at all times.

[0074] If a suggested view is unavailable or unsuitable for the entire scene, suggestions (or recommendations) can be provided to ensure the viewer doesn't miss any important action. A hint view (or preview) can be provided at the start of a new scene. This view can then be moved to examine the given angle and center the view on the main action. In one or more implementations, if the user wishes to act less directly (or independently), graphical arrows can be used on the screen to indicate that the user might be looking in the wrong direction and missing something interesting.

[0075] Two commonly used types of projection are equirectangular projection and cubic projection. These types of projection map video from a sphere (equirectangular) or a cube (cubic) onto a flat 2D surface. Examples are in Fig. Figure 16 illustrates examples of an equirectangular projection (for example, 1602) and a cube projection (for example, 1604).

[0076] Fig. Figure 17 schematically illustrates an example of a 360-degree video rendering 1700. The video rendering device 1510 can receive a 360-degree video bitstream as input and render multiple areas of the 360-degree video with respective direction angles on the 2D plane.

[0077] Currently, most 360-degree video content from streaming media services (YouTube, Facebook, Hulu, etc.) is viewed on computers or smartphones. However, it is expected that 360-degree videos will be available for broadcast over standard cable / satellite networks in the near future. Sporting events, travelogues, extreme sports, movies, and many other types of programming can be presented as 360-degree videos to capture and engage users.

[0078] In any type of transmission DVR (Digital Video Recording) environment, the transmitted programs are recorded for later playback. With 360-degree videos in particular, it can be fun to watch a scene multiple times to experience it from a different angle or to observe something else interesting within the scene.

[0079] For 360-degree videos, the concept of "recording" can be defined as follows: 1) Recording the full view of the 360-degree video for later viewing (and selecting a different view next time) (hereinafter referred to as "Definition (1)"); 2) Recording a specific view as viewed by a user during one or more specific viewing sessions (hereinafter referred to as "Definition (2)"); and 3) Recording both the full view of the 360-degree video and a specific view as viewed by a user (hereinafter referred to as "Definition (3)"). Recording based on Definition (3) may include the functionality of recording based on Definitions (1) and (2).In some implementations, recording can be defined as recording both a specific view viewed by a user and another view that is tracked in a background process using an object tracking engine.

[0080] When recording according to definition (1), the audio / video can be stored in a storage device (for example, volatile memory or non-volatile memory) and played back in the same way as any normal program. In some manifestations, the 360-degree video can, for example, be converted from one projection format to another (for example, from the equirectangular format to the cube format) if a display device (for example, 1512) prefers one of these formats over another. In this example, the recording and playback do not differ from any regular audio / video stream.

[0081] Recording based on definition (2) can allow the user to record their experience when first viewing the video content and then recreate the exact same experience on a subsequent viewing. This can be helpful if the user was satisfied with their viewing angles when previously viewing a scene and wants to revisit the scene without additional effort or intervention.

[0082] Fig. Figure 18 illustrates a block diagram (1800) of an example for encoding a specific 2D view from a 360-degree video stream. A brute-force approach to capturing a previous view would be to re-encode a standard 2D video as it was viewed by the user. This would involve capturing the 2D image output from the 360-degree video rendering process and running it through a video encoder. The audio would also need to be re-encoded to obtain the correct left / right / center / back audio mix, corresponding to the video's direction. After this process, the video can be viewed on any client without requiring a 360-degree video rendering process. This process is described in Fig. 18 illustrated.

[0083] In Fig. A compressed bitstream (for example, a 360-degree video stream) is received as input at the video decoding device 1508 and decoded into a decoded stream. The video rendering device 1510 receives the decoded video stream as input and renders the received stream. The video rendering device 1510 can receive user input that includes one or more user-defined viewing angles. The rendered stream is re-encoded using audio and video components via an audio encoder 1802 and a video encoder 1804, respectively. Their respective output signals are then fed into a multiplexer 1806 so that they are combined into a compressed 2D bitstream and stored in a storage device 1808.

[0084] However, the processing for re-encoding is expensive in terms of hardware, and if the user desires the flexibility to retain the full 360-degree version, the generated 2D view would require additional storage space. Instead, the claimed technology describes a methodology for achieving recording and playback based on definitions (1), (2), and (3) without requiring significant storage space or processing overhead.

[0085] With further reference to Fig. 10. The concept of recording a user's view at any given time can be expressed as a field of view angle, and three distinct viewing directions are defined by rotation angles of the 3D viewing coordinate system (x', y', z') relative to the 3D capture coordinate system (x, y, z). As in Fig. As shown in Figure 10, the viewing direction is determined by the clockwise rotation angle θ around the y-axis (for example, 1002, yaw angle), the counterclockwise rotation angle γ around the x-axis (for example, 1004, pitch angle), and the counterclockwise rotation angle ∈ around the z-axis (for example, 1006, roll angle).

[0086] As a means of effectively storing the data required for recording / playback in accordance with definitions (2) and (3) above, the field of view angles and yaw angle, pitch angle and roll angle can be stored for each frame. In combination with the data already recorded for definition (1) (the complete original 360-degree video data), any previously stored view can be recreated.

[0087] Fig. Figure 19 illustrates a block diagram (1900) of an example for recording a 360-degree video stream, including a specific set of viewing angles. Fig. 19. A compressed bitstream (for example, the 360-degree video stream) is received as input at the video decoding device 1508 and decoded into a decoded stream. The video rendering device 1510 receives the decoded video stream as input and renders the decoded stream for display. The video rendering device 1510 can receive user input in the form of viewing history data, which includes one or more user-defined viewing angles. In some manifestations, the user input also includes one or more user-selected field-of-view angles. Both the compressed bitstream and the received user input are stored in the storage device 1808, so that a playback sequence can be reconstructed using a specific user-defined viewing angle sequence.In some cases, the compressed bitstream may include one or more predetermined viewing angles. In this respect, the 1510 video rendering device can select between the predetermined viewing angles and the user-selected viewing angles when rendering the decoded video stream.

[0088] Fig. Figure 20 illustrates a Block Diagram 2000 of an example for playing back a 360-degree video stream with recorded viewing angles. In one or more implementations, the field-of-view angles and the yaw, pitch, and roll angles (the "viewing angle data") for each frame are stored in the 1808 storage device. This viewing angle data can be stored in any number of ways. For example, the viewing angle data can be stored as a separate file along with a video sequence of the compressed bitstream, inserted into the compressed bitstream as user image data (for example, an AVC / HEVC SEI message (Supplemental Enhancement Information)), or it can be carried as a separate data stream within the video sequence (for example, another MPEG-2 TS-PID or MP4 data stream).

[0089] Assuming the user wants to save the full 360-degree version for a later playback session, saving at least one viewing angle per frame may require relatively little effort to activate this feature.

[0090] In Fig. 20. A compressed bitstream (for example, the 360-degree video stream) is received as input from the storage device 1808 by the video decoding device 1508 and decoded into a decoded stream. The video rendering device 1510 receives the decoded video stream as input and renders the decoded stream for display using viewing angles recorded by the storage device 1808. Fig. 20. The 360-degree video stream can be encoded with multiple, predetermined viewing angles defined by a content provider. In another example, the recorded viewing angles can be user-selected viewing angles from a previous rendering session of the 360-degree video stream.

[0091] Fig. Figure 21 illustrates a block diagram (2100) of an example for extracting view angles from a compressed bitstream. In applications where the view angles are stored within the compressed bitstream itself, a process for extracting the view angles can be used. For example, the video decoder can extract view angles from the SEI messages within an HEVC bitstream.

[0092] In Fig. 21. A compressed bitstream (for example, the 360-degree video stream) is received as input from the storage device 1808 by the video decoding device 1508 and decoded into a decoded stream. The video decoding device 1508 can extract predetermined viewing angles, including predetermined field-of-view angles, from the decoded video stream. In some examples, the video decoding device 1508 can extract the predetermined viewing angles from one or more messages containing additional enhancement information within the decoded video stream. The video rendering device 1510 receives the decoded video stream along with the extracted viewing angles as input and renders the decoded stream for display using the extracted viewing angles.In this example, the video rendering device 1510 receives the decoded video stream separately from the extracted predetermined viewing angles via respective channels.

[0093] This approach allows the user to relive a previous experience, but since the full 360-degree video is also available, if the user wishes to deviate from the predetermined (or suggested) viewing angle and manually change the viewing angle, they can do so seamlessly at any point in the playback sequence. The user's manual input can override the recorded values ​​(or corresponding predetermined viewing angles) and control the view, regardless of the previously chosen perspective.

[0094] In one or more implementations, if a user initiates control of the rendered view but becomes bored after a while, the system can revert to the previous view angle setting once it detects that the user has ceased manual control for a predetermined period. In other implementations, the user can select a control key to return to a playback mode where the recorded view is re-rendered once the user has finished manually controlling the view angle.

[0095] In most 360-degree viewing applications, the user is not permitted to adjust the roll angle. The camera is typically fixed in a vertical orientation. The view can be rotated up / down and left / right, but it cannot be panned sideways. In this respect, the claimed technology may apply to an application where a system has recorded two of the three possible viewing angles.

[0096] It should be noted that not all 360-degree video streams cover a complete 360° x 180° field of view. Some sequences may restrict viewing to the "front" direction (180° x 180°). Some of these may also have limitations regarding viewing height (how high or low the user can see). In this respect, the claimed technology may be applicable to any of these use cases.

[0097] In one or more implementations, a 360-degree video playback system has the ability to "remember" the rendering settings for the 360-degree video (for example, the field of view angles, viewing direction angles, rendering image size, etc.) immediately before playback stops or the program channel switches to another, so that the remembered rendering settings can be used when playback resumes on the same channel. Object tracking

[0098] While a 360-degree video application can be a rewarding immersive experience, it can often become tiring during longer programs if it requires constantly manually controlling the view to track the most important objects. For example, it might be interesting to occasionally look around during a sporting event, but after a while, the user will want to focus their view solely on the ball, a specific player, a particular race car, and so on. Object tracking can be used for this purpose, tracking relevant objects and keeping them centered in the frame.

[0099] Object tracking has been a subject of research for many years. Applications such as video surveillance, video compression, and medical imaging all attempt to track objects as they move from frame to frame. The technology described can utilize standard object tracking algorithms that recognize the limitations of 360-degree video formats.

[0100] Fig. Figure 22 illustrates an exemplary network environment 2200 in which object tracking for 360-degree videos can be implemented according to one or more implementations. Not all components shown may be used; however, one or more implementations may include additional components not shown in the figure. Variations in the arrangement and type of components are possible without derogating from the essence or scope of protection of the claims set forth in this document. Additional components, different components, or fewer components may be provided.

[0101] The exemplary network environment 2200 comprises a 360-degree video capture device 2202, a 360-degree video stitching device 2204, a video encoding device 2206, a video decoding device 2208, an object tracking device 2214, and a 360-degree video rendering device 2210. In one or more implementations, one or more of the devices 2202, 2204, 2206, 2208, 2210, 2214 may be combined in the same physical device. For example, the 360-degree video capture device 2202, the 360-degree video stitching device 2204 and the video encoding device 2206 can be combined in a single device, and the video decoding device 2208, the object tracking device 2214 and the 360-degree video rendering device 2210 can be combined in a single device.In some embodiments, the video decoding device 2208 may include an audio decoding device (not shown), or in other embodiments, the video decoding device 2208 may be communicatively coupled to a separate audio decoding device to process an incoming or stored compressed 360-degree video bitstream.

[0102] On the playback side for 360-degree videos, the network environment 2200 may further include a demultiplexer device (not shown) that can demultiplex the incoming compressed 360-degree video bitstream and provide the demultiplexed bitstream to the video decoding device 2208 and the audio decoding device, respectively. In some manifestations, the demultiplexer device may be configured to decompress the 360-degree video bitstream. An object tracking device 2214 may be coupled to the video decoding device 2208 and the audio decoding device for tracking objects in the 3D space of the decoded 360-degree video stream. In this respect, the object tracking device may provide tracking angles to the video rendering device 2210 for rendering the 360-degree video content using object tracking.The network environment 2200 may further include a 360-degree video layout format conversion device (not shown), which can perform a 360-degree video layout format conversion before video encoding by the video encoding device 2206 and / or after video decoding by the video decoding device 2208. The network environment 2200 may also include a 360-degree video playback device (not shown) that plays back the content of the rendered 360-degree video. In one or more implementations, the video encoding device 2206 may be communicatively coupled to the video decoding device 2208 via a transmission link, such as a network.

[0103] In the claimed system, object tracking can be used to track relevant objects and keep these objects centered in a single frame of a 360-degree video. The field of view angles and the yaw, pitch, and roll angles (viewing angle data) can be stored in the storage device for each frame. In some 360-degree viewing applications, the user may not be able to adjust the roll angle. The camera is typically fixed in a vertical orientation. The angle can be rotated up / down and left / right, but it cannot be tilted laterally. Therefore, to center the view on the relevant object, the yaw and pitch angles can be adjusted so that the viewing angle is aligned with the vector pointing to the center point of the relevant object.

[0104] Switching between tracking view and manual view may require the user to make a user-defined selection (for example, pressing a button) to enter or exit this mode. The object detected closest to the center of the current view can be tracked. Alternatively, the system can use an input device (for example, mouse, remote control, or voice command) to lock onto the object selected by the user in the scene.

[0105] Alternatively, the system can automatically switch between "manual" and "tracking" modes. For example, if the user manually moves the view (with the mouse, a remote control, hand gestures, a headset, etc.), the view updates to follow the user's request. If the user hasn't made any manual adjustments for a certain period of time, the view can begin tracking the object that is closest to the center of the scene at that time.

[0106] Thousands of algorithms have been proposed for object tracking, with varying degrees of success (for example, point tracking, kernel tracking, boundary tracking). All of them track features such as color, edges, optical flows, textures, and so on. These object tracking algorithms each have their strengths and weaknesses, and they can differ significantly in their computational requirements.

[0107] The disclosed system does not attempt to suggest that one algorithm is better suited for object tracking than another. This depends to a large extent on the type of objects to be tracked and on the available computing power of the object tracking device 2214 (for example, the set-top box).

[0108] Object detection in 360-degree videos is not fundamentally different from object detection in 2D videos. Object detection would typically be performed using the 2D projection data (equirectangular projection, cube projection, etc.) rather than attempting to track objects in 3D space.

[0109] Fig. Figure 23 schematically illustrates an example of spherical distortion in an equirectangular projection. In the equirectangular projection, the 2D projection is a single, continuous space. This 2D projection can allow the implementation of conventional tracking algorithms. However, the curvature caused by the sphere, especially near the poles, can cause problems for some tracking algorithms. For example, objects may be rotated and distorted as they approach the poles, and linear motion may become more curved and circular.

[0110] Fig. Figure 24 schematically illustrates an example of side discontinuities in a space of cube projection 2400. For a tracking algorithm to function in the space of cube projection 2400, it must recognize side boundaries and understand how each side is connected to its neighboring sides. As an object moves from one side to the next, the tracking algorithm must, if necessary, understand its side boundaries and continuously track this movement.

[0111] Once the object is tracked in the 2D projection space, it can be mapped back onto a point / vector in 3D space. This can be done using the same inverse projection algorithms that are used to map the 2D projection onto the 3D surface.

[0112] Additionally, surround sound can potentially provide extra tracking data that helps improve object tracking accuracy. Surround sound can provide audio signals known to originate from the right, down, left, front, top, and back directions. These directional audio signals can be the same type of information provided by a microphone array. Such arrays can be used to delineate and detect sound sources in 3D space. If the tracked object is assumed to be a sound source, then tracking the sound source through space and time can improve the accuracy of the tracking algorithm.

[0113] With reference to Fig. 10. In many 360-degree viewing applications, the user cannot adjust the roll angle. The camera is typically fixed in a vertical orientation. It can be rotated up / down and left / right, but it cannot be tilted sideways. Therefore, to center the view on the relevant object, the yaw and pitch angles can be adjusted so that the viewing angle is aligned with the vector pointing to the center point of the relevant object.

[0114] Fig. Figure 25 illustrates a block diagram (2500) of an example of a rendering system for 360-degree videos with object tracking. Fig. In device 25, a compressed bitstream (for example, the 360-degree video stream) is received as input by a demultiplexer 2502 and divided into separate data streams (for example, video stream, audio stream) by demultiplexing. In this example, the demultiplexed bitstream can be received by the respective decoders on separate channels. For example, the video stream portion of the compressed bitstream can be received and decoded by a video decoder 2208-1, and the audio stream portion can be received and decoded by an audio decoder 2208-2. An object tracking device 2214 can receive both the video and audio streams from decoders 2208-1 and 2208-2, respectively, and track one or more objects in the decoded 360-degree video stream.The object tracking device 2214 can provide one or more tracking angles associated with the video rendering device 2210. In some manifestations, the object tracking device 2214 can detect objects that are closest to the center point of a current view. In other manifestations, the object tracking device 2214 can lock onto an object based on the user's selection of the object displayed in the scene via an input device (for example, a mouse).

[0115] The video rendering device 2210 receives the decoded video stream and the decoded audio stream as input and renders the decoded stream for display. The video rendering device 2210 also receives the tracking angles from the object tracking device 2214 as input. The video rendering device 2210 also receives user input, which includes one or more viewing angles defined by the user via a user input device (for example, mouse, remote control, etc.). In some cases, the user input also includes one or more field-of-view angles selected by the user. In this respect, the video rendering device 2210 can select between the tracking angles and the user-selected viewing angles when rendering the decoded video stream.The object tracking device 2214 can accept various forms of user input to initiate the tracking of objects in a scene, including but not limited to user eye movement data, user head movement data, speech recognition data, and the like.

[0116] In one or more implementations, the video rendering device 2210 can receive a set of user-selected view angles as input. The video rendering device 2210 can select between the set of user-selected view angles and tracking angles from the object tracking device 2214. The video rendering device 2210 can render one or more video sequences of the decoded 360-degree video stream using one or more view angles from the set of user-selected view angles, provided that the user-selected view angles take precedence over the tracking angles during selection. In some manifestations, the one or more video sequences are re-rendered using one or more suggested tracking angles after a predetermined period of inactivity.

[0117] Switching between tracking angles and user-selected view angles can be triggered by a control command in the user interface that toggles between the display modes of the rendered streams. In one or more implementations, switching between "manual" and "tracking" can be automatic (i.e., without user input). For example, if the user manually moves the view (with the mouse, a remote control, hand gestures, a headset, etc.), the view updates to follow the user's request. If the user hasn't made any manual adjustments for a certain period of time, the view can begin tracking any object that is closest to the center of the scene at that time.

[0118] Using split screens on a display device (for example, 2212), more than one suggested view can be tracked simultaneously. For example, a user can have a split screen with four (4) different views displayed simultaneously, where one predetermined view tracks a first object (for example, the goalkeeper of a soccer team), another predetermined view tracks one or more second object(s) (for example, the forwards of the soccer team), while another view is manually controlled by the user via the viewing angles selected by the user.

[0119] It should be noted that not all 360-degree video streams actually cover the full 360° x 180° field of view. Some sequences may restrict the viewing direction to the "front" direction (180° x 180°). Some of these may also have limitations regarding viewing height (how high or low the user can change the viewing direction). In this respect, the 2214 object tracking device can accommodate the viewing direction limitations imposed by some applications.

[0120] Fig. Figure 26 schematically illustrates an electronic system 2600 with which one or more implementations of the claimed technology can be implemented. For example, the electronic system 2600 can be a network device, a media converter, a desktop computer, a laptop computer, a tablet computer, a server, a switch, a router, a base station, a receiver, a telephone, or generally any electronic device that transmits signals over a network. Such an electronic system 2600 has various types of computer-readable media and interfaces to various other types of computer-readable media.In one or more implementations, the electronic system 2600 may be or include one of the devices 102, 104, 106, 108, 110, the 360-degree video layout format conversion device, and / or the 360-degree video playback device. The electronic system 2600 comprises a bus 2608, one or more processing units 2612, a system memory 2604, a read-only memory (ROM) 2610, a permanent storage device 2602, an input device interface 2614, an output device interface 2606, and a network interface 2616, or subsets and variations thereof.

[0121] The bus 2608 encompasses all system buses, peripheral buses, and chipset buses that communicatively connect the numerous internal devices of the electronic system 2600. In one or more implementations, the bus 2608 communicatively connects the one or more processing unit(s) 2612 to the ROM 2610, the system memory 2604, and the permanent storage device 2602. From these various storage units, the one or more processing unit(s) 2612 retrieve instructions to be executed and data to be processed in order to carry out the processes of the claimed disclosure. The one or more processing unit(s) 2612 can be a single processor or a multi-core processor, depending on the implementation.

[0122] The ROM 2610 stores static data and instructions required by the one or more processing unit(s) 2612 and other modules of the electronic system. The permanent storage device 2602, on the other hand, is a read / write storage device. The permanent storage device 2602 is a non-volatile storage unit in which instructions and data are stored even when the electronic system 2600 is switched off. In one or more implementations of the claimed disclosure, a mass storage device (such as a magnetic disk or an optical disk and the corresponding disk drive) can be used as the permanent storage device 2602.

[0123] In other implementations, a removable storage device (such as a floppy disk, a flash drive, and the corresponding disk drive) is used as the permanent storage device 2602. Like the permanent storage device 2602, the system memory 2604 is a read / write storage device. However, unlike the permanent storage device 2602, the system memory 2604 is a volatile read / write memory, such as random access memory. Any instructions and data required by the one or more processing units 2612 at runtime are stored in the system memory 2604. In one or more implementations, the processes of the claimed disclosure are stored in the system memory 2604, in the permanent storage device 2602, and / or in the ROM 2610.From these various storage units, the one or more processing unit(s) retrieve 2612 instructions to be executed and data to be processed in order to execute the processes of one or more implementations.

[0124] Bus 2608 also provides a connection to the 2614 input device interface and the 2606 output device interface. The 2614 input device interface allows a user to transmit information and select commands to the electronic system. Input devices used with the 2614 input device interface include, for example, alphanumeric keyboards and pointing devices (also known as cursor control devices). The 2606 output device interface allows, for example, the display of images generated by the 2600 electronic system.Output devices used with the 2606 Output Device Interface include, for example, printers and display devices such as an LCD (Liquid Crystal Display), an LED (Light Emitting Diode), an OLED (Organic Light Emitting Diode), a flexible display, a flat panel display, a solid-state display, a projector, or any other device for outputting information. One or more implementations may include devices that function as both input and output devices, such as a touchscreen.In these implementations, the feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including auditory, speech, or tactile input.

[0125] Finally, as in Fig.Figure 26 shows that the bus 2608 also connects the electronic system 2600 to one or more networks (not shown) via one or more network interfaces 2616. In this way, the computer can be part of one or more networks of computers (such as a local area network, a wide area network, an intranet, or a network consisting of networks, such as the Internet). Any or all components of the electronic system 2600 can be used in connection with the claimed disclosure.

[0126] Implementations within the scope of protection of this disclosure may be carried out in whole or in part using a physical, computer-readable storage medium (or several physical, computer-readable storage media of one or more types) which encode one or more instructions. The physical, computer-readable storage medium may also be persistent by its nature.

[0127] The computer-readable storage medium can be any storage medium that can be read and written to, or otherwise accessed by a general-purpose or specialized computer device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, the computer-readable medium can include, without limitation, any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. The computer-readable medium can also include any non-volatile semiconductor memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, Flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, Racetrack memory, FJG, and Millipede memory.

[0128] Furthermore, the computer-readable storage medium can comprise any non-semiconductor storage medium, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In some implementations, the physical, computer-readable storage medium can be directly coupled to a computer device, while in other implementations, the physical, computer-readable storage medium can be indirectly coupled to a computer device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0129] Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or as instructions in a higher-level language that can be compiled to produce executable or non-executable machine code. Furthermore, instructions can be implemented as data or include data. Computer-executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, applets, functions, and so on. As experts in this field recognize, details, including but not limited to the number, structure, sequence, and organization of instructions, can vary considerably without altering the underlying logic, function, processing, and output.

[0130] While the above discussion mainly concerns microprocessors or multi-core processors that execute software, one or more implementations are executed by means of one or more integrated circuits, such as ASICs (Application Specific Integrated Circuits) or FPGAs (Field Programmable Gate Arrays). In one or more implementations, such integrated circuits execute instructions that are stored within the circuit itself.

[0131] Experts in this field would recognize that the various blocks, modules, elements, components, procedures, and algorithms described in this document for illustrative purposes can be implemented as electronic hardware, computer software, or a combination of both. To illustrate this interchangeability of hardware and software, various blocks, modules, elements, components, procedures, and algorithms used for illustrative purposes have been described above in general terms with regard to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints to which the overall system is subject. Experts in this field can implement the described functionality in different ways for each specific application.Different components and blocks may be arranged in other ways (for example, arranged in a different order or divided in other ways) without deviating from the scope of protection of the claimed technology.

[0132] It is understood that any specific order or hierarchy of blocks in the disclosed processes is merely an illustration of exemplary approaches. It is understood that the specific order or hierarchy of blocks in the processes can be rearranged based on design preferences, or that all illustrated blocks can be executed. Any of the blocks can be executed concurrently. In one or more implementations, multitasking and parallel processing may be advantageous. Furthermore, the separation of different system components in the embodiments described above should not be interpreted as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0133] As used in this patent specification and in any claims of this patent application, the terms "base station", "receiver", "computer", "server", "processor", and "memory" all refer to electronic or other technological devices. These terms do not refer to any person or group of persons. For the purposes of this patent specification, the terms "display" and "display" mean displaying on an electronic device.

[0134] In one or more implementations, the statement that a processor is configured to monitor and control an operation or component can also mean that the processor is programmed to monitor and control the operation, or that the processor is operational in such a way as to monitor and control the operation. Similarly, the statement that a processor is configured to execute code can be interpreted as meaning that a processor is programmed to execute code, or that it is operational in such a way as to execute code.

Claims

[1] Device comprising the following: a decoding device configured to receive a 360-degree video stream as input and to decodecode the 360-degree video stream; an object tracking device configured to track an object detected in the decoded 360-degree video stream, and providing one or more tracking angles associated with the detected object, the object being detected as being closest to the center point of the current view of the 360-degree video stream; a storage device configured to store the 360-degree video stream and viewing history data associated with the 360-degree video stream; and a rendering device configured to render the decoded stream from the viewing history data using one or more viewing angles. [2] Device according to claim 1, wherein the 360-degree video stream is encoded with a plurality of predetermined viewing angles from the viewing history data. [3] Device according to claim 2, wherein the decoding device is further configured for the following: Extracting the multitude of predetermined viewing angles from the 360-degree video stream. [4] Device according to claim 3, wherein the plurality of predetermined viewing angles are extracted from one or more messages with additional enhancement information within the 360-degree video stream. [5] Device according to claim 4, wherein the rendering device is further configured to receive the decoded 360-degree video stream separately from the extracted predetermined viewing angles. [6] - Device according to claim 2, wherein the rendering device is further configured to receive a set of user-selected viewing angles as input, wherein the set of user-selected viewing angles is provided by means of a user input device. [7] Device according to claim 6, wherein the rendering device is further configured for the following: Making a selection between the multitude of predefined viewing angles and the set of user-selected viewing angles to render the decoded 360-degree video stream. [8] Device according to claim 6, wherein the rendering device is further configured for the following: Rendering the decoded 360-degree video stream with one or more viewing angles from the set of user-selected viewing angles, wherein the one or more viewing angles from the set of user-selected viewing angles override corresponding viewing angles from the plurality of predetermined viewing angles. [9] A computer-implemented method comprising the following: Decoding a 360-degree video stream; Tracking an object detected in the decoded 360-degree video stream and providing one or more tracking angles associated with the detected object, where the object is detected as being closest to the center point of the current view of the 360-degree video stream; Extracting a variety of predetermined viewing angles from the decoded 360-degree video stream; Rendering the decoded 360-degree video stream using the extracted predetermined viewing angles; and Providing the rendered 360-degree video stream for display.

Images