A method of video streaming using cartesian or spherical coordinates to indicate a point of interest

Abstract

A method of streaming video content, the video content comprising a captured image of a field of view of a camera, the method comprising determining Cartesian co-ordinates of a point of interest in the field of view in a Cartesian co-ordinate system of the camera, converting the determined Cartesian co-ordinates from the Cartesian co-ordinate system to a spherical co-ordinate system of the camera to produce spherical co-ordinates of the point of interest, the spherical co-ordinates being suitable for use in orientating the captured image for display on a display device.

Description

[0001] A METHOD OF VIDEO STREAMING USING CARTESIAN OR SPHERICAL COORDINATES TO INDICATE A POINT OF INTEREST

[0002] Video streaming can be used to provide coverage of events, such as sporting events entertainment events etc. Providing high-quality coverage of such events can be difficult. For example, if a single camera is used, that camera needs to maintain focus and zoom level, on a point of interest as the event proceeds.

[0003] For larger scale events multiple cameras may be required. This creates further problems. Take the example of a football match. As the ball moves around the pitch, the coverage needs to switch between the cameras in order to provide the viewer with the best possible view of the action. This means that focus and zoom level should be maintained on the point of interest, before and after a switch of cameras.

[0004] If this cannot be achieved the viewer may need to manually re-orientate the image on their device in order to maintain a good view. Requiring the viewer to do this is undesirable as it breaks the directorial narrative and inevitably causes some of the live action to be missed.

[0005] It would be desirable to provide a streaming method in which the above-mentioned and / or other disadvantages of the prior art are addressed.

[0006] According to an aspect of the invention there is provided a method of streaming video content, the video content comprising a captured image of a field of view of a camera, the method comprising:

[0007] Determining Cartesian co-ordinates of a point of interest in the field of view in a Cartesian co-ordinate system of the camera;

[0008] Converting the determined Cartesian co-ordinates from the Cartesian co-ordinate system to a spherical co-ordinate system of the camera to produce spherical coordinates of the point of interest;

[0009] The spherical co-ordinates being suitable for use in orientating the captured image for display on a display device.

[0010] The method may further comprise transmitting the spherical co-ordinates for use in displaying the image at a display device. The present invention enables a method to be performed in which the point of interest is continuously maintained in the centre of the image presented to the user, even after the coverage switches from one camera to another. In an embodiment, a display device can receive the spherical co-ordinates and use them to orientate the image so that the point of interest is always located in the centre of the image. Thus seamless ‘action tracked’ switching in 360-degree multi-camera productions is provided. This is particularly beneficial when done immediately after a switch of camera, as the user experience is significantly enhanced.

[0011] The co-ordinates in the spherical co-ordinate system may be:

[0012] (i) an angle between a datum point of the camera in a vertical plane and the point of interest; and

[0013] (ii) an angle between a datum point of the camera in a horizontal plane and the point of interest.

[0014] The camera may be a wide angled-camera and may be a 180 degree camera and may be a 360 degree camera. The method may further comprise transmitting the image for display at the display device.

[0015] The method may be continuously repeated. The method may be repeated with a frequency of at least 50Hz. The method may further comprise capturing the captured image at the 360-degree camera. The method may be performed at a production unit.

[0016] The method may further comprise receiving the transmitted image and / or the transmitted spherical co-ordinates on the cloud, which may be a GPU server cluster.

[0017] The method may further comprise determining if the point of interest has entered a jurisdiction of a second camera. If it is determined that the point of interest has entered the jurisdiction of the second camera, the second camera may replace the first camera as the active camera. The method may be repeated, the 360-degree camera being the second camera.

[0018] The method may further comprise receiving the transmitted image at a display device and may further comprise receiving the transmitted spherical co-ordinates at the display device. The method may further comprise presenting the captured image on the display device. The method may further comprise orientating the captured image by field of view angles corresponding to the transmitted spherical co-ordinates. This may cause the point of interest to be located in the centre of a display of the display device.

[0019] If a manual re-orientation has been made to an image presented on the display device the method may further comprise subtracting an angle by which the image has been reoriented from the field of view angles prior to the step of orientating the captured image by field of view angles is performed.

[0020] In some embodiments the step of converting the determined co-ordinates from the Cartesian co-ordinate system of the camera to a spherical co-ordinate system of the camera is performed at the production unit. These embodiments may further comprise transmitting the spherical co-ordinates for receipt by the display device for use in presenting the transmitted content. The image and the spherical co-ordinates may be transmitted in the same data stream.

[0021] In other embodiments the step of converting the determined co-ordinates from the Cartesian co-ordinate system of the camera to a spherical co-ordinate system of the camera is performed at the display device. These embodiments further comprise the step of transmitting the co-ordinates of a point of interest in the field of view in a Cartesian co-ordinate system of the camera.

[0022] According to a further aspect of the invention there is provided a method of video streaming comprising:

[0023] Receiving a captured image of a field of view of a camera; receiving spherical co-ordinates in a spherical co-ordinate system of a point of interest in the field of view of the camera;

[0024] Re-orientating the image by field of view angles corresponding to the spherical coordinates to produce a re-oriented image.

[0025] The method may further comprise displaying the re-orientated image on a display. The method may further comprise transmitting the re-orientated image to a display device for display. According to a further aspect of the invention there is provided a video streaming system for streaming video content, the video content comprising a captured image of a field of view of a camera, the system comprising:

[0026] A co-ordinate determining unit adapted to determine Cartesian co-ordinates of a point of interest in the field of view in a Cartesian co-ordinate system of the camera;

[0027] A co-ordinate conversion unit adapted to convert the determined Cartesian co-ordinates from the Cartesian co-ordinate system to a spherical co-ordinate system of the camera to produce spherical co-ordinates;

[0028] The spherical co-ordinates being suitable for use in orientating the captured image for display on a display device.

[0029] The system may further comprise a transmitter adapted to transmit the spherical coordinates for use in displaying the image at a display device.

[0030] An embodiment of the invention will now be described in detail, for illustration only, and with reference to the appended drawings, in which:

[0031] Fig 1 is a schematic view of a football pitch with cameras for use in accordance with the invention;

[0032] Fig 2 is a flow chart showing the method according to the invention;

[0033] Fig 3 is a flow chart showing the method performed at a user device according to one embodiment of the invention;

[0034] Fig 4 is a flow chart showing the method performed at the user device according to a further embodiment of the invention.

[0035] Fig 1 is a schematic view of a stadium, shown generally at 1 , hosting a football match for broadcast. There are three cameras 3,4,5 located on gantries (not shown) surrounding a football pitch 2. The cameras are 360-degree cameras. This term will be familiar to those skilled in the art. It is desirable to provide the user with a video stream that maintains focus on a point of interest on the pitch. In the case of a football match the point of interest is typically the ball. Fig 2 is a flow chart showing a method for achieving this objective. At step s1 , the position of a point of interest in a pitch co-ordinate system is determined. The pitch coordinate system is a Cartesian co-ordinate system in which every location on the pitch has x and y values.

[0036] The pitch is divided into three “jurisdictions”, one associated with each camera. Step s2 determines in which camera’s jurisdiction the point of interest lies at a given moment. At this point the zoom level of the determined camera is also calculated. The next step, s3, is to determine if that camera is different to the currently active camera. If so, the active camera is changed to the thus-determined camera at s4, and any previously-recorded field of view angles associated with the previous active camera are removed (s5).

[0037] Step s6 calculates the co-ordinates of the point of interest in the active camera’s pitch co-ordinate system. This produces an (x,y) co-ordinate for the point of interest. At step s7, this (x,y) co-ordinate is mapped to the spherical co-ordinate system of the active camera. This involves converting the (x,y) co-ordinate to two field of view angles, each being the angle between the point of interest and the 360-degree datum point of the active camera, in orthogonal planes. In particular, the angles are:

[0038] (i) an angle between a datum point of the camera in a vertical plane and the point of interest, and

[0039] (ii) an angle between a datum point of the camera in a horizontal plane and the point of interest.

[0040] These two angles are recorded at step s8. At step s9 a filter is used calculate new client field of view angles from these recorded field of view angles. This is to ensure that the angle and zoom level of the image output to the user does not vary too greatly from one frame to the next, which would be uncomfortable to watch.

[0041] At step s10 the calculated client field of view angles and the identity of the active camera are distributed to all client devices. This is done by transmission over a telecomms network in a manner that would be familiar to the skilled person. In alternative embodiments the calculated client field of view angles and the identity of the active camera are distributed to the cloud, in this case a GPU server cluster. As shown in Fig 2, the process then returns to step 1 and the process repeats with a new position of the point of interest. The process iterates continuously to provide updated field of view angles to the client devices as the match proceeds and the point of interest moves around the pitch.

[0042] Fig 3 is a flow chart showing the method performed at a client device that receives field of view angles determined in Fig 1. The client device could be, for example, a television, a laptop computer, a mobile telephone, a VR head-mounted display or a tablet.

[0043] At step s11 the client device receives the field of view angles and identity of active camera distributed at step s10 of Fig 1 . At step s12 the client device changes its field of view angles to the field of view angles received at step s11. At step s13 the client device displays the image using the received field of view angles. In embodiments in which the calculated client field of view angles and the identity of the active camera are distributed to the cloud (i.e. a GPU server cluster) the step of changing the field of view angles to the field of view angles received at step s11 is performed at the GPU server cluster.

[0044] It may be that the device is operating in an “immersion experience” mode. This means that the user is able to re-orientate the image presented on the device so as to view a particular part of the scene. In this situation, the field of view angles received at the client device need to be modified to take this re-orienting into account, if distortion of the image is to be avoided. This is done at step s13 by subtracting the field of view angles received at step s11 from the re-orientation angles applied by the user to the media presented on the device.

[0045] The process then returns to step s11 at which point the client device receives new field of view angles from the process shown in Fig 2 and performs steps s11 to s13 using the new field of view angles. The method of Fig 3 iterates continuously.

[0046] Client-side Orientation Control Mode

[0047] In this mode of operation, the user is given a choice of multiple points of interest to follow. An example of this is when a user wishes to follow a particular driver in a grand prix. In this case the user selects that particular driver. The video stream transmitted to the user uses the selected driver as the point of interest as that driver’s car travels around the track, switching from camera to camera as appropriate. This process is different from that described at Fig 3, because in this process the mapping of the co-ordinates of the point of interest (the selected driver) in a camera’s pitch co-ordinate system to spherical co-ordinates is done at the user device, rather than at the production stage. The process is described in detail in Fig 4.

[0048] At step s30 the user device receives from the production system, the co-ordinates of multiple points of interest in the active camera’s pitch co-ordinate system. At step s31 the user selects a point of interest. At step s32 the difference in coordinates between the selected point of interest and the active camera is calculated. At step s33 this difference in coordinates is converted to angles relative to the orientation of the active camera. In other words, the co-ordinates of the selected point of interest in the pitch coordinate system of the active camera is mapped to angles in the spherical co-ordinate system of the active camera. At step s34 these angles are recorded. At step 35 new client field of view angles are calculated from recorded field of view angles using a filter. At step s38 the client field of view angles are changed to the calculated field of view angles. At step s39 the client field of view angles are changed to the calculated field of view angles. At step s40, the media is displayed on the user device using the client field of view angles and any encoding orientation angles. The process then returns to step s30 for the next frame.

Claims

Claims1. A method of streaming video content, the video content comprising a captured image of a field of view of a camera, the method comprising:Determining Cartesian co-ordinates of a point of interest in the field of view in a Cartesian co-ordinate system of the camera;Converting the determined Cartesian co-ordinates from the Cartesian co-ordinate system to a spherical co-ordinate system of the camera to produce spherical coordinates of the point of interest;The spherical co-ordinates being suitable for use in orientating the captured image for display on a display device.

2. A method according to claim 1 , the method further comprising transmitting the spherical co-ordinates for use in displaying the captured image at a display device.

3. A method according to claim 1 or claim 2 in which the co-ordinates in the spherical coordinate system comprise an angle between a datum point of the camera in a vertical plane and the point of interest.

4. A method according to any preceding claim in which the co-ordinates in the spherical co-ordinate system comprise an angle between a datum point of the camera in a horizontal plane and the point of interest.

5. A method according to any preceding claim, wherein the camera is a 360 degree camera and the method further comprised capturing the captured image at the 360- degree camera.

6. A method according to any preceding claim, the method further comprising transmitting the captured image for use in displaying the captured image at the display device.

7. A method according to any preceding claim, the method further comprising receiving the transmitted captures image and the transmitted spherical co-ordinates at the display device.

8. A method according to any preceding claim, the method further comprising, at the display device, orientating the captured image by field of view angles corresponding to the spherical co-ordinates.

9. A method according to any preceding claim, the method further comprising presenting the captured image on the display device.

10. A method according to any preceding claim, the method further comprising transmitting the co-ordinates of a point of interest in the field of view in a Cartesian coordinate system of the camera.

11. A method of video streaming comprising:Receiving a captured image of a field of view of a camera receiving spherical co-ordinates in a spherical co-ordinate system of a point of interest in the field of view of the camera;Re-orientating the image by field of view angles corresponding to the spherical coordinates to produce a re-oriented image.

12. A method according to claim 11 , further comprising displaying the re-orientated image on a display.

13. A method according to claim 11 , further comprising transmitting the re-orientated image to a display device for display.

14. A computer-readable medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any preceding claim.

15. A video streaming system for streaming video content, the video content comprising a captured image of a field of view of a camera, the system comprising:A co-ordinate determining unit adapted to determine Cartesian co-ordinates of a point of interest in the field of view in a Cartesian co-ordinate system of the camera;A co-ordinate conversion unit adapted to convert the determined Cartesian co-ordinates from the Cartesian co-ordinate system to a spherical co-ordinate system of the camera to produce spherical co-ordinates;The spherical co-ordinates being suitable for use in orientating the captured image for display on a display device.

Images