Telepresence system

Abstract

A telepresence system comprising a display screen for displaying a composition image to a user, wherein the display screen comprises a plurality of display nodes, and wherein each display node corresponds to a camera of a camera array and the display nodes are spaced according to the geometry of the camera array. An eye tracking device identifies the three-dimensional location of an eye with respect to the display nodes and a controller receives a plurality of image frames, wherein each image frames corresponds with a camera point of view of a camera of a camera array and determines the location of an extrapolated point of view a distance from the rear side of a camera plane formed by the camera points of view that corresponds with the eye location. An extrapolated frame location is identified that corresponds with a light ray captured by a corresponding camera at its camera point of view wherein the light ray is aligned with a vector from the extrapolated point of view to the corresponding camera point of view. A composition image comprising a plurality of virtual centre of views arranged to correspond with the arrangement of the camera point of views is provides wherein each virtual centre of view corresponds with a camera point of view and positions each image frame on the composition image such that its extrapolated frame location is coincident with its image frame's corresponding virtual centre of view. Portions of the composition image between the virtual centre of views are interpolated and the composition image is displayed such that the virtual centre of views are co-located with corresponding display nodes.

Claims

1. A method of providing an extrapolated view, the method comprising the steps of:receiving a plurality of image frames, wherein each image frame corresponds to a camera point of view of a camera of a camera array;determining the location of an extrapolated point of view a distance from the rear side of a camera plane formed by the camera points of view;identifying an extrapolated frame location on each image frame that corresponds with a light ray captured by a corresponding camera, wherein the light ray is aligned with a vector from the extrapolated point of view to the corresponding camera point of view;providing a composition image comprising a plurality of virtual centre of views arranged to correspond with the arrangement of the camera point of views; and wherein each virtual centre of view corresponds with a camera point of view; andpositioning each image frame on the composition image such that its extrapolated frame location is coincident with its image frame's corresponding virtual centre of view.

2. The method according to claim 1, wherein:the plurality of image frames comprise one or more first image frames and a plurality of second image frames; andwherein the plurality of second image frames are lower definition than the plurality of first image frames;wherein a lower definition image has a smaller file size per pixel; andwherein the number of second image frames is greater than the number of first image frames; andwherein the step of positioning each image frame on the composition image further comprises enhancing portions of the composition image corresponding to second image frames with information from the first image frames, and wherein enhancing does not alter the locations of features on the composition image.

3. The method according to claim 2, wherein the second image frames are line images.

4. The method according to claim 2 or 3, wherein the second image frames are vector images.

5. The method according to any of claims 2 to 4, wherein:the second image frames have 1-bit colour and the first image frames have greater than 1-bit colour; andenhancing the composition image comprises enhancing the colour of the composition image.

6. The method according to any of claims 2 to 5, wherein:the plurality of image frames further comprises a plurality of third image frames;wherein the third image frames are lower definition than the plurality of first image frames and higher definition than the plurality of second image frames;and number of third image frames is greater than the number of first image frames and less than the number of second image frames.

7. The method according to claim 6, wherein the number of second image frames is greater than 200 and the number of third image frames is greater than 100.

8. The method according to any previous claim, wherein the step of obtaining an extrapolated point of view comprises identifying an eye location of a user in relation to display nodes on a display; wherein each display node corresponds to a (e.g. remote) camera of the camera array; and wherein the method further comprises displaying the composition image on the display such that virtual centre of views of the composition are coincident with display nodes.

9. The method according to claim 8, wherein the display is provided and / or viewed using an augmented or virtual reality headset; and wherein the display comprises a virtual screen a distance from the user; wherein the display nodes are located on the virtual screen; and wherein the extrapolated point of view corresponds with an eye location relative to the position of the display nodes.

10. The method according to any previous claim, wherein the method further comprises the steps of:capturing images of the local scene in front of the display screen using a local camera array;transmitting the images to a remote telepresence display device for enabling two-way telepresence;receiving geometry from a remote telepresence device comprising a separation distance between a remote display screen and a remote camera plane; andactuating the local display screen (or locating the local virtual screen) to be a distance from the camera plane equal to the received separation distance.

11. The method according to any previous claim, wherein after the step of positioning each image frame, the method further comprises applying transparency to overlapping portions of image frames such that the combined effect of overlapping transparencies renders the composition image opaque across the area covered by the image frames.

12. The method according to any previous claim, wherein after the step of positioning each image frame, the method further comprises applying an interpolation algorithm to resolve portions of the composition image between the virtual centre of views.

13. The method according to claim 12, wherein applying an interpolation algorithm comprises identifying the same feature in a plurality of image frames; and wherein the method further comprises positioning the feature at a single interpolated location on the composition image.

14. The method according to claim 13, wherein the step of positioning comprises transforming (for example warping, stretching or shearing) a portion of such image frames distal to theircorresponding extrapolated frame location along radial lines extending away from the extrapolated frame location.

15. The method according to any of claims 12 to 14, wherein the interpolation algorithm comprises an artificial neural network.

16. A camera array suitable for capturing images for the method according to claim 1,wherein the camera array is planar with a height less than 3 metres and a width less than 3 metres, and comprises one or more first definition cameras and a plurality of second definition cameras;wherein the first definition cameras have higher definition than the second definition cameras;wherein a higher definition camera outputs an image with larger file size per pixel when at maximum camera settings; andwherein the number of cameras is greater than 200; and wherein the number of second definition cameras is greater than the number of first definition cameras.

17. The camera array according to claim 16, wherein the first definition cameras output a file size per pixel when at maximum camera settings greater than 5 times the second definition cameras.

18. The camera array according to claim 17, wherein the camera array further comprises a controller; and wherein the controller is configured to convert the images from the second definition cameras into monochrome line images or vector images.

19. The camera array according to claim 17 or 18, wherein the camera array further comprises a plurality of third definition cameras; wherein the third definition cameras have higher definition than the second definition cameras and lower definition that the first definition cameras; and wherein the number of second definition cameras is greater than the number of third definition cameras.

20. A telepresence system comprising:a display screen for displaying a composition image to a user, wherein the display screen comprises a plurality of display nodes, and wherein each display node corresponds to a camera of a camera array and the display nodes are spaced according to the geometry of the camera array;an eye tracking device for identifying the three-dimensional location of an eye with respect to the display nodes;a controller, wherein the controller is configured to:receive an eye location from the eye tracking device;receive a plurality of image frames, wherein each image frames corresponds with a camera point of view of a camera of a camera array;determine the location of an extrapolated point of view a distance from the rear side of a camera plane formed by the camera points of view that corresponds with the eye location;identify an extrapolated frame location on each image frame that corresponds with a light ray captured by a corresponding camera at its camera point of view, wherein the light ray is aligned with a vector from the extrapolated point of view to the corresponding camera point of view;provide a composition image comprising a plurality of virtual centre of views arranged to correspond with the arrangement of the camera point of views; and wherein each virtual centre of view corresponds with a camera point of view; andposition each image frame on the composition image such that its extrapolated frame location is coincident with its image frame's corresponding virtual centre of view;interpolate portions of the composition image between the virtual centre of views; anddisplay the composition image on the display screen such that the virtual centre of views are co-located with corresponding display nodes.

21. The telepresence system according to claim 20, further comprising a local camera array; wherein the display screen is located proximal the local camera array, and wherein each camera of the local camera array is positioned with respect to the display screen such that it can capture images through the display screen.

22. The telepresence system according to claim 20 or 21, wherein the telepresence system comprises an augmented or virtual reality headset; and wherein the display screen is a virtual screen positioned in the environment of the user when viewed through the headset.

23. The telepresence system according to claim 21, wherein the display screen comprises holes, and wherein each local camera comprises a capture portion positioned within a hole.

24. The telepresence system according to claim 21, wherein the display screen is semitransparent.

25. The telepresence system according to any of claims 21, wherein the display screen is moveable by actuators to increase or decrease a separation distance from the camera array.

Images