Extended reality device

Abstract

An extended reality device comprising: a semi-transparent and partially reflective screen, an image sensor and an image source. The image sensor comprises a camera situated at a viewing position on a first side of the screen and arranged to capture images of radiation arriving at the image sensor having been transmitted through the screen. The image source is configured to project an image on to the second side of the screen, such that the image is reflected by the screen so as to appear visible on the screen when viewed from the second side of the screen. The projected image comprises an image of an animal including an eye. The image is projected onto the second side of the screen such that the eye included in the image is reflected at a position on the screen which aligns with the viewing position of the camera.

Description

EXTENDED REALITY DEVICETECHNICAL FIELD

[0001] The present disclosure relates to an extended reality device, a communication system and a method of operating an extended reality device.BACKGROUND

[0002] Extended reality (XR) encompasses augmented reality (AR), virtual reality (VR), and mixed reality (MR) technologies, all of which are designed to immerse users in a digitally enhanced environment. Extended reality devices can overlay digital content onto the real world (AR), create entirely virtual environments (VR), or merge digital elements with the physical world (MR).

[0003] An extended reality device may be used as a telepresence device to provide a user with the impression that they are present at a place other than their true location and / or to provide a user with the impression that another person or object which is situated in a geographically separate location is present with them. For example, an extended reality device may be used as a telepresence device to enable visual and / or auditory communication with another person who is situated at a geographically separate location.

[0004] Additionally or alternatively, an extended reality device may be used to realise a visual and / or auditory output associated with an artificially generated character or person. For example, an extended reality device may be used to display images of and / or output sound associated with a virtual assistant in order to provide the impression that the virtual assistant is present with them in the same location as the user.

[0005] Existing extended reality devices may suffer from limitations that they do not produce a realistic impression of the presence of another person in the same location as a user. Additionally or alternatively, existing extended reality device may suffer from limitations that they require expensive and / or complex hardware in order to produce a sufficiently realistic impression of the presence of another person in the same location as a user.

[0006] It is in this context that the present disclosure has been devisedSUMMARY

[0007] An extended reality device may include an image sensor comprising at least one camera which captures images of a user which is interacting with the device. The extended reality device may be arranged to display an image of an animal (e.g., a human) which a user of the extended reality device may seek to interact with. The image of the animal may bebased on captured images of a real animal and / or may be artificially generated. It has been realised that a more realistic impression of the displayed animal may be achieved by locating a camera of the image sensor at a viewing position which approximately corresponds to a viewing position of an eye of the displayed animal. In this way, the images which are captured by the camera are taken from a perspective which approximately corresponds with the viewing perspective of the displayed animal (if the displayed animal were present in the location at that it appears when displayed). It has been found that such an arrangement can greatly enhance the ability of an extended reality device to realistically provide an impression of interaction with the displayed animal as though the displayed animal were present in the same vicinity as the user. Furthermore, devices and methods have been developed which do not require overly expensive and / or complex hardware to realise the effects described herein.

[0008] According to a first aspect of the present disclosure there is provided an extended reality device comprising: a semi-transparent and partially reflective screen; an image sensor comprising a camera situated at a viewing position on a first side of the screen and arranged to capture images of radiation arriving at the image sensor having been transmitted through the screen; an image source configured to emit light so as to project an image on to the second side of the screen, such that the image is reflected by the screen so as to appear visible on the screen when viewed from the second side of the screen. The projected image comprises an image of an animal including an eye and wherein the image source is configured to project the image onto the second side of the screen such that the eye included in the image is reflected at a position on the screen which substantially aligns with the viewing position of the camera of the image sensor on the first side of the screen.

[0009] By projecting the image such that the eye is reflected at a position on the screen which substantially aligns with the viewing position of the camera, the camera captures images from a perspective which substantially corresponds with a perspective of the eye (if the eye were real) as it appears on the screen. This allows a highly realistic impression of lifelike interaction with the animal whose image is projected onto the screen. For example, the image of the animal may be artificially generated and based, at least in part, on images captured by the camera. For example, images captured by the camera may be used to detect a position of a user interacting with the extended reality device and / or gestures performed by the user. These detections may be used to generate images of the animal which realistically respond to the user. For example, an eye gaze of the image of the animal may be directed to a detected position of the user to give the user the impression that the animal is maintaining eye contact with the user. Directing an eye gaze towards the user is enabled by capturing images from the perspective of the eye of the animal such that the position of the user isdetermined (from captured images) from the perspective of the eye. Additionally or alternatively, gestures (which may include facial expressions) of the user may be detected based on captured images of the user. Since the user is interacting with the extended reality device their gestures are likely to be directed towards the eye of the animal. The position of the eye of the animal therefore provides an optimal perspective from which to capture images of the user and accurately detect gestures performed by the user.

[0010] In some examples, the extended reality device may be used to project images based on captured images of a real animal (e.g., human). For example, the extended reality device may be used to facilitate live video communication between two users situated at different locations. A user interacting with the extended reality device is likely to direct their eye gaze and / or gestures towards an eye of a projected image on the screen. Images captured by a camera which is substantially aligned with the position of the eye on the screen will capture the user substantially from the perspective of the eye (if the eye were real). Such captured images may be provided to another device (e.g., another extended reality device of the type described herein) for display to a user of the another device. Due to the perspective from which the images are captured, the images may appear to the another user to realistically replicate lifelike interaction. For example, it may appear to the another user that the user of the extended reality device is maintaining direct eye contact with them and / or directing gestures directly towards them.

[0011] The extended reality device therefore provides a realistic and lifelike impression of real face-to-face interaction with the animal (e.g., human) whose image is projected on to the screen, in a way which is not possible with other device arrangements. It will be appreciated that the effects which are utilised to achieve these advantages utilise physical parameters which are based on human physiology. For example, the effects are achieved by virtue of the way in which a scene is viewed and perceived by a human eye. Furthermore, an extended reality device of the type disclosed herein does not require complex hardware such as a holographic display or a virtual reality headset.

[0012] The semi-transparent and partially reflective screen is semi-transparent and partially reflective to visible radiation. That is, if visible radiation is incident on the screen then a first portion of the visible radiation is reflected by the screen and a second portion of the visible radiation is transmitted through the screen. The screen may further be semi-transparent and partially reflective to other wavelengths of electromagnetic radiation (other than visible wavelengths). For example, the screen may be semi-transparent and partially reflective to infrared radiation. The screen may comprise a transparent substrate (e.g., glass or perspex) and a partial reflector attached to the substrate.

[0013] The image sensor may comprise a single camera or a plurality of cameras. A camera included in the image sensor may be configured to capture images of visible radiation. A camera included in the image sensor may additionally or alternatively be configured to capture images of infra-red radiation (e.g., near-infrared radiation). The image sensor may be positioned close to the second side of the screen. The image sensor may be positioned directly behind the second side of the screen. For example, at least part of the image sensor may be in contact with the second side of the screen.

[0014] The image source may be positioned outside of a field of view of the camera such that the light emitted from the image source is not captured by the camera.

[0015] The animal may comprise a human. The animal may comprise a real animal (e.g., may be based on a captured image of a real animal) or may be artificially generated. The animal may comprise an artificially generated character and / or a mythical creature.

[0016] References herein to an image may include a plurality of images. For example, an image as referred to herein may comprise a sequence of a plurality of images which form a moving image (video). For example, the image source may be configured to emit light so as to project a moving image (video) on to the second side of the screen.

[0017] The image sensor may comprise a depth sensor configured to determine a distance of objects situated on the second side of the screen from the image sensor.

[0018] A depth sensor may enable the position of objects in a scene captured by the image sensor to be determined in three dimensions. This may facilitate accurate detection of the position (relative to the extended reality device) of a user interacting with the extended reality device. Additionally or alternatively, this may facilitate accurate detection of gestures (which may include facial expressions) performed by the user. The depth sensor may comprise a stereo depth sensor. A stereo depth sensor may comprise at least two cameras separated from each other by a separation distance.

[0019] The image sensor may comprise a first camera situated at a first viewing position on the first side of the screen and a second camera situated at a second viewing position on the first side of the screen.

[0020] The first viewing position and the second viewing position may approximately correspond to viewing positions of two eyes of an animal (e.g., human) whose image is projected onto the screen. The first and second cameras may therefore capture images from the perspective of eyes of the animal whose image is projected onto the screen.

[0021] The image may comprise an image of an animal including a first eye and a second eye. The image source may be configured to project the image onto the second side of thescreen such that the first eye included in the image is reflected at a position on the screen which substantially aligns with the first viewing position of the first camera of the image sensor on the first side of the screen and the second eye included in the image is reflected at a position on the screen which substantially aligns with the second viewing position of the first camera of the image sensor on the second side of the screen.

[0022] The extended reality device may further comprise an actuator operable to move the image sensor so as to change the viewing position at which the camera is situated on the first side of the screen.

[0023] The actuator may be operable to move the image sensor such that the camera is situated at a viewing position which substantially aligns with a position on the screen from which an eye in the projected image is reflected. For example, a position in the projected image of an eye may be determined (e.g., using image processing) and a corresponding position on the screen at which the eye will be reflected may be determined. The actuator may then be used (e.g., a control signal sent to the actuator) to move the camera to align with the position at which the eye is reflected. This enables dynamic positioning of the image sensor to adapt to different images which may be projected onto the screen. In examples, in which the image sensor comprises a plurality of cameras, a single actuator (or group of actuators working together) may be provided to move the entire image sensor comprising the plurality of cameras. Alternatively, a plurality of actuators may be provided to enable independent movement of different cameras in the image sensor.

[0024] The extended reality device may further comprise an absorber arranged around the image sensor on the first side of the screen and configured to absorb visible radiation incident on the absorber.

[0025] The absorber may, for example, comprise an enclosure arranged to enclose (in cooperation with the screen) the image sensor. At least a portion of the absorber which faces the first side of the screen may be configured to absorb visible radiation such that it appears to be darkly coloured (e.g., black) when viewed through the screen. The absorber may be arranged such that the image sensor appears to be dimly lit when viewed through the screen. This may mean that the image sensor does not appear to be visible when viewed through the screen (from the second side of the screen). For example, when the image is projected onto the screen the reflection of the image from the screen may appear (when viewed from the second side of the screen) to be significantly brighter than any visible radiation reflected from the image sensor. Consequently, the image sensor may not appear visible when viewed from the second side of the screen.

[0026] The absorber may extend to form a dark region on the first side of the screen. Visible radiation transmitted through the screen and incident on the dark region may be absorbed by the absorber. The image source may be configured to project the image onto the second side of the screen such that the image is reflected at positions on the screen which substantially align with positions within the dark region on the second side of the screen.

[0027] Visible radiation which is transmitted through the screen to be incident on the dark region may be absorbed by the absorber and not reflected back to the second side of the screen. The absorber may further function to block visible radiation from passing from the first side of the screen to the second side of the screen in the dark region. When viewed from the second side of the screen, the dark region (over which the absorber extends) may therefore appear to be substantially dark such that nothing in the dark region appears visible when viewed from the second side of the screen.

[0028] The absorber may extend such that the dark region corresponds to a region onto which part of an image is projected a majority of the time during operation of the extended reality device. In at least some implementations, a portion of an image may be projected onto a portion of the screen which is not included in the dark region.

[0029] The image source may be arranged to project the image along an optical axis. The image source and the screen may be arranged relative to each other such that the optical axis is not aligned with a surface normal of the screen.

[0030] Arranging the image source at an angle with respect to the screen may enable the image which is reflected from the screen to be viewed without the view being blocked by the image source. It may further enable the image source to be hidden from view to a user viewing the image reflected from the screen. The optical axis may be arranged at an angle of approximately 45 degrees with respect to a surface normal of the screen.

[0031] The image source may comprise an electronic display configured to display the image on the electronic display. The electronic display and the screen may be arranged relative to each other such that light emitted from the electronic display is incident on the second side of the screen such that the image is reflected by the screen.

[0032] The electronic display may, for example, comprise a television or other form of electronic monitor. The electronic display may comprise a flat-panel display.

[0033] The image source may comprise an image projector arranged to project the image onto the second side of the screen.

[0034] The image sensor may be configured to provide images captured by the camera to a control system comprising an image generator configured to generate the image to projectonto the second side of the screen and wherein the image source is configured to receive the generated image from the image generator and project the generated image on to the second side of the screen.

[0035] The image to be projected onto the screen may be based, at least in part, on an image provided to the control system.

[0036] The extended reality device may further comprise the control system.

[0037] The control system may be configured to generate an artificial image of the animal for projection onto the second side of the screen. The control system may be configured to generate the artificial image based on at least one captured image received from the image sensor.

[0038] The control system may comprise at least one Artificial Intelligence (Al) and / or Machine Learning (ML) engine or model for generating an artificial image. The artificial image may be generated based on at least one captured image of a user interacting with the extended reality device. This may enable artificial images to be generated which create a realistic impression of interaction with the user. One or more properties which are detected from at least one captured image may be used to provide a prompt to an Al and / or ML model for generating the artificial image. For example, one or more properties such as a position of the user and / or a gesture performed by the user may be determined from captured images of a user. Such determined properties may be used to form a prompt to an Al and / or ML model to generate an artificial image which is responsive to the determined properties. For example, an image may be generated which directs an eye gaze to a determined position of the user and / or responds to a determined gesture of a user.

[0039] The control system may be configured to: determine a position of an object relative to the image sensor based on the at least one image captured by the image sensor; and generate the artificial image such that when the generated image is projected onto the second side of the screen and viewed from the second side of the screen, the eye included in the generated image appears to be directed to the determined position of the object.

[0040] A direction in which an eye appears to be directed may be referred to herein as an eye gaze direction. The object whose position is determined may comprise all or part of a user interacting with the extended reality device. For example, the object may comprise an eye of the user. Generating an image in which an eye appears to be directed towards the determined position of an object may provide for a realistic impression of interaction with the animal whose image is projected onto the screen. For example, where the object is an eye of the user, the generated image may appear to be maintaining eye contact with the user. Thisis enabled by the positioning of a camera (based on which the position of the object is determined) to be aligned with the eye in the projected image.

[0041] The control system may be configured to: detect one or more of a gesture of a subject in at least one image received from the image sensor; and generate the artificial image based on the detected gesture.

[0042] A detected gesture may include a detected expression such as a facial expression. A detected gesture may include detected body positioning and / or body language. A detected gesture may include a detected positioning of a body part. For example, a detected gesture may comprise detection of the subject waving or performing some other form of gesture. The subject may comprise a user interacting with the extended reality device.

[0043] The extended reality device may further comprise a sound output device. The image source may be configured to provide images captured by the camera to a control system comprising a language generator, wherein the control system is configured to generate natural language outputs based on at least one image captured by the camera. The sound output device may be configured to output sound corresponding to speech of a natural language output received from the control system.

[0044] The language generator may be further provided with detected speech of a user interacting with the extended reality device. For example, the extended reality device may comprise an audio sensor configured to record audio of a user speaking. The control system may be configured to perform speech recognition (e.g., by a language detector included in the control system) on the recorded audio to generate text corresponding to the user's speech. The text may be provided to the language generator to generate an output. The language model may comprise a large language model.

[0045] The control system may be configured to determine one or more properties relating to a user interacting with the extended reality device from at least one captured image. For example, one or more properties such as a position of the user and / or a gesture performed by the user may be determined from at least one captured image of a user. Such determined properties may be used to form a prompt to the language generator. For example, a determined property such as a detected gesture performed by a user may be provided in a prompt to the language generator such that the language generator generates natural language which is responsive to the detected gesture.

[0046] The extended reality device may be configured to generate the image of the animal based on an image captured by another device in communication with the extended reality device.

[0047] The extended reality device may be used as a telepresence device to enable video communication with a user interacting with the another device.

[0048] According to a second aspect of the present disclosure there is provided a communication system comprising a first extended reality device according to the first aspect and a second extended reality device according to first aspect. The first extended reality device is configured to send images captured by the image sensor of the first extended reality device to the second extended reality device. The second extended reality device is configured to generate and project images based on the images received from the first extended reality device onto the second side of the screen of the second extended reality device. The second extended reality device is configured to send images captured by the image sensor of the second extended reality device to the first extended reality device. The first extended reality device is configured to generate and project images based on the images received from the second extended reality device onto the second side of the screen of the first extended reality device.

[0049] According to a third aspect of the present disclosure there is provided a method of operating an extended reality device, the method comprising: capturing an image using an image sensor comprising a camera situated at a viewing position on a first side of a semitransparent and partially reflective screen; projecting an image of an animal including an eye onto a second side of the screen such that the image is reflected by the screen so as to appear visible on the screen when viewed from the second side of the screen, wherein the image is projected onto the second side of the screen such that the eye included in the image is reflected at a position on the screen which substantially aligns with a viewing position of the camera of the image sensor on the first side of the screen.

[0050] The image sensor may comprise a depth sensor and the method may comprise determining a distance of objects situated on the second side of the screen from the image sensor.

[0051] The image sensor may comprise a first camera situated at a first viewing position on the first side of the screen and a second camera situated at a second viewing position on the first side of the screen.

[0052] The image may comprise an image of an animal including a first eye and a second eye. The method may comprise projecting the image onto the second side of the screen such that the first eye included in the image is reflected at a position on the screen which substantially aligns with the first viewing position of the first camera of the image sensor on the first side of the screen and the second eye included in the image is reflected at a positionon the screen which substantially aligns with the second viewing position of the first camera of the image sensor on the second side of the screen.

[0053] The method may comprise moving the image sensor so as to change the viewing position at which the camera is situated on the first side of the screen.

[0054] The extended reality device may further comprise an absorber arranged around the image sensor on the first side of the screen and configured to absorb visible radiation incident on the absorber.

[0055] The absorber may extend to form a dark region on the first side of the screen. Visible radiation transmitted through the screen and incident on the dark region may be absorbed by the absorber. The image source may be configured to project the image onto the second side of the screen such that the image is reflected at positions on the screen which substantially align with positions within the dark region on the second side of the screen.

[0056] The image may be projected along an optical axis. The optical axis may not be aligned with a surface normal of the screen.

[0057] The image source may comprise an electronic display configured to display the image on the electronic display. The electronic display and the screen may be arranged relative to each other such that light emitted from the electronic display is incident on the second side of the screen such that the image is reflected by the screen.

[0058] The image source may comprise an image projector arranged to project the image onto the second side of the screen.

[0059] The method may comprise providing images captured by the camera to a control system comprising an image generator configured to generate the image to project onto the second side of the screen. The method may comprise the image source receiving the generated image from the image generator and projecting the generated image on to the second side of the screen.

[0060] The extended reality device may further comprise the control system.

[0061] The method may comprise generating an artificial image of the animal for projection onto the second side of the screen. The artificial image may be generated based on at least one captured image received from the image sensor.

[0062] The method may further comprise determining a position of an object relative to the image sensor based on the at least one image captured by the image sensor; and generate the artificial image such that when the generated image is projected onto the second side of the screen and viewed from the second side of the screen, the eye included in the generated image appears to be directed to the determined position of the object.

[0063] The method may comprise detecting one or more of a gesture of a subject in at least one image received from the image sensor; and generating the artificial image based on the detected gesture.

[0064] The extended reality device may further comprise a sound output device. The method may comprise providing images captured by the camera to a control system comprising a language generator, wherein the control system is configured to generate natural language outputs based on at least one image captured by the camera. The method may comprise outputting, by the sound output device, sound corresponding to speech of a natural language output received from the control system.

[0065] The method may comprise generating the image of the animal based on an image captured by another device in communication with the extended reality device.

[0066] It will be appreciated from the foregoing disclosure and the following detailed description of the examples that certain features and implementations described as being optional in relation to any given aspect of the disclosure set out above should be understood by the reader as being disclosed also in combination with the other aspects of the present disclosure, where applicable. Similarly, it will be appreciated that any attendant advantages described in relation to any given aspect of the disclosure set out above should be understood by the reader as being disclosed as advantages of the other aspects of the present disclosure, where applicable. That is, the description of optional features and advantages in relation to a specific aspect of the disclosure above is not limiting, and it should be understood that the disclosures of these optional features and advantages are intended to relate to all aspects of the disclosure in combination, where such combination is applicable.BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Examples of the present disclosure will now be described, with reference to the accompanying drawings, in which:- FIG. 1 is a schematic illustration of an example extended reality device;- FIG. 2 is a schematic illustration of an example extended reality device and a user;- FIG. 3 is a schematic illustration of an example image sensor;- FIG. 4 is a schematic illustration of an example extended reality device as viewed from a second side of a screen of the extended reality device;- FIG. 5 is a further schematic illustration of an example extended reality device as viewed from a second side of a screen of the extended reality device;- FIG. 6 is a still further schematic illustration of an extended reality device as viewed from a second side of a screen of the extended reality device;- FIG. 7 is a schematic illustration of an example communication system;- FIG. 8 is a schematic illustration of functional components of an example extended reality device;- FIG. 9 is a flow chart of an example method of operating an extended reality device; and- FIG. 10 is a schematic illustration of an example electronic device which may be used to implement all or part of a method or device disclosed herein.DETAILED DESCRIPTION

[0068] Hereinafter, examples of the disclosure are described with reference to the accompanying drawings. However, it should be appreciated that the disclosure is not limited to the described examples, and all changes and / or equivalents or replacements thereto also belong to the scope of the disclosure. The same or similar reference denotations may be used to refer to the same or similar elements throughout the specification and the drawings.

[0069] As used herein, the terms “have,” “may have,” “include,” or “may include” a feature (e.g., a number, function, operation, or a component such as a part) indicate the existence of the feature and do not exclude the existence of other features. Throughout the description and claims of this specification, the words “comprise” and “contain” and variations of them mean “including but not limited to”, and they are not intended to (and do not) exclude other components, integers or steps. Throughout the description and claims of this specification, the singular encompasses the plural unless the context otherwise requires. In particular, where the indefinite article is used, the specification is to be understood as contemplating plurality as well as singularity, unless the context requires otherwise.

[0070] As used herein, the terms “A or B,” “at least one of A and / or B,” or “one or more of A and / or B” may include all possible combinations of A and B. For example, “A or B,” “at least one of A and B,” “at least one of A or B” may indicate all of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.

[0071] As used herein, the terms “first” and “second” may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, reference to a first component and a second component may indicate different components from each other regardless of the order or importance of the components.

[0072] It will be understood that when an element (e.g., a first element) is referred to as being (physically, operatively or communicatively) “coupled with / to,” or “connected with / to” another element (e.g., a second element), it can be coupled or connected with / to the other element directly or via a third element. In contrast, it will be understood that when an element (e.g., a first element) is referred to as being “directly coupled with / to” or “directly connected with / to” another element (e.g., a second element), no other element (e.g., a third element) intervenes between the element and the other element.

[0073] The terms as used herein are provided merely to describe some embodiments thereof, but not to limit the scope of other embodiments of the disclosure. It is to be understood that the singular forms “a,” “'an,” and “the” include plural references unless the context clearly dictates otherwise. All terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of the disclosure belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0074] FIG. 1 is a schematic illustration of an extended reality (XR) device 102. The XR device 102 comprises a screen 104, an image sensor 110 and an image source 114. In the example shown in FIG. 1, the XR device 102 further comprises a control system 116, an audio sensor 118, a sound output device 120 and an enclosure 124. However, in some examples, the XR device 102 need not necessarily include the control system 116, audio sensor 118, sound output device 120 and / or enclosure 124. The image sensor 110, image source 114, audio sensor 118 and / or sound output device 120 may be communicatively coupled with the control system 116 as indicated by lines joining these components in FIG. 1. For example, wired connections (e.g., with cables) may be established between the control system 116 and one or more of the image sensor 110, image source 114, audio sensor 118 and / or sound output device 120. Additionally or alternatively wireless connections may be established between components. At least some of the components of the XR device 102 may be located within and / or may be supported by a housing 130, as shown in FIG. 1.

[0075] The screen 104 is semi-transparent and partially reflective. The screen 104 may in particular be semi-transparent and partially reflective with respect to visible radiation. That is, if visible radiation is incident on the screen 104 then a fist portion of the visible radiation is transmitted by the screen 104 and a second portion of the visible radiation is reflected by the screen 104. Additionally or alternatively, the screen 104 may be semi-transparent and partially reflective to other wavelengths of electromagnetic radiation. For example, the screen 104 may be semi-transparent and partially reflective to infra-red radiation (e.g., near-infrared radiation).

[0076] The screen 104 may comprise a transparent substrate and a partial reflector attached to or included in the substrate. For example, the screen 104 may comprise a pane or sheet (which may be referred to as a substrate) of a substantially transparent (e.g., with respect to visible radiation) material such as glass or perspex and a partially reflective coating may be attached to at least one surface of the substrate. Additionally or alternatively, a partial reflector may be embedded within a transparent substrate or may be sandwiched between two transparent substrates. A partial reflector may take any suitable form such as a thin film of metalised material.

[0077] The screen 104 has a first side 106 and a second side 108. Whilst the screen 104 is shown as being entirely flat in FIG. 1, in other examples the screen 104 may include curvature (e.g., a curved screen). The screen 104 generally includes two opposing surfaces on the first and second sides of the screen 104. The surfaces may be referred to as a display surface (situated on the second side 108 of the screen 104) and a backside surface (situated on the first side 106 of the screen 104).

[0078] The image sensor 110 is arranged on the first side 106 of the screen 104. As shown in FIG. 1, the image sensor 110 may be positioned in close proximity to the first side 106. For example, at least a portion of the image sensor 110 may be in contact with or at least in close proximity to a portion of the backside surface (on the first side 106) of the screen 104. The image sensor 110 is generally situated at a viewing position 112 on the first side 106 of the screen 104.

[0079] The image sensor 110 includes at least one camera arranged to capture images of radiation arriving at the image sensor 110 having been transmitted through the screen 104. That is, the at least one camera is arranged to capture images of radiation which is incident on the second side 108 of the screen 104 and which is transmitted through the screen 104 to reach the image sensor 110. The at least one camera is therefore arranged to capture images of at least a portion of a scene situated on the second side 108 of the screen 104.

[0080] The XR device 102 is generally configured for being viewed by and / or for interaction with a user 202 situated on the second side 108 of the screen 104. The image sensor 110 is arranged to capture images of at least a part of a user 202 viewing and / or interacting with the XR device 102. FIG. 2 is a schematic illustration of an XR device 102 and a user 202 situated on the second side 108 of the screen 104. For ease of illustration, some of the components of the XR device 102 have been omitted in FIG. 2. The XR device 102 of FIG. 2 may have any of the features and components of the XR device 102 which is described with reference to FIG. 1 and no detailed description of the XR device 102 will be provided with reference to FIG. 2.

[0081] The at least one camera included in the image sensor 110 may be configured to capture visible radiation (and may be referred to as a visible camera). Additionally or alternatively, the at least one camera included in the image sensor may be configured to capture other wavelengths of radiation. For example, the at least one camera may be configured to capture infra-red radiation such as near-infrared radiation.

[0082] In some examples, the image sensor 110 may comprise a plurality of cameras. For example, the image sensor 110 may comprise two cameras. The image sensor 110 may comprise a plurality of cameras configured to capture substantially the same or overlapping wavelengths of radiation. For example, the image sensor 110 may comprise a plurality of visible cameras. In some examples, the image sensor 110 may comprise a plurality of cameras configured to capture different wavelength ranges. For example, the image sensor 110 may comprise at least one visible camera and at least one infrared camera.

[0083] In some examples, the image sensor 110 may comprise a depth sensor. A depth sensor may be configured to determine a distance of objects situated on the second side 108 of the screen 104 from the image sensor 110. For example, the depth sensor may be configured to determine a distance of all or part of a user 202 from the image sensor 110. The depth sensor may comprise a stereo depth sensor comprising two cameras situated with a separation (of, for example, a few centimetres) between them. One or more common objects may be detected in images captured by both of the two cameras. A difference in position of the one or more objects detected in images detected by different cameras may then be used to determine a distance of the one or more objects from the image sensor 110.

[0084] FIG. 3 is a schematic illustration of an image sensor 110 comprising a first camera 302 and a second camera 304 separated by a separation distance 306. The image sensor 110 may be used as part of a depth sensor such as a stereo depth sensor. For example, an object may be detected in a first image captured by the first camera 302 and in a second imagedetected by the second camera 304. Due to the separation distance 306 between the first camera 302 and the second camera 304, the detected object may appear in different locations in the first image detected by the first camera 302 and the second image detected by the second camera 304. A difference in location at which the object is detected in the first image and at which the object is detected in the second image may be used to determine a distance of the object from the image sensor 110. In some examples, the image sensor 110 may further comprise a radiation source for projecting radiation (different to the image projected onto the screen 104 by the image source 114), such as near-infrared radiation onto objects situated on the second side 108 of the screen 104. The position of the projected radiation (which may be patterned radiation) may be detected in images captured by the first camera 302 and the second camera 304 and may, in particular, be used to determine a distance to featureless objects such as flat surfaces. In some examples, the projected radiation may be non-visible radiation such as infrared (e.g., near-infrared radiation) such that the projected radiation is not visible to a human eye. The first camera 302 and second camera 304 may be configured to detect in a wavelength range in which the projected radiation (different to the image projected onto the screen 104 by the image source 114) falls so as to be able to detect the projected radiation.

[0085] Whilst the image sensor 110 shown in FIG. 3 has been described above in the context of a depth sensor, in some examples an image sensor 110 comprising a plurality of cameras (such as the image sensor of FIG. 3) need not necessarily be a depth sensor but may be used for different purposes.

[0086] Returning again to FIG. 1, the image source 114 is configured to project an image on to the second side 108 of the screen 104. The image is projected onto the second side 108 of the screen 104 such that the image is reflected by the screen so as to appear visible on the screen 104 when viewed from the second side 108 of the screen 104. For example, the image appears visible to a user 202 located on the second side 108 of the screen 104.

[0087] The image source 114 may, for example, comprise an electronic display configured to display an image on the electronic display. The electronic display may comprise a television (such as a flat-panel television) and / or a computer monitor or any other suitable form of electronic display. The electronic display emits light in the form of an image displayed on the display. As is shown in FIG. 1, the image source 114 is arranged such that light which is emitted from the electronic display (shown generally with the arrows numbered 126 in FIG. 1) is incident on the second side 108 of the screen 104. At least a portion of the light 126 which is emitted from the electronic display and which is incident on the second side 108 of the screen 104 is reflected by the screen 104. Light which is reflected by thescreen 104 is shown generally with the arrows numbered 128 in FIG. 1. The reflected light 126 propagates away from the second side 108 of the screen 104 such that it appears visible on the screen when viewed from the second side 108 of the screen 104 (e.g., when viewed by a user 202 situated on the second side 108 of the screen 104).

[0088] The image source 114 may be arranged at an angle with respect to the screen 104. In the example shown in FIG. 1, the image source 114 is in the form of a flat-panel electronic display which is arranged at an angle of approximately 45 degrees with respect to the screen 104. Such an arrangement allows the image displayed by the image source 114 to be projected onto the second side 108 of the screen 104 without obstructing the view of the second side 108 of the screen 104 to, for example, a user 202 situated on the second side 108 of the screen 104. Whilst an angle of approximately 45 degrees between the electronic display which forms the image source 114 and the screen 104 is shown in the example of FIG. 1, in other examples, other similar angles may be used to obtain a similar effect.

[0089] In the example, of FIG. 1, the image source 114 (along with the audio sensor 118, sound output device 120 and control system 116) is located within a housing 130. The image source 114 is situated in a recess 132 in the top of the housing 130 (as indicated by the dotted lines in FIG. 1 such that the image source 114 itself is hidden from view of a user 202 situated on the second side 108 of the screen 104. The user 202 may not therefore be able to see the image source 114 itself but can see the image projected by the image source 114 as it is reflected from the screen 104.

[0090] Whilst the image source 114 is embodied in the example of FIG. 1 as an electronic display (e.g., a flat-panel display), in other examples other forms of image source 114 may be used. For example, the image source 114 may comprise an image projector (e.g., a video projector) arranged to project an image onto the second side 108 of the screen 104. The projector may be configured, for example, such that a focal length of the projector corresponds with a distance between the projector and the second side 108 of the screen 104 and such that the projected image appears in focus when reflected from the screen 104.

[0091] In general, and regardless of the form of the image source 114, the image source 114 may project an image along an optical axis. In examples, in which the image source 114 comprises an electronic display, such as a flat-panel display, the optical axis may be parallel with a surface normal of the electronic display and may pass through the centre of the display and / or image displayed on the display. In examples, in which the image source 114 comprises a projector, the optical axis may correspond with an optical axis of the projector (e.g., a central axis of light emitted by the projector). The image source 114 and the screen 104 may bearranged relative to each other such that the optical axis is not aligned with a surface normal of the screen 104 (i.e., there is a non-zero angle between the optical axis and surface normal of the screen 104).

[0092] The image source 114 may be arranged such that it is situated outside of a field of view of the image sensor 110. For example, the image source 114 may be situated outside of the field of view of at least one camera included in the image sensor 110. In this way, radiation which is emitted from the image source 114 may not be captured by the image sensor 110 and may not affect images captured by the image sensor 110.

[0093] In the example, of FIG. 1, the image sensor 110 is situated within an enclosure 124. The enclosure 124 may further extend between the image sensor 110 and the housing 130 to order to encase cabling extending between the enclosure 124 and the housing 130. The enclosure 124 may comprise an opaque housing in which the image sensor 110 is situated. The enclosure 124 may be arranged to block visible radiation from reaching the image sensor 110 (and / or cabling). For example, the enclosure 124 may be arranged to block visible radiation originating from the second side 108 of the screen 104 from being incident on the image sensor 110 (and / or cabling). In this way the only visible radiation which is incident on the image sensor 110 is visible radiation which has been transmitted through the screen 104.

[0094] At least a portion of the enclosure 124 which faces the first side 106 of the screen 104 may be configured to be substantially absorbing to visible radiation. For example, at least a portion of the enclosure 124 which faces the first side 106 of the screen 104 may be black. When viewed from the second side 108 of the screen 104, the enclosure 124 and components enclosed by the enclosure 124 may therefore appear to be dark.

[0095] The image which is projected onto the screen 104 may be projected onto portions of the second side 108 of the screen 104 which coincide with the position of the image sensor 110 and / or the enclosure 124 on the first side 106 of the screen 104. The image which is reflected from the screen 104 may appear to a user 202 situated on the second side 108 of the screen 104 to be significantly brighter then the image sensor 110 and any components inside the enclosure 124. Consequently, the image sensor 110 and / or the enclosure 124 (and any other components situated inside the enclosure) may not appear to be visible to the user 202.

[0096] This effect is similar, for example, to a one-way mirror which comprises a semitransparent and partially reflective mirror. The effect of a one-way mirror is achieved by illuminating one side of the mirror such that it is brightly lit and keeping the other side of the mirror dark. When viewed from the brightly illuminated side of the mirror, reflections from the mirror are significantly brighter than any light which is transmitted from the dark tobrightly lit side of the mirror, such that the dark side of the mirror is not visible when viewed from the brightly lit side. A similar effect is achieved with the XR device 102 since the reflected image appears brighter than the image sensor 110 and enclosure 124 when viewed from the second side 108 of the screen 104. This effect is also similar to the Pepper's ghost illusion.

[0097] FIG. 4 is a schematic illustration of the XR device 102 as viewed from the second side 108 of the screen 104 (for example, from the perspective of the user 202 shown in FIG. 2). The same components of the XR device 102 are labelled with the same reference numerals in FIG. 4 as in FIG. 1, FIG. 2, and FIG. 3 and no further detailed description of these components is provided with reference to FIG. 4. As viewed from the second side 108 of the screen 104, the housing 130 includes an opening 404. The opening 404 may be an acoustic opening to facilitate propagation of sound into and out of the housing 130 for the purpose of sensing audio by the audio sensor 118 and / or outputting sound by the sound output device 120. The audio sensor 118 may comprise any suitable means for detecting and recording sound and may comprise one or more microphones. The sound output device 120 may comprise any suitable means for outputting sound and may comprise one or more speakers.

[0098] When viewed from the second side 108 of the screen 104, a projected image 402 is visible on the screen 104 (as shown in FIG. 4). The projected image 402 is an image of an animal including an eye. In the example shown in FIG. 4, the projected image 402 is an image 402 of a human including the human's face. The image 402 of the human includes a first eye 406 and a second eye 408.

[0099] According to examples disclosed herein, the image source 114 is configured to project the image 402 onto the second side 108 of the screen 104 such that an eye included in the image 402 is reflected at a position on the screen 104 which substantially aligns with the viewing position 112 of a camera of the image sensor 110 on the first side 106 of the screen 104. The viewing position 112 of the camera may therefore substantially correspond with the viewing position of an eye if the eye in the image were a real eye.

[0100] In some examples, the image 402 may include a plurality of eyes and the image sensor 110 may include a plurality of cameras. For example, as explained above with reference to FIG. 3, the image sensor 110 may comprise a first camera 302 and a second camera 304 separated by a separation distance 306. The image 402 may include a first eye 406 and a second eye 408 as shown in FIG. 4. The first camera 302 may be situated at a first viewing position on the first side 106 of the screen 104 and the second camera 304 may be situated at a second viewing position on the first side 106 of the screen 104. The image 402 may beprojected onto the second side 108 of the screen 104 such that the first eye 406 included in the image 402 is reflected at a position on the screen 104 which substantially aligns with the first viewing position of the first camera 302 on the first side 106 of the screen 104. The image 402 may further be projected onto the second side 108 of the screen 104 such that the second eye 408 included in the image 402 is reflected at a position on the screen 104 which substantially aligns with the second viewing position of the second camera 304.

[0101] As will be explained in further detail below, the image 402 which is projected on to the second side 108 of the screen 104 may be an artificially generated image. In such examples, the artificially generated image may be generated for interaction with a user 202 (e.g., to provide a virtual assistant to the user 202). An image 402 may be artificially generated based on the images captured by the image sensor 110. By positioning one or more cameras of an image sensor 110 at viewing positions which substantially align with positions of one or more eyes in the image 402, the image 402 may be generated to appear more realistic and to more accurately simulate life-like interaction with a user 202. For example, eyes in the image 402 may be generated to appear to accurately gaze towards the user 202 based on images 402 of the user 202 captured by the image sensor 110. That is, eyes in the image may appear to be directed towards the user 202. Additionally or alternatively, gestures (which may include facial expressions) and / or positions of the user 202 may be accurately determined by the image sensor 110 and used to accurately simulate life-like interaction with the user 202.

[0102] In some examples, the image 402 may be generated based on one or more images captured by another device in communication with the XR device 102. For example, the XR device 102 may be used as a telepresence device for interacting with another person situated in a remote location, which captures images of the user of the remote device and displays images of the user 202 of the XR device 102. By positioning one or more cameras of an image sensor 110 at viewing positions which substantially align with positions of one or more eyes in the image 402, the images which are captured by the one or more cameras may be displayed by a remote device and may deliver a realistic replication of face-to-face interaction.

[0103] FIG. 5 is a further schematic illustration of the XR device 102 as viewed from the second side 108 of the screen 104 (for example, from the perspective of the user 202 shown in FIG. 2). The same components of the XR device 102 are labelled with the same reference numerals in FIG. 5 as in FIG. 1, FIG. 2, FIG. 3, and FIG. 4 and no further detailed description of these components is provided with reference to FIG. 5. The illustration of FIG. 5 is similar to that of FIG. 4 except that the image 402 of a human is shown schematically in outline onlyand the position of components which are situated on the first side 106 of the screen 104 are shown using dashed lines. Whilst the position of components situated on the first side 106 of the screen 104 are shown in FIG. 5, these components may not be visible when viewed from the second side 108 of the screen 104 due to the effects described above.

[0104] The position of a first eye 406 and a second eye 408 in the image 402 are shown in outline in FIG. 5. Also shown in FIG. 5 is the position of a first camera 302 and a second camera 304 which form part of an image sensor 110 and are positioned on the first side 106 of the screen 104. As shown in FIG. 5 the viewing position of the first camera 302 is substantially aligned with a position on the screen 104 at which an image of the first eye 406 is reflected on the screen 104. Furthermore, the viewing position of the second camera 304 is substantially aligned with a position on the screen 104 at which an image of the second eye 408 is reflected on the screen 104.

[0105] Also shown in FIG. 5 is the position of the outer edge of the enclosure 124. As can be seen in FIG. 5, the enclosure 124 may be located entirely within a region of the screen 104 onto which the image 402 is projected. That is, the projected image 402 may cover substantially all portions of the screen 104 (on the second side 108 of the screen 104) behind which the enclosure 124 is situated (on the first side 106 of the screen 104). As was explained above, the reflection of an image 402 from the screen 104 may be significantly brighter than any light reflected from the enclosure 124 and / or any components situated inside of the enclosure 124 (such as the image sensor 110). Consequently, any portions of the enclosure 124 and / or any components situated inside of the enclosure 124 which coincide with a position at which a portion of the image 402 is reflected from the screen 104 may not be visible when viewed from the second side 108 of the screen 104. In an arrangement such as that which is illustrated in FIG. 5, the enclosure 124 (and / or any components situated inside of the enclosure 124) may not therefore appear visible when viewed from the second side 108 of the screen 104.

[0106] The enclosure 124 may be considered to function as an absorber arranged around the image sensor 110 and configured to absorb visible radiation incident on the absorber (enclosure 124). The enclosure 124 therefore appears to be relatively dark and may not therefore be visible when viewed from the second side 108 of the screen 104.

[0107] In some examples, an additional or alternative absorber may be arranged on the second side 108 of the screen 104. For example, any form of absorber may be arranged on the first side 106 of the screen 104 and configured to absorb visible radiation incident on the absorber (e.g., may be coloured black). In particular a side of the absorber which faces thescreen 104 may be configured to absorb visible radiation incident on it. The absorber may extend to form a dark region on the first side 106 of the screen 104. Visible radiation which is transmitted through the screen 104 to be incident on the dark region may be absorbed by the absorber and not reflected back to the second side 108 of the screen 104. The absorber may further function to block visible radiation from passing from the first side 106 of the screen 104 to the second side 108 of the screen 104 in the dark region. When viewed from the second side 108 of the screen 104, the dark region (over which the absorber extends) may therefore appear to be substantially dark such that nothing in the dark region appears visible from viewed from the second side 108 of the screen 104. The absorber may arranged around the image sensor 110 and / or any other components (e.g., cabling) situated on the first side 106 of the screen 104 so that any such components are included in the dark region.

[0108] FIG. 6 is a further schematic illustration of the XR device 102 as viewed from the second side 108 of the screen 104 (for example, from the perspective of the user 202 shown in FIG. 2). The same components of the XR device 102 are labelled with the same reference numerals in FIG. 6 as in FIG. 1, FIG. 2, FIG. 3, FIG. 4 and FIG. 5 and no further detailed description of these components is provided with reference to FIG. 6. The illustration of FIG. 6 is similar to that of FIG. 4 except that the outer extent of an absorber 602 and the position of a first camera 302 and second camera 304 are shown using dashed white lines in FIG. 6.

[0109] The absorber 602 may, for example, comprise a dark material (e.g., a material which is coloured black) which is arranged on the first side 106 of the screen 104 to form a dark region. The image source 114 may be configured to project the image 402 onto the second side 108 of the screen 104 such that the image 402 is reflected at positions on the screen 104 which substantially align with positions within the dark regions on the second side 108 of the screen 104. In the example shown in FIG. 6, an absorber 602 is arranged on the first side 106 of the screen 104 and extends to form a dark region (the region inside the white-dashed line in FIG. 6). The absorber 602 is arranged to form a dark region which has a similar extent and shape to the image 402 which is projected on to the second side 108 of the screen 104. Consequently, the majority of the image 402 is projected on to positions on the screen 104 which substantially align with positions within the dark region. Such portions of the image 402 will be visible when viewed from the second side 108 of the screen 104. However, objects located on the first side 106 of the screen 104 and which are situated behind the dark region may not be visible when viewed from the second side 108 of the screen 104 due to the optical properties of the absorber 602.

[0110] Whilst the absorber 602 in FIG. 6 has a similar extent and shape to the image 402 which is projected on to the second side 108 of the screen 104, in other examples this maynot be the case. For example, the absorber 602 may have a smaller extent than the image 402 which is projected on to the second side 108 of the screen 104. The image 402 which is projected onto the second side 108 of the screen 104 may be dynamic such that the position and / or extent of the image changes over time. The absorber 602 may be arranged to form a dark region onto which a portion of an image 402 is projected a majority of the time during operation of the XR device 102.

[0111] Depending on the lighting conditions, portions of the screen 104 which are not aligned with a dark region and / or onto which no image 402 is projected (e.g., an outer region of the screen 104 falling outside of the extent of the image 402) may appear at least semitransparent when viewed from the second side 108 of the screen 104. When viewed from the second side 108 of the screen 104, the image 402 may therefore appear to form part of the environment in which the XR device 102 is situated.

[0112] As was explained above, an XR device 102 of the type described herein may be used to display artificially generated images and / or may be used to display images based on images captured by another device so as to operate as a telepresence device. An example will be described below with reference to FIG. 7 in which the XR device 102 operates as a telepresence device as part of a communication system 702. A further example will be described below with reference to FIG. 9 in which the XR device 102 displays artificially generated images.

[0113] FIG. 7 is a schematic illustration of a communication system 702 comprising a first XR device 704a and a second XR device 704b. Both the first XR device 704a and the second XR device 704b may comprise an XR device 102 as described above with reference to FIG. 1 to FIG. 6. Any of the features and / or components of the XR device 102 described above with reference to FIG. 1 to FIG. 6 may be included in the first XR device 704a and / or the second XR device 704b of FIG. 7.

[0114] The first XR device 704a and the second XR device 704b are communicatively coupled to each other by way of a communication link 708. The communication link 708 may be a dedicated communication link 708 between devices 704a, 704b and / or may be established over a communications network. For example, the communication link 708 may be established over the internet.

[0115] The first XR device 704a and the second XR device 704b may be situated at remote locations (from each other). The communication system 702 may enable communication between a first user 706a situated in proximity to the first XR device 704a and a second user 706b situated in proximity to the second XR device 704b. The first XR device 704a isconfigured to capture images of the first user 706a (using the image sensor 110 of the first XR device 704a) and send the captured images to the second XR device 704b through the communication link 708. Similarly, the second XR device 704b is configured to capture images of the second user 706b (using the image sensor 110 of the second XR device 704b) and send the captured images to the first XR device 704a through the communication link 708.

[0116] The second XR device 704b is configured to generate and project images based on the images received from the first XR device 704a onto the second side 108 of the screen 104 of the second XR device 704b. For example, an image of the first user 706a which is captured by the first XR device 704a may be projected onto the second side 108 of the screen 104 and viewed by the second user 706b. As was explained above, the image of the first user 706a is projected on to the second side 108 of the screen 104 of the second XR device 704b such that an eye of the first user 706a (included in the image) is reflected at a position on the screen 104 which substantially aligns with a viewing position 112 of a camera (of the image sensor 110) on the first side 106 of the screen 104. In this way, if the second user 706b looks directly at the eye included in the image 402 projected onto the screen 104 of the second XR device 704b then they will be looking substantially at the camera located at the viewing position on the first side 106 of the screen 104 of the second XR device 704b. The camera of the second XR device 704b can therefore capture images of the second user 706b in which the second user 706b is looking directly into the camera.

[0117] The first XR device 704a is configured to generate and project images based on the images received from the second XR device 704b onto the second side 108 of the screen 104 of the first XR device 704a. For example, an image of the second user 706b which is captured by the second XR device 704b may be projected onto the second side 108 of the screen 104 and viewed by the first user 706a. As with the second XR device 704b, the image of the second user 706b is projected on to the second side 108 of the screen 104 of the first XR device 704a such that an eye of the second user 706b (included in the image) is reflected at a position on the screen 104 which substantially aligns with a viewing position 112 of a camera (of the image sensor 110) on the first side 106 of the screen 104. In this way, if the first user 706a looks directly at the eye included in the image 402 projected onto the screen 104 of the first XR device 704a then they will be looking substantially at the camera located at the viewing position on the first side 106 of the screen 104 of the first XR device 704a. The camera of the first XR device 704a can therefore capture images of the first user 706a in which the first user 706a is looking directly into the camera.

[0118] As was explained above, the arrangement of the first XR device 704a and the second XR device 704b means that if the first user 706a looks at an eye of the second user 706b in an image 402 of the second user 706b projected on to the first XR device 704a then a camera of the first XR device 704a can capture an image of the first user 706a in which the first user 706a appears to be looking directly at the camera. Consequently, when the captured image of the first user 706a is projected on to the screen 104 of the second XR device 704b, it appears to the second user 706b (who looks at the image projected on to the screen 104 of the second XR device 704b) as though the first user 706a is looking directly at the second user 706b.

[0119] Similarly, if the second user looks at an eye of the first user 706a in an image 402 of the first user 706a projected on to the second XR device 704a than a camera of the second XR device 704b can capture an image of the second user 706b in which the second user 706b appears to be looking directly at the camera. Consequently, when the captured image of the second user 706b is projected on to the screen 104 of the first XR device 704a, it appears to the first user 706a (who looks at the image projected on to the screen 104 of the first XR device 704a) as though the second user 706b is looking directly at the first user 706a.

[0120] A plurality of images of the first user 706a may be captured at the first XR device 704a and sent to the second XR device 704b at a plurality of successive times (so as to capture moving images or video of the first user 706a). These images may be displayed by the second XR device 704b in order and as soon as they are received such that a moving image (video) of the first user 706a is displayed by the second XR device 704b in real time (subject to processing and transmission delays). Similarly, a plurality of images of the second user 706b may be captured by the second XR device 704b and sent to the first XR device 704a at a plurality of successive times (so as to capture moving images or video of the second user 706b). These images may be displayed by the first XR device 704a in order and as soon as they are received such that a moving image (video) of the second user 706b is displayed by the first XR device 704a.

[0121] In this way the communication system 702 may function to facilitate video communication (a video call) between the first user 706a and the second user 706b. The video communication may be supplemented by audio communication (voice call) by similarly capturing audio at each of the first XR device 704a and the second XR device 704b (using the audio sensor 118 of each device) and sending the captured audio to the other of the first XR device 704a and the second XR device 704b. Each of the first XR device 704a and the second XR device 704b may then output audio (using the sound output device 120 of each device) received from the other of the first XR device 704a and the second XR device 704b to enablereal time voice communication between the first user 706a and the second user 706b (subject to processing and transmission delays).

[0122] In examples in which images which are projected onto a screen 104 of an XR device 102 are based on images captured by another device (e.g., in a communication system 702 such as the communication system 702 described above with reference to FIG. 7) an image for projection may be processed in order to determine a position of at least one eye in the image. The control system 116 of each XR device 102 may comprise an image generator (not shown in FIG. 1 - FIG. 7). The image generator may be configured to generate an image for projection onto the screen 104 of the XR device 102 based on an image received from another device. The generated image for projection may comprise the image received from another device. In some examples, the image generator may perform some image processing on an image received from another device in order to generate an image for projection by the image source 114. For example, an image generator may perform image processing (e.g., object detection) to detect a position of at least one eye in each image which is received from another device. The detected position of at least one eye in an image may be used by the image processor and / or the image source 114 to project a corresponding image on to the screen 104 of the XR device 102 such that the eye is reflected from a position on the screen 104 which substantially aligns with a viewing position 112 of a camera on the first side 106 of the screen 104. For example, an image may be resized, rescaled and / or repositioned by the image generator such that the eye is reflected at a position which substantially aligns with a viewing position 112 of a camera. Additionally or alternatively, the detected position of the eye in an image may be provided to the image source 114 (e.g., from the image generator) and the image source 114 may adopt a configuration which causes the eye to be reflected at a position which substantially aligns with a viewing position 112 of a camera.

[0123] In some examples, the image sensor 110 and / or at least one camera which forms part of the image sensor 110 may be moveable. For example, an XR device 102 may comprise an actuator (not shown) operable to move at least a part of the image sensor 110 so as to change the viewing position 112 at which a camera is situated on the first side 106 of the screen 104. The actuator may receive a control signal from the control system 116 identifying the position of an eye in an image 402 to be projected onto the screen 104 of the XR device 102. The actuator may then move at least a camera of the image sensor 110 to a viewing position 112 on the first side 106 of the screen 104, which substantially aligns with a position at which the eye in the image 402 is to be reflected from the screen 104. In this way a position of a camera may be adapted for different images in which an eye appears at a different position in the image.

[0124] In some examples, alignment between a viewing position 112 of a camera and a position at which an eye in an image is reflected may be achieved through one or both of: configuring an image to be projected so that the eye in the image is reflected at a determined reflection location on the screen and configuring the viewing position 112 of a camera to align with the determined reflection location on the screen at which the eye in the image is reflected.

[0125] As was explained above, in the communication system 702 of FIG. 7, the arrangement of the image sensor 110 of each XR device 704a, 704b relative to the position of an eye of a user 706a, 706b in an image projected on to the respective XR device 704a, 704b, gives the impression to each user 706a, 706b that the other user 706a, 706b is looking directly at them. Furthermore, the images which are captured by each camera (of a user) are captured from a perspective at which a user would view the other user if the user were to be present at the location at which their image is projected onto the screen 104. These effects provide a realistic life-like impression of in-person communication with the other user 706a, 706b in a way that other device arrangements cannot provide. Furthermore, this effect is achieved through the arrangement of the components described above and the projection of images so as to align an eye in a projected image with viewing position 112 of an image sensor 110 and does not require extensive post-processing of images. As was described above, some form of image processing may be used to determine a position of an eye in an image and / or to position the image on the screen 104 such that the reflection of the eye is aligned with a viewing position of a camera. However, no image processing may be needed to change a perspective from which a user is viewed since a camera captures the first user 706a from the same perspective as a second user 706b would view the first user 706a, if the second user 706b were positioned as shown in the projected image of the second user 706b. Furthermore, a camera captures the second user 706b from the same perspective as a first user 706a would view the second user 706b if the first user 706a were positioned as shown in the projected image of the first user 706a. The realistic appearance of users 706a, 706b when viewed on an XR device 704a, 704b may be further enhanced by the semi-transparency of the screen 104 since a portion of the environment situated behind an XR device 706a, 706b (e.g., a portion which is viewed through a portion of the screen 104 on which no image is projected) may appear visible to a user 704a, 704b. This may create an impression that the user 706a, 706b whose image appears on the screen 104 is situated in the same location (e.g., the same room) as the user 706a, 706b viewing the screen 104.

[0126] Examples were described above with reference to FIG. 7 in which images are projected onto a screen 104 of an XR device 102, where the images are based on images captured by another device. For example, the first XR device 704a projects images capturedat the second XR device 704b and the second XR device 704b captures images captured at the first XR device 704a. In this way the first XR device 704a and the second XR device 704b device form a communication system 702 which facilitates video communication between a first user 706a and a second user 706b.

[0127] In other examples, artificially generated images 402 may be projected onto a screen 104 of an XR device 102. In such examples, an XR device 102 may display a virtual assistant with which a user 202 may interact. The virtual assistant may perform any suitable role such as a role of a receptionist / host, a medial advisor, a caregiver, a customer service representative, a real estate agent, a tour guide, a trainer, a therapist, a teacher, an instructor, an advisor, a collaborator and / or a personal assistant.

[0128] FIG. 8 is a schematic illustration of the functional components of an XR device 102 which is configured to display artificially generated images. The XR device 102 may comprise an XR device 102 as described above with reference to FIG. 1 to FIG. 6. Any of the features and / or components of the XR device 102 described above with reference to FIG. 1 to FIG. 6 may be included in the XR device 102 of FIG. 8.

[0129] As was explained above, the XR device 102 comprises an image sensor 110, an image source 114, an audio sensor 118 and a sound output device 120. No further detailed description of these components will be provided with reference to FIG. 8 since they were described in detail above. Each of the image sensor 110, image source 114, audio sensor 118 and sound output device 120 may be communicatively coupled to each other and / or the control system 116 as depicted by the solid lines linking components in FIG. 8. Such coupling may be realised in practice by any suitable communication link such as a wired and / or wireless communication link.

[0130] In the example shown in FIG. 8 the control system 116 includes a gesture detector 802, a position detector 804, a language detector 806, a language generator 808 and an image generator 810. Each of these components may be communicatively coupled to each other and / or the image sensor 110, image source 114, audio sensor 118 and / or sound output device 120 as depicted by the solid lines linking components in FIG. 8. The gesture detector 802, position detector 804, language detector 806, language generator 808 and image generator 810 have been depicted as separate functional components, but may be realised by a single electronic device (such as a general purpose computing device) or any suitable combination of devices. All or part of the functionality of the control system 116 may be realised by one or more Artificial Intelligence (Al) and / or machine learning (ML) models.

[0131] The image sensor 110 generates image data indicative of images captured by a camera of the image sensor 110. The image data may, for example, comprise images of a user 202 which is positioned on the second side 108 of the screen 104 and interacting with the XR device 102. The images may include images captured by a plurality of cameras. The images may include depth images captured by a depth sensor. The audio sensor 118 generates audio data indicative of audio captured by the audio sensor 118. The audio data may, for example, comprise sound recordings of the user 202 talking to the XR device 102.

[0132] The image data may be provided to the gesture detector 802 and / or the position detector 804. The gesture detector 802 may detect from the image data gestures performed by a user 202 captured in the image data. Detected gestures may include facial expressions such as smiling, frowning, winking, eye contact etc. Detected gestures may include sentiment and emotion expressed through facial expressions such as happiness, excitement, sadness, anger, fear, confusion, contempt, surprise etc. Detected gestures may include hand gestures such as waving, thumbs up, thumbs down etc. Detected gestures may include sentiment and emotion expressed through body language.

[0133] Gestures may be detected using image processing of images included in the image data. The gesture detector 802 may comprise one or more Al and / or ML models configured through training to detect gestures from captured image data. For example, an Al and / or ML model may be trained through supervised learning on a training data set comprising captured images of a user performing different gestures. The training data set may be labelled with gestures which are being performed by a user 202 in each image so as to enable the Al and / or ML model to learn to detect gestures from captured image data.

[0134] The accuracy of gesture detection may be improved by the positioning of a camera of the image sensor 110 at a viewing position 112 substantially aligned with a reflection of an eye on the screen 104. A user 202 interacting with the XR device 102 may direct gestures toward the image 402 of the animal which is projected onto the screen 104. For example, a user may direct their eye gaze towards an eye of the animal in the image 402. The position of the camera at a viewing position 112 aligned with the reflection of the eye may therefore be optimal for observing and detecting gestures made by the user 202.

[0135] The position detector 804 may detect from the image data a position of a user 202 interacting with the XR device 102. For example, the position detector 804 may detect a position of a user relative to the XR device 102. A detected position may include an angular orientation and / or a distance from the XR device 102. The position detector 804 may perform image processing (e.g., object detection) on the image data to detect a user 202 in imagescaptured by the image sensor 110 and determine the position of the user 202 based on the detection.

[0136] The accuracy and / or usability of position detection may be improved by the positioning of a camera of the image sensor 110 at a viewing position 112 substantially aligned with a reflection of an eye on the screen 104. For example, the detected position of a user 202 may be used to direct gestures and / or an eye gaze of an image of an animal (e.g., human) towards the user 202. This direction of gestures and / or eye gaze may be performed from the perspective of the animal (e.g., human) whose image is projected onto the screen 104. A position which is detected (by the position detector 804) based on images captured by a camera aligned with an eye of the projected image of the animal (e.g., human) is detected from the perspective of the eye of the projected animal. Such detected positions may therefore be directly used for applications such as gesture direction and / or eye gaze without the need for further processing. The alignment of an eye in a projected image with a viewing position of a camera may therefore allow for accurate and realistic gesture and eye gaze direction without the need for complicated further processing of detected positions.

[0137] The position of the camera at a viewing position 112 aligned with the reflection of the eye may therefore be optimal for observing and detecting gestures made by the user 202.

[0138] The audio data may be provided to the language detector 806. The language detector 806 may be configured to determine words spoken by a user 202 and detected in the audio detector. For example, the language detector 806 may be configured to perform speech recognition (e.g., speech-to-text) on the audio data to determine spoken language from the audio data. The output of the language detector 806 may comprise text corresponding to language spoken by the user 202. The output of the language detector 806 may further comprise emotion and / or sentiment detected in speech (for example, based on intonation and / or tone detected in the speech). The language detector may comprise one or more Al and / or ML models configured through training to detect speech, emotion and / or sentiment from captured audio data. For example, an Al and / or ML model may be trained through supervised learning on a training data set comprising captured audio data of a speech of one or more users.

[0139] The language generator 808 is configured to generate speech to be output by the XR device 102 (e.g., by the sound output device 120). The speech may be generated in response to speech and / or gestures of the user 202 and detected by the language detector 806 and / or the gesture detector 802. The language generator 808 may comprise one or more Al and / or ML models. For example, the language generator 808 may comprise a large language model.The large language model may be prompted based on speech and / or gestures of the user 202. The language generator 808 may receive speech detected by the language detector 806 and / or gestures detected by the gesture detector 802 and may generate a prompt to the large language model based on the detected speech and / or gestures. For example, if the detected speech includes a question posed by the user 202, text corresponding to the question may be provided as a prompt to the large language model. Additionally or alternatively, sentiment, emotion and / or gestures detected by the language detector 806 and / or the gesture detector 802 may be used to formulate a prompt to the large language model.

[0140] A large language model may be trained to perform a given role or function. For example, a large language model may be trained to output responses to prompts in the manner of a given roll such as one or more of a receptionist / host, a medial advisor, a caregiver, a customer service representative, a real estate agent, a tour guide, a trainer, a therapist, a teacher, an instructor, an advisor, a collaborator and / or a personal assistant. A large language model may be configured through training based on training data which is specific to a given roll.

[0141] A large language model may output text in response to a prompt. The language detector 806 may perform speech synthesis on the output of the large language model to generate speech to be output by the XR device 102 (e.g., the sound output device 120). The speech synthesis may be configured to output speech in a given voice associated with a given roll or character.

[0142] The image generator 810 is configured to generate images 402 for projection onto the screen 104 of the XR device 102 by the image source 114. The image source 114 may be configured to generate images 402 of a given animal. The animal may comprise a human or other form of animal. In some examples, the animal may comprise an artificially generated character and / or a mythical creature. The animal may be based on a real animal such as a real human. For example, training data may be collected comprising images and / or sound recordings of a given person. One or more aspects of the control system 116 may then be trained based on the training data to replicate the appearance and behavior of the given person. For example, one or more aspects of the language generator 808 and / or image generator 810 may be trained based on the training data.

[0143] The image generator 810 may be configured to generate images 402 based on an output of one or more of the gesture detector 802, position detector 804, language detector 806 and language generator 808. For example, the image generator 810 may use a gesture detected by the gesture detector 802 to generate images of an animal (e.g., human) performinga gesture in response to a detected gesture. In one illustrative example, the gesture detector 802 may detect a user 202 waving at the XR device 102. In response, the image generator 810 may generate images (i.e., a sequence of images which form a moving image or video) of an animal waving back at the user 202.

[0144] The image generator 810 may use a detected position of a user 202 to generate images of an animal (e.g., human) such that gestures performed by the animal and / or an eye gaze of the animal are directed to the position of the user 202. In this way the user 202 is given the impression that the animal is communicating with the user 202. As was described above, a position of a user 202 is detected from the perspective of an eye of a projected image of the animal (e.g., human). Consequently, there may be no need to perform any processing or correction of a determined position for the purposes of generating images of gestures and / or an eye gaze which is directed towards the user 202.

[0145] The image generator 810 may use language detected by the language detector 806 to generate images of an animal (e.g., human) responding to detected language. For example, the image generator 810 may generate images of an animal (e.g., human) performing gestures which respond to language detected by the language detector 806. In one illustrative example, the language detector 806 may detect language which is interpreted (by the language detector 806 and / or the image generator 810) to be shocking. In response, the image generator 810 may generate images of an animal (e.g., human) in which the animal appears to be shocked.

[0146] The image generator 810 may use an output of the language generator 808 to generate images of an animal (e.g., human) which are consistent with the animal speaking the language generated by the language generator 808. For example, the image generator 810 may generate images (e.g., a sequence of images which form a moving image or video) of an animal (e.g., human) in which a mouth of the animal moves in sequence with speech generated by the language generator 808 such that it appears in the images that the animal is speaking the speech. Additionally or alternatively, the image generator 810 may generate images (e.g., a sequence of images which form a moving image or video) of an animal (e.g., human) in which the animal performs a gesture (which may include a facial expression) which is consistent with a sentiment and / or emotion associated with language generated by the language generator 808. For example, if the language generator 808 generates language which is associated with a friendly sentiment then the image generator 810 may generate images of an animal (e.g., human) adopting positive body language and smiling.

[0147] The language generator 808 and the image generator 810 may provide outputs to at least the image source 114 and the sound output device 120. For example, language generatedby the language generator 808 (which may be in the form of speech) may be provided to the sound output device 120 which outputs sound corresponding to the generated language. Images generated by the image generator 810 may be output to the image source 114, which may project the images 402 onto the screen 104 of the XR device 102 as described above. Speech and / or any other sound may be output from the sound output device 120 in synchronisation with images 402 projected by the image source 114 such that the images 402 which appear on the screen 104 are consistent with sound being output from the sound output device 120. For example, a sequence of images 402 may be projected by the image source 114 to form a moving image of an animal (e.g., human) talking in synchronisation with sound output from the sound output device 120 of speech of the animal (e.g., human) such that it appears as though the animal (e.g, human) in the moving image is producing the speech being output by the sound output device 120.

[0148] As was described above, images 402 of an animal (e.g., human) are projected onto the screen 104 such that an eye in the image is reflected from a position on the screen 104 which substantially aligns with a viewing position 112 of a camera on the first side 106 of the screen 104. As was explained above with reference to the communication system 702 of FIG. 7 this alignment may be achieved through one or both of projecting an image 402 such that an eye in the image is reflected at a given reflection position on the screen 104 and aligning a camera such that it is positioned at a viewing position 112 which substantially aligns with the given reflection position.

[0149] The image generator 810 may be configured to generate images of an animal (e.g., human) such that an eye of the animal is reflected from a position on the screen 104 which substantially aligns with a viewing position 112 of a camera on the first side 106 of the screen 104. For example, the image generator 810 may be configured to position an eye in a generated image at a position in the generated image such that when the generated image is projected by the image source 114, the eye is reflected from a position on the screen 104 which substantially aligns with a viewing position 112 of a camera on the first side 106 of the screen 104. Additionally or alternatively, the position of the eye in an image may be provided to the image source 114 (e.g., from the image generator 810) and the image source 114 may adopt a configuration which causes the eye to be reflected at a position which substantially aligns with a viewing position 112 of a camera.

[0150] In some examples, the image sensor 110 and / or at least one camera which forms part of the image sensor 110 may be moveable. For example, the XR device 102 may comprise an actuator (not shown) operable to move at least a part of the image sensor 110 so as to change the viewing position 112 at which a camera is situated on the first side 106 of thescreen 104. The actuator may receive a control signal from the control system 116 (e.g., from the image generator 810) identifying the position of an eye in an image 402 to be projected onto the screen 104 of the XR device 102. The actuator may then move at least a camera of the image sensor 110 to a viewing position 112 on the first side 106 of the screen 104, which substantially aligns with a reflection position at which the eye in the image 402 is to be reflected from the screen 104. In this way a position of a camera may be adapted for different images in which an eye appears at a different position in the image.

[0151] In some examples, alignment between a viewing position 112 of a camera and a position at which an eye in an image is reflected may be achieved through one or both of: configuring an image to be projected so that the eye in the image is reflected at a determined reflection position on the screen and configuring the viewing position 112 of a camera to align with the determined location on the screen at which the eye in the image is reflected.

[0152] The operation of an example control system 116 was described above with reference to FIG. 8 in the context of different functional components (the gesture detector 802, the position detector 804, the language detector 806, the language generator 808 and the image generator 810). In other examples at least some of the functionality described herein may be performed by different functional components to those described above with reference to FIG.8. Additionally or alternatively, the control system 116 may comprise different and / or additional functional components to those described with reference to FIG. 8.

[0153] In the example, of FIG. 8 each of the functional components (the gesture detector 802, the position detector 804, the language detector 806, the language generator 808 and the image generator 810) are realised as part of the same control system 116, which forms part of the XR device 102. However, in some examples all or part of the functionality of the control system 116 may be performed by one or more other devices which may be physically situated as part of the XR device 102. For example, all or part of the functionality of the control system 116 may be performed by one or more remote computing devices which are communicatively coupled to the XR device 102. For example, one or more Al or ML models which realise all or part of the functionality of the control system 116 may be implemented by one or more remote computing devices (e.g., by a server and / or a cloud computing platform). The XR device 102 may be communicatively coupled to one or more remote computing devices through a network connection (which may include a wired and / or wireless connection) such as through an internet connection.

[0154] FIG. 9 is a flow chart of a method 900 of operating an XR device 102. The XR device 102 may be any XR device 102 described herein and may have any of the featuresdescribed above with reference to any of FIG. 1 to FIG. 8. No further detailed description of the XR device 102 will be provided again with reference to FIG. 9.

[0155] At step 902 an image is captured using an image sensor 110 comprising a camera situated at a viewing position 112 on a first side of a semi-transparent and partially reflective screen 104.

[0156] At step 904, an image 402 of an animal including an eye is projected onto a second side 108 of the screen 104. The image 402 is projected such that the image 402 is reflected by the screen 104 so as to appear visible on the screen 104 when viewed from the second side 108 of the screen 104. The image 402 is further projected onto the second side 108 of the screen 104 such that the eye included in the image 402 is reflected at a position on the screen 104 which substantially aligns with a viewing position 112 of the camera of the image sensor 110 on the first side 106 of the screen 104.

[0157] The method may further include any of the steps and features described above with reference to FIG. 1 - FIG. 8. For example, the image 402 may be artificially generated (e.g., to provide a virtual assistant) and / or may be based on at least one image captured by another device (e.g., to form part of a communication system 702).

[0158] Various methods, features and implementations have been described herein in which some of the steps may be implemented by any suitable electronic device (such as a computing device) and / or combination of electronic devices (e.g. computing devices). Furthermore devices, modules, functions and components have been described herein which may be realised using a suitable electronic device (such as a computing device) and / or combination of electronic devices (e.g. computing devices). FIG. 10 is a schematic illustration of an example electronic device 1002 which may be used to implement all or part of any method, feature or implementation described herein and / or to realise all or part of any device, module, function and / or component described herein. For example, a control system 116 (and / or any component of the control system 116) as described herein may be realised at least in part using an electronic device 1002 of the type shown in FIG. 10. An electronic device 1002 of the type shown in FIG. 10 and as described below may be used to implement all or part of a method of operating an extended reality device as described herein.

[0159] The electronic device 1002 may include at least one processing unit 1004, memory 1008 and an input / output interface 1006 (I / O). The processing unit 1004 may include any suitable processor and / or combination of processors. For example, the processing unit 1004 may include one or more of a Central Processing Unit (CPU) and a Graphical Processing Unit (GPU). The memory 1008 may include volatile memory and / or non-volatile / persistentmemory. The memory 1008 may, for example, be used to store data such as an operating system, instructions to be executed by the processing unit (e.g. in the form of software to be executed by the processing unit), configuration information related to the electronic device 1002, session information and / or configuration or registration information associated with any other device, node or module in a network. In some examples, the memory 1008 may be used to store instructions for executing any of the methods and / or steps disclosed herein.

[0160] At least the processing unit 1004 is connected to the input / output interface 1006. The input / output interface 1006 may facilitate communication with one or more other devices. For example, the input / output interface 1006 may be operable to transmit and / or receive communications to / from other devices in a network. The input / output interface 1006 may, for example, comprise one or more antennas to facilitate wireless communication with one or more other devices. Additionally, or alternatively, the input / output interface 1006 may comprise a wired interface for establishing a wired connection with one or more other devices.

[0161] Optionally, the electronic device 1002 may further include a display (not shown). The display may comprise any suitable electronic display such as a touch sensitive display. The display may be connected to at least to the processing unit 1004. The processing unit 1004 may generate display signals which are sent to the display in order to cause the display information.

[0162] Features, integers, characteristics or groups described in conjunction with a particular aspect, embodiment or example of the invention are to be understood to be applicable to any other aspect, embodiment or example described herein unless incompatible therewith. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and / or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and / or steps are mutually exclusive. The invention is not restricted to the details of any foregoing embodiments. The invention extends to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed. In particular, any dependent claims may be combined with any of the independent claims and any of the other dependent claims.

[0163] Each feature disclosed in this specification (including any accompanying claims, abstract and drawings), may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of equivalent or similar features.The invention is not restricted to the details of any foregoing embodiments. The invention extends to any novel one, or any novel combination, of the features disclosed in this specification (including any accompanying claims, abstract and drawings), or to any novel one, or any novel combination, of the steps of any method or process so disclosed. The claims should not be construed to cover merely the foregoing embodiments, but also any embodiments which fall within the scope of the claims.

Claims

CLAIMS1. An extended reality device comprising: a semi-transparent and partially reflective screen; an image sensor comprising a camera situated at a viewing position on a first side of the screen and arranged to capture images of radiation arriving at the image sensor having been transmitted through the screen; an image source configured to emit light so as to project an image on to the second side of the screen, such that the image is reflected by the screen so as to appear visible on the screen when viewed from the second side of the screen, wherein the projected image comprises an image of an animal including an eye and wherein the image source is configured to project the image onto the second side of the screen such that the eye included in the image is reflected at a position on the screen which substantially aligns with the viewing position of the camera of the image sensor on the first side of the screen.

2. The extended reality device of claim 1, wherein the image sensor comprises a depth sensor configured to determine a distance of objects situated on the second side of the screen from the image sensor.

3. The extended reality device of claim 1 or 2, wherein the image sensor comprises a first camera situated at a first viewing position on the first side of the screen and a second camera situated at a second viewing position on the first side of the screen.

4. The extended reality device of claim 3, wherein the image comprises an image of an animal including a first eye and a second eye and wherein the image source is configured to project the image onto the second side of the screen such that the first eye included in the image is reflected at a position on the screen which substantially aligns with the first viewing position of the first camera of the image sensor on the first side of the screen and the second eye included in the image is reflected at a position on the screen which substantially aligns with the second viewing position of the first camera of the image sensor on the second side of the screen.

5. The extended reality device of any one of claims 1 to 4, further comprising an actuator operable to move the image sensor so as to change the viewing position at which the camera is situated on the first side of the screen.

6. The extended reality device of any one of claims 1 to 5, further comprising an absorber arranged around the image sensor on the first side of the screen and configured to absorb visible radiation incident on the absorber.

7. The extended reality device of claim 6, wherein the absorber extends to form a dark region on the first side of the screen, wherein visible radiation transmitted through the screen and incident on the dark region is absorbed by the absorber, and wherein the image source is configured to project the image onto the second side of the screen such that the image is reflected at positions on the screen which substantially align with positions within the dark region on the second side of the screen.

8. The extended reality device of any one of claims 1 to 7, wherein the image source is arranged to project the image along an optical axis and wherein the image source and the screen are arranged relative to each other such that the optical axis is not aligned with a surface normal of the screen.

9. The extended reality device of any one of claims 1 to 8, wherein the image source comprises an electronic display configured to display the image on the electronic display and wherein the electronic display and the screen are arranged relative to each other such that light emitted from the electronic display is incident on the second side of the screen such that the image is reflected by the screen.

10. The extended reality device of any one of claims 1 to 8, wherein the image source comprises an image projector arranged to project the image onto the second side of the screen.

11. The extended reality device of any one of claims 1 to 10, wherein the image sensor is configured to provide images captured by the camera to control system comprising an image generator configured to generate the image to project onto the second side of the screen and wherein the image source is configured to receive the generated image from the image generator and project the generated image on to the second side of the screen.

12. The extended reality device of claim 11, further comprising the control system.

13. The extended reality device of claim 12, wherein the control system is configured to generate an artificial image of the animal for projection onto the second side of the screen, wherein the control system is configured to generate the artificial image based on at least one captured image received from the image sensor.

14. The extended reality device of claim 13, wherein the control system is configured to: determine a position of an object relative to the image sensor based on the at least one image captured by the image sensor; and generate the artificial image such that when the generated image is projected onto the second side of the screen and viewed from the second side of the screen, the eye included in the generated image appears to be directed to the determined position of the object.

15. The extended reality device of claim 13 or 14, wherein the control system is configured to: detect one or more of a gesture of a subject in at least one image received from the image sensor; and generate the artificial image based on the detected gesture.

16. The extended reality device of any one of claims 1 to 15, wherein the extended reality device further comprises a sound output device, wherein the image source is configured to provide images captured by the camera to a control system comprising a language generator, wherein the control system is configured to generate natural language outputs based on at least one image captured by the camera, and wherein the sound output device is configured to output sound corresponding to speech of a natural language output received from the control system.

17. The extended reality device of claim 12, wherein the control system is configured to generate the image of the animal based on an image captured by another device in communication with the control system.

18. A communication system comprising a first extended reality device according to claim 12 and a second extended reality device according to claim 12, wherein: the first extended reality device is configured to send images captured by the image sensor of the first extended reality device to the second extended reality device; the second extended reality device is configured to generate and project images based on the images received from the first extended reality device onto the second side of the screen of the second extended reality device; the second extended reality device is configured to send images captured by the image sensor of the second extended reality device to the first extended reality device; and the first extended reality device is configured to generate and project images based on the images received from the second extended reality device onto the second side of the screen of the first extended reality device.

19. A method of operating an extended reality device, the method comprising: capturing an image using an image sensor comprising a camera situated at a viewing position on a first side of a semi-transparent and partially reflective screen; projecting an image of an animal including an eye onto a second side of the screen such that the image is reflected by the screen so as to appear visible on the screen when viewed from the second side of the screen, wherein the image is projected onto the second side of the screen such that the eye included in the image is reflected at a position on the screen which substantially aligns with a viewing position of the camera of the image sensor on the first side of the screen.

Images