Method and apparatus for extending field of view of augmented reality device

Abstract

Methods and systems for extending a field of view of an augmented reality (AR) device are disclosed. In some aspects, an AR device establishes communication between an AR device having an image sensor and a display device having a display screen, time synchronizes the display device with the AR device, determines a position of the display screen of the display device relative to the AR device, configures content to be presented on the display device, and displays the configured content on the AR device. And transmitting the content to the display device. In some aspects, the AR device determines its position based at least in part on visual information received by the image sensor while ignoring visual information within the display screen of the display device. In some aspects, the display device includes a three-dimensional (3D) display, and the AR device includes at least one lens through which a user may view a 3D image displayed by the 3D display.

Description

Background Art 1. Technical Field

[0001] Aspects of the present disclosure generally relate to augmented reality.

[0002] 2. Description of related technologies

[0003] As used herein, extended reality (XR) is an umbrella term that includes virtual reality (VR), mixed reality (MR), and augmented reality (AR). In VR, a user wears a VR headset that presents a view of the world that is completely computer-generated (the user does not see the real world at all). In MR, the user is seeing a mix of images of the real world seen through a camera and computer-generated images displayed together on the VR headset. In AR, the user sees the real world directly, for example, through a transparent lens in front of the user's eyes, in a device that also displays computer-generated images on a transparent lens.

[0004] XR devices have significant disadvantages. For example, VR headset users cannot see the real world and can harm themselves and others, such as by bumping into furniture, pets, people, or other obstacles while using a VR headset. MR devices present a real-world view, but at a lower resolution and often with some lag due to processing video images. AR devices allow users to see the real world directly, but are limited in what can be displayed to the user. Summary of the invention

[0005] The following presents a simplified summary of the invention related to one or more aspects disclosed herein. Thus, the following summary of the invention should neither be considered as an exhaustive overview related to all conceived aspects, nor should it be considered to identify key or decisive elements related to all conceived aspects or to delineate the scope associated with any particular aspect. Therefore, the sole purpose of the following summary of the invention is to present certain concepts related to one or more aspects of the mechanisms disclosed herein in a brief form before the detailed embodiments presented below.

[0006] In one aspect, a method performed by an augmented reality (AR) device for using a display screen to expand the field of view of the AR device includes: establishing communication between the AR device having an image sensor and a display device having a display screen; synchronizing time between the display device and the AR device; determining a position of the display screen of the display device relative to the AR device; configuring content to be presented on the display device; and transmitting the content to the display device.

[0007] In one aspect, a method performed by a display device having a display screen for using a display screen to expand the field of view of an AR device includes: establishing communication between the display device having a display screen and an AR device having an image sensor; synchronizing the display device with the AR device time; determining a position of the AR device relative to the display device; receiving a request to display at least one object based on the position of the AR device; and displaying at least one object on the display screen based on the position of the AR device.

[0008] In one aspect, an AR device includes: at least one lens through which a user can view a display; a projector for projecting an AR image onto the at least one lens for viewing by the user; an image sensor; a memory; at least one transceiver; and at least one processor, the at least one processor being communicatively coupled to the projector, the memory, and the at least one transceiver, the at least one processor being configured to: establish communication with a display device having a display screen; synchronize the display device with the AR device in time; determine a position of the display screen of the display device relative to the AR device; configure content to be presented on the display device; and transmit the content to the display device.

[0009] In one aspect, a display device includes: a display screen; a memory; at least one transceiver; and at least one processor, the at least one processor being communicatively coupled to the display screen, the memory, and the at least one transceiver, the at least one processor being configured to: establish communication with an AR device; synchronize a timestamp with the AR device; receive a position of the AR device via at least one transceiver; receive a request to display at least one object based on the position of the AR device via at least one transceiver; and display at least one object on the display screen based on the position of the AR device.

[0010] Other objects and advantages associated with the various aspects disclosed herein will be apparent to those skilled in the art based on the drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are presented to aid in describing various aspects of the present disclosure and are provided solely for illustration and not limitation of the various aspects.

[0012] Figure 1 Portions of an augmented reality (AR) headset for using a display screen to extend the field of view of an AR device in accordance with aspects of the present disclosure are illustrated.

[0013] Figure 2 is a block diagram illustrating a system for extending a field of view of an AR device using a display screen according to aspects of the present disclosure.

[0014] Figure 3A and Figure 3Bis a flow chart illustrating portions of a process associated with using a display screen to expand the field of view of an AR device according to aspects of the present disclosure.

[0015] Figure 4 is a flow chart illustrating a process associated with using a display screen to expand the field of view of an AR device according to aspects of the present disclosure.

[0016] Figure 5 is a flow chart illustrating a portion of a process associated with using a display screen to expand the field of view of an AR device according to aspects of the present disclosure.

[0017] Figure 6 is a flow chart illustrating another portion of a process associated with using a display screen to expand the field of view of an AR device according to aspects of the present disclosure.

[0018] Figure 7 A visual example of using a display screen to expand the field of view of an AR device according to aspects of the present disclosure is illustrated.

[0019] Figure 8 is a flow chart of an example process associated with using a display screen to expand the field of view of an AR device according to aspects of the present disclosure.

[0020] Fig. 9 is a flow chart of an example process associated with using a display screen to expand the field of view of an AR device according to aspects of the present disclosure. DETAILED DESCRIPTION

[0021] Methods and systems for extending the field of view of an augmented reality (AR) device are disclosed. In some aspects, the AR device establishes communication between the AR device having an image sensor and a display device having a display screen, synchronizes the display device with the AR device time, determines the position of the display screen of the display device relative to the AR device, configures content to be presented on the display device, and transmits the content to the display device. In some aspects, the AR device determines its position based at least in part on visual information received by the image sensor while ignoring visual information within the display screen of the display device. In some aspects, the display device includes a three-dimensional (3D) display, and the AR device includes at least one lens through which a user can view a 3D image displayed by the 3D display.

[0022] These technologies include combining an AR headset with 3D TV glasses so that a user can watch through an AR display to view a working 3D TV. As used herein, the term "on-working" display refers to an XR-enabled display that is turned on and is displaying or ready to display something on the display screen. In some aspects, a timestamp or other synchronization signal is used to synchronize all the working screens in the room, for example, so that all the working 3D screens in the room display the left eye image simultaneously, and then all the working 3D screens in the room display the right eye image simultaneously, and so on. Multiple screens can be combined to form a larger display, such as a TV wall. In the examples described below, the term "display" will be assumed to refer to a working display unless otherwise explicitly identified.

[0023] Various aspects of the present disclosure are provided in the following description and related drawings for various examples provided for illustrative purposes. Alternative aspects may be designed without departing from the scope of the present disclosure. Additionally, well-known elements of the present disclosure will not be described in detail or will be omitted to avoid making the relevant details of the present disclosure difficult to understand.

[0024] The words "exemplary" and / or "example" are used herein to mean "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" and / or "example" is not necessarily to be construed as preferred or superior to other aspects. Likewise, the term "aspects of the disclosure" does not require that all aspects of the disclosure include the discussed feature, advantage, or mode of operation.

[0025] Those skilled in the art will appreciate that any of a variety of different techniques and methods may be used to represent the information and signals described below. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the following description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof, which depends in part on the specific application, in part on the desired design, in part on the corresponding technology, and so on.

[0026] In addition, many aspects are described in terms of a sequence of actions to be performed by, for example, an element of a computing device. It should be understood that the various actions described herein may be performed by a specific circuit (e.g., an application specific integrated circuit (ASIC)), by a program instruction executed by one or more processors, or by a combination of the two. Additionally, the sequence of actions described herein may be considered to be fully embodied in any form of non-transient computer-readable storage medium, in which a corresponding computer instruction set is stored, which, when executed, will cause or command the associated processor of the device to perform the functionality described herein. Therefore, various aspects of the present disclosure may be embodied in a variety of different forms, all of which have been contemplated to be within the scope of the claimed subject matter. In addition, for each of the various aspects described herein, the corresponding form of any such aspect may be described herein as, for example, "a logical component configured to perform the described action".

[0027] Three-dimensional (3D) television or 3DTV is a technology that displays a simulated 3D world to the view of the device, for example, by providing two different images to each eye of the viewer to produce parallax to simulate the placement of the displayed objects in the 3D field. However, 3DTV programs are not interactive, that is, the content displayed on the 3DTV does not change based on the position of the viewer relative to the screen. Instead, 3DTV displays images that will be seen from a fixed point, for example, from the viewpoint of a person who is centered in front of the screen and looking straight instead of at an angle. This is necessary, in part, because 3DTV programs can be viewed by multiple viewers at the same time, and 3DTV cannot provide a unique view for each individual viewer at each viewer's individual position, so the displayed content is created under the assumption of a single, front-center viewer. Therefore, conventional 3DTV does not need or care to know the position and orientation of the viewer, does not include any ability to determine this information, and is therefore not suitable for (cannot be used for) XR applications. As described above, AR devices provide a direct view of the real world, but have a limited field of view (e.g., about 50 degrees), which is much smaller than the human field of view (about 190 degrees left and right). Furthermore, AR headsets cannot create entirely new scenes or landscapes, but can only overlay images or information on top of existing scenes or landscapes.

[0028] To overcome this shortcoming of AR devices, methods and systems for extending AR using 3D displays such as those used by 3DTVs (referred to herein as "3D displays") are presented herein. By coupling an AR headset with a 3D display, the AR experience is greatly enhanced. For example, a 3D display can provide a very beautiful, very detailed 3D scene directly to the user through the glasses of the AR headset, i.e., the virtual image is generated by the 3D display rather than by the AR headset. As the end user walks around, the view of the 3D scene presented to the user will change, giving the illusion that the user is looking through the 3D display, as if it were a window into another world. When the user approaches the 3D display, more of the virtual world is displayed on the screen of the 3D display; when the user leaves the 3D display, less of the virtual world is displayed on the screen of the 3D display; when the user walks past the 3D display, the view of the virtual world displayed to the user on the screen of the 3D display changes; all of this is just like the user walking towards, away from, or past an actual window.

[0029] Coupling an AR device with a 3D display can also enrich the user's experience in other ways. For example, in some aspects, the user can pull an object from the 3D world presented by the 3D display into the AR glasses to view the details, and push the object back into the 3D world for storage, select another object to pull out, etc.

[0030] Figure 1 1 illustrates portions of an AR headset 100 for extending the field of view of an AR device using a display screen according to aspects of the present disclosure. Figure 1 In the example shown in , the AR headset 100 includes at least one projector 102 that projects an AR image onto a pair of lenses 104, which are made of glass, plastic, or some other transparent material so that the viewer has an unobstructed view of the real world. The lenses 104 are angled so as to reflect the AR image into the user's eyes. The number of lenses in the AR headset 100 is illustrative and not limiting. In an alternative aspect, the projector 102 may project the AR image onto different portions of a single lens, which may include, for example, an optical waveguide. Likewise, the AR headset 100 may have more than two lenses.

[0031] The two lenses 104 provide different views of the AR image to each eye, for example, because each lens 104 reflects the image from its respective projector 102, such as Figure 1 As shown, or because each lens 104 reflects a different portion of an image provided by a single projector, the single projector can be mounted, for example, above the two eyes (this embodiment is not shown). Figure 1In some aspects, the different images provided to each eye by one or more projectors 102 make the projected images appear to be three-dimensional. The transparent lens allows the user to see the image on the display screen directly. As will be explained in detail below, the display screen can be used to expand the field of view of the AR device.

[0032] In some aspects, lens 104 also provides two different views of the image displayed by each of the one or more 3D display screens. The different views provided to each eye by the 3D display screens make the image within the 3D display appear to be three-dimensional. Therefore, in some aspects, the user will see the real world image, the 3D image within each 3D display screen, and optionally another 3D image generated by one or more projectors 102.

[0033] In some aspects, the lenses are polarized lenses, where the lens for one eye has a different polarization orientation than the lens for the other eye. This type of lens is used for a 3D display that projects two different images simultaneously, each image at a different polarization orientation than the other image, so that one image is seen by one eye and the other image is seen by the other eye. Because the two images produced by the 3D display and seen by each eye are slightly different from each other, the image on the 3D display appears to the user as a 3D image. This only applies to images displayed by the 3D display - real-world images, and any images produced by projector 102 are not affected by polarization.

[0034] In some aspects, the lens is a liquid crystal display (LCD) lens operated as an electronic light shutter, where the two lenses are alternately opened, for example, when one of the lenses is transparent, the other lens is opaque, and vice versa. This type of lens is used for a 3D display that projects two different images in a time domain multiplexed manner, for example, alternating between the two images in time. Because the alternating images produced by the 3D display and seen by each eye are slightly different from each other, the image on the 3D display appears to the user as a 3D image. This only applies to the image displayed by the 3D display - the real world image, and any image produced by the projector 102 is not affected by the alternating shutter.

[0035] exist Figure 1 In the illustrated example, a single set of lenses 104 is used both as a surface on which an AR image is displayed and as a polarizing lens or light shutter for use with a 3D display. However, in another aspect, a first set of lenses can be used as a surface on which an AR image is displayed, while a second set of lenses is a polarizing lens or light shutter required for a 3D display. Having two sets of lenses allows each set to have a size, position, and orientation that is best suited to a particular need, i.e., as a reflective surface on which an AR image is displayed or as a lens required for a 3D display.

[0036] exist Figure 1 In the example shown in , the AR headset 100 includes a controller 106 that controls the operation of the projector 102, and may include a wireless transceiver 108 for communicating via WiFi, Bluetooth, cellular, or other wireless communication links.

[0037] exist Figure 1 In the example shown in , the AR headset 100 includes an image sensor 110 (eg, a camera) that captures views of the local environment and can use those images to determine the position and orientation of the AR headset 100 .

[0038] exist Figure 1 In the example shown in , the AR headset 100 includes an inertial measurement unit (IMU) 112, which provides inertial data to a controller 106, which can use the IMU data to predict the position and / or orientation of the AR headset 100 as the user moves around in the real world.

[0039] Figure 2 is a block diagram illustrating an AR system 200 for extending the field of view of an AR device using a display screen according to aspects of the present disclosure. Figure 2 In the example illustrated in , the AR headset 100 is communicatively coupled to one or more display devices 202, each having one or more display screens 203, via one or more controllers, such as controller 204, for example. In some aspects, the one or more display devices 202 may include at least one 3D display for displaying paired images to simulate a 3D image. In some aspects, the controller 204 may be part of the AR headset 100. In some aspects, the controller 204 may be part of one or more of the display devices 202. In some aspects, the controller 204 may be separate from the AR headset 100 and the one or more display devices 202. For example, the controller 204 may be a set-top box (STB), a media server, a streaming platform, a game console, a personal computer (PC), or other computing source, or may be part of a set-top box (STB), a media server, a streaming platform, a game console, a personal computer (PC), or other computing source. In Figure 2 In the illustrated example, the display device 202 includes its own controller 206 and wireless transceiver 208. The AR headset 100, one or more display devices 202 and their respective controllers 206, and the controller 204 can communicate with each other via a wired or wireless connection. In some aspects, the AR headset 100 can communicate with multiple 3D screens, each with a dedicated controller, where the headset controller and the 3D screen controller act as a cloud to decide / compute the view to be displayed by the 3D screen and / or the AR headset 100.

[0040] In some aspects, the AR headset 100 communicates exclusively with the controller 204, and the controller 204 handles the exchange of information and data between the AR headset 100 and each of the one or more display devices 202. In some aspects, the AR headset 100 may communicate directly with the one or more display devices 202 as well as with the controller 204.

[0041] In some aspects, the AR headset 100 is a system master, e.g., communicating instructions to the controller 204 and optionally to one or more display devices 202. In some aspects, the controller 204 operates as a system master, e.g., communicating instructions to the AR headset 100 and to one or more display devices 202.

[0042] It should be noted that any of the wireless communication links described herein may alternatively be a wired communication link, and vice versa. For example, the AR headset 100 may be connected to the controller 204 via a wireless communication link, and the controller 204 may be connected to the display device 202 via a wired communication link, and so on. Examples of wireless communication links include, but are not limited to, WiFi, cellular, Bluetooth, etc. In some aspects, the AR headset 100 and / or the controller 204 may be connected to multiple devices of each type. For example, the controller 204 may be connected to the AR headset 100 and multiple display devices 202.

[0043] Figure 3A and Figure 3B is a flow chart illustrating portions of a process 300 associated with using a display screen to extend the field of view of an AR device according to aspects of the present disclosure. In some implementations, Figure 3A and Figure 3B One or more blocks of may be executed by one or more components within the AR system 200 (e.g., the AR headset 100, the display device 202, the controller 204, etc.). In some specific implementations, Figure 3A and Figure 3B One or more blocks of may be performed by another device or a group of devices separate from the AR system 200.

[0044] like Figure 3AAs shown, process 300 may include detecting and wirelessly connecting to a display device in a room or area at block 302. For example, the AR headset 100 may join a WiFi or other wireless network (which may be hosted by the controller 204 or by a wireless router or some other device) and determine that the display device 202 is also on the wireless network, for example, by issuing a general query and detecting a response from the display device 202, by receiving a notification message broadcast by the display device 202, etc. Likewise, the AR headset 100 may connect directly to the display device 202 in a peer-to-peer manner rather than via a WiFi network (e.g., via Bluetooth or other wireless connection). In some aspects, the display device may be a 3D display that displays paired images to simulate a 3D image.

[0045] like Figure 3A As shown, the process 300 may further include synchronizing the display device with the AR device at box 304. For example, the AR headset 100 may exchange timestamp messages with the display device 202, the controller 204, or both. Once synchronized, the AR headset 100 is synchronized with the display device. This synchronization includes, but is not limited to, synchronizing a light shutter within the AR headset 100 with an alternating view presented by a 3D display. In some aspects, the display device 202 may display a calibration sequence that includes a timestamp displayed or encoded within an image, and the AR headset 100 receives (e.g., via a camera, image sensor, light sensor, etc.), decodes the timestamp, and uses it to synchronize with the display device 202.

[0046] like Figure 3A As shown, process 300 may also include detecting a screen of a display device and marking the position of the display device and the size of the optional display device at block 306. For example, display device 202 may present a test pattern, a black screen, or other visual display on its screen that the AR headset 100 may detect and identify as display screen 203 of display device 202, for example, using its image sensor 110. As the user of the AR headset 100 moves, detected changes in the size, shape, aspect ratio, etc. of the display screen 203 may be used to calculate the relative position of the display device 202 relative to the AR headset 100. In some aspects, the AR headset 100 may receive information about the size of the display screen 203, information about the position of the display device 202, other information about the display device 202, or some combination thereof from the display device 202 and / or the controller 204, and the AR headset 100 may use this information to detect and identify the display screen 203.

[0047] like Figure 3AAs shown, process 300 may include configuring content to be presented on a display screen of a display device at block 308. For example, the AR headset 100 may generate a synthetic scene to be displayed by the display device such that the synthetic scene is visible to a user of the AR headset 100 through one or more transparent lenses of the AR headset 100. Examples of synthetic scenes may include, but are not limited to, 3D scenery, such as such that a screen of the display device appears as a window through which the 3D scenery may be viewed.

[0048] like Figure 3A As shown, process 30 may include transmitting the content to a display device at block 310. For example, AR headset 100 may transmit the content to display device 202 via a wireless connection therebetween.

[0049] like Figure 3B As shown, process 300 may optionally include determining the position of the AR device at box 312. In some aspects, the AR device may use wireless RF technology, such as using an ultra-wideband (UWB) positioning method, or receive an indication of its position from another entity. In some aspects, the AR device may derive a relative position from an absolute position. In some aspects, the AR device may determine its position based at least in part on visual information received by an image sensor (which may occur during calibration), such as a virtual simultaneous localization and mapping (vSLAM) positioning method, while ignoring visual information within the display screen 203 of the display device. This is because the image being displayed on the display screen 203 will be synthetic and should not be used to calculate the position or orientation of the AR headset 100; however, only images of real objects (e.g., nothing displayed by the display screen 203) should be used for this calculation. In some aspects, determining the position of the AR device includes performing a six-degree-of-freedom (6DOF) calculation.

[0050] like Figure 3BAs shown, process 300 may optionally include reporting the position of the AR headset 100 to the display device 202 at block 314. For example, after the AR headset 100 has performed its 6DOF calculations to determine its position and orientation relative to the room, the AR headset 100 or the controller 204 may report the position of the AR headset 100 to the display device 202. In some aspects, the display device 202 is only informed of the position of the AR headset 100 in the room. Using this information, the display device 202 may calculate a view to be displayed to the user of the AR headset 100 via the display screen 203. In some aspects, the display device 202 is also informed of the orientation of the AR headset 100 relative to the room and / or the display screen 203. For example, this allows the display device 202 to stop displaying an image (and thus save power) if the user of the AR headset 100 is facing away from the display screen 203 and cannot see the image anyway, and to start displaying an image when the user of the AR headset 100 can see the display screen 203.

[0051] like Figure 3B As shown, process 300 may optionally include requesting the display device to display or stop displaying one or more objects based on the position of the AR headset at block 316. In some aspects, the AR headset 100 indicates to the display device 202 that the user is facing the display screen 203, which may be considered a request to display a previous scene, some default scene, or other scene. In some aspects, the user of the AR headset 100 may request the display device 202 to display a particular view or scene, for example, by selecting an option using a controller, by speaking a command, by performing a gesture, or by performing some other agreed action recognized by the AR system 200.

[0052] like Figure 3BAs shown, process 300 may further include reporting position updates (and optionally orientation updates) to display device 202, reporting interactions with objects being displayed by display screen 203, or some combination thereof, at block 318. For example, as the user of AR headset 100 moves around the physical room in which display device 202 is positioned, display device 202 is updated with the new position of AR headset 100 and will adjust the image displayed on display screen 203 accordingly to match the way that the image of the real-world scene viewed through the window will change as the user's position relative to the window changes. In another example, display device 202 may start by not displaying anything at all until AR headset 100 requests display device 202 to start displaying something (e.g., a scene, object, or a combination thereof). When the display device 202 is not displaying anything, the AR headset 100 may continue to report its position to the display device 202 so that when the AR headset 100 requests the display device 202 to display something, the display device 202 already knows the position of the AR headset 100 and can display the requested scene or object without delay.

[0053] Likewise, the user of the AR headset 100 may interact with objects displayed by the display device 202, e.g., by selecting options using a controller, by speaking a command, by performing a gesture, by performing some other agreed action recognized by the AR system 200, etc. For example, if the display screen 203 is presenting one or more virtual objects on a virtual shelf, and the user selects a virtual object for closer inspection (e.g., by making a grab gesture detected by the AR headset 100, pressing a button on a controller connected to the AR headset 100, selecting a drop-down menu displayed by the AR headset 100, etc.), the AR headset 100 may notify the display device 202 of this interaction. In response, the display device 202 may change the image being displayed on the display screen 203, e.g., changing the shape and / or color of the object, highlighting the object, zooming in on the selected object, moving the selected object off the display screen 203 and displaying the selected object using the AR headset 100 instead, etc. In some aspects, a virtual object being displayed on the AR headset 100 may be moved to the display screen 203 so that the display screen 203 displays the object as seen by the user through the transparent lenses of the AR headset 100, rather than having the AR headset 100 generate an image of the virtual object and display it on the lenses (e.g., using the projector 102). Examples of virtual objects include, but are not limited to, pictures, videos, vector graphics, and 3D models, such as those written in Unity, DirectX, or OpenGL and described by shape, texture, lighting, etc. For example, when transferring the 3D model back and forth from the AR headset 100 to the display device 202, the AR headset 100 may present the 3D model in a position and pose relative to the display device 202 or relative to the AR headset 100, and may also present the 3D model as if it were illuminated by the real-world environment of the user of the AR headset 100. The AR headset 100 may also modify other properties of the 3D model, such as size, texture, lighting, resolution, etc.

[0054] In the simplified process 300 shown in FIG. 3 , blocks 316 and 318 are continuously repeated until the AR system is turned off or the particular operation is terminated. However, it should be understood that in some aspects, the detection and use of additional display screens may occur in parallel, and the operations of blocks 302 to 318 may be repeated for each of the additional display screens detected. In some aspects, the AR system 200 will update object information, AR headset position, etc. with timestamps, sensor data, frame order, shutter signals, etc. through a virtual wireless connection channel. In various aspects where there are multiple display screens, the multiple display screens may be synchronized to the same time base so that all display screens will display the left eye image at the same time, and then all display screens will display the right eye image at the same time, regardless of where each display screen is positioned relative to the AR headset. For example, in some aspects, the display screens will all use the same vertical synchronization (VSYNC) signal or the same VSYNC signal timing.

[0055] Figure 4 is a flow chart illustrating another process 400 associated with using a display screen to expand the field of view of an AR device in accordance with aspects of the present disclosure. Figure 4 An example interaction between an AR headset and a display device is illustrated. In some aspects, the display device may be a 3D display device, such as a 3D TV. In some implementations, Figure 4 One or more blocks of may be executed by one or more components within the AR system 200 (e.g., the AR headset 100, the display device 202, the controller 204, etc.). In some specific implementations, Figure 4 One or more blocks of may be performed by another device or a group of devices separate from the AR system 200.

[0056] exist Figure 4 In the example shown, at block 402, the AR headset is connected to the display device, and at block 404, the display device is connected to the AR headset. When these two blocks are completed, the AR headset and the display device have established a communication link between them (event 406).

[0057] exist Figure 4 In the example shown in , at block 408, the AR headset transmits the objects to the display device for display by the display device, and at block 410, the display device receives the objects. The AR headset may create a visual scene containing one or more objects (event 412), which the display device will display toward the user of the AR headset. If the display device is a 3D display, the AR headset may create a 3D scene.

[0058] From the AR headset side, the AR headset enters a loop that may include one or more tasks. In some aspects, the AR headset may occasionally (e.g., at a low frequency) perform timestamp synchronization (box 414) with a display device (box 416), for example, using a timestamp synchronization signal or message (event 418). For example, in the case where both the crystal oscillator (XO) of the AR headset and the XO of the display device have an accuracy within 50 parts per million (ppm), a crystal marked as 1 MHz may actually have 999950 to 1000050 cycles per second. In a use case where timestamp synchronization accuracy must be within 1 ms, timestamp synchronization operations should occur every 10 seconds (1 ms / 100 ppm) or more frequently.

[0059] Other tasks that the AR headset may perform during the loop include, but are not limited to, updating its position (block 420), updating local objects (e.g., objects that the AR headset will project onto its lenses) (block 422), and updating remote objects (e.g., objects that the display device will display on its screen) (block 424). If the AR headset is interacting with a "shutter" type 3D display, it may optionally provide a shutter sync signal to the 3D display (block 426). The AR headset then displays the (possibly updated) AR frame (block 428).

[0060] From the display device side, the display device enters a loop that may include one or more tasks (box 416) in addition to low-frequency timestamp synchronization, such as obtaining a refined updated position from the AR headset (box 430) and obtaining updated remote objects from the AR headset (box 432). The 3D TV then displays the (possibly updated) display frame (box 434). If the display is a "shutter" type 3D TV, it may use the shutter sync signal from the AR headset to synchronize alternating frames so that the image on the 3D TV is correctly seen by the user of the AR headset.

[0061] Figure 5 is a flow chart illustrating another process 500 associated with using a display screen to expand the field of view of an AR device in accordance with aspects of the present disclosure. Figure 5 Shown in more detail Figure 4 In some implementations, Figure 5 One or more blocks of may be executed by one or more components within the AR system 200 (e.g., the AR headset 100, the display device 202, the controller 204, etc.). In some specific implementations, Figure 5 One or more blocks of may be performed by another device or a group of devices separate from the AR system 200.

[0062] exist Figure 5In the example process 500 shown in FIG. 5 , the display device waits for a connection from an AR headset (block 502), while the AR headset attempts to establish a connection (block 504). After making a connection (event 505), the display device is connected to the AR headset (block 506) and the AR headset is connected to the display device (block 508). The display device synchronizes its timestamp with the AR headset (block 510), and the AR headset synchronizes its timestamp with the display device (block 512). At this point, the AR headset and display device are time synchronized (event 513).

[0063] The display device then waits for the AR headset to determine the position of the display device (block 514). The AR headset begins determining the position of the display device (block 516). Figure 5 In the example shown, the display device reports its position and screen size to the AR headset (block 518), and the AR headset uses this information (event 519) to obtain the position of the display device (block 520).

[0064] Both the display device and the AR headset check to see if the TV position is calibrated (box 522 and box 524, respectively). If not, the display device displays a calibration image (box 526), ​​which the AR headset uses to calibrate the position of the display device (box 528). The source of the calibration image can be the AR headset or the display device. If the calibration image is provided by the AR headset, the AR headset should transmit the image to the display device; if the calibration image is provided by the display device, the display device should transmit the image or corner point information (e.g., the position and descriptor of the corner point) to the AR headset (event 529). In some aspects, the AR headset then saves the position and size of the display device (box 530). In some aspects, the AR headset reports the position of the display device to the display device (event 531). In some aspects, the display device then saves its position for later use (box 532). At the end of the process, the display device position is calibrated and the AR headset can be connected to the display device more quickly in the future. The AR headset may then use the position and size of the display device screen to exclude the on-screen image when calculating the position of the AR headset.

[0065] Figure 6 is a flow chart illustrating another process 600 associated with using a display screen to expand the field of view of an AR device in accordance with aspects of the present disclosure. Figure 6 Shown in more detail Figure 4 In some implementations, Figure 6One or more blocks of may be executed by one or more components within the AR system 200 (e.g., the AR headset 100, the display device 202, the controller 204, etc.). In some specific implementations, Figure 6 One or more blocks of may be performed by another device or a group of devices separate from the AR system 200.

[0066] like Figure 6 As shown, process 600 may include estimating an AR headset position and / or orientation based on previous AR headset positions / orientations and inertial measurement unit (IMU) data at block 602. For example, the AR headset 100 may estimate its own position (and optionally, its own orientation) based on its previous position and its own IMU data.

[0067] like Figure 6 As shown, process 600 may further include re-projecting (re-predicting) a display device screen position and / or orientation based on the estimated AR headset position and / or orientation at block 604. For example, the AR headset 100 may calculate a position of the display device 202 relative to a current position of the AR headset 100.

[0068] like Figure 6 As shown, process 600 may further include estimating the AR headset position based on the scene as viewed by the AR headset camera but excluding any images that may be currently displayed by the display device at box 606. This allows traditional, vision-based positioning estimates to be performed without having images on the display device potentially confounding the calculations. In some aspects, this may involve estimating the position of occluded images, i.e., the AR headset may attempt to determine the exact location of the 3D screen relative to the current position and orientation of the AR headset, and / or occluding the camera image within the display device screen position, i.e., ignoring portions of the AR headset's viewpoint that appear within the display device's screen. In this way, when the AR headset is trying to determine its position in the room based on visual cues, the AR headset ignores anything that is currently being displayed by the display device.

[0069] Figure 7 An example 700 of using a display screen to expand the field of view of an AR device according to aspects of the present disclosure is illustrated. Figure 7 In the example shown in , the views seen by the user of the AR headset are shown, namely, a left eye view 702 and a right eye view 704. In this example, the user is looking at a room containing a chair 706 and a 3D screen 708.

[0070] exist Figure 7In the example shown, the 3D screen 708 is displaying a 3D landscape containing a single tree. The position of the tree within the frame of the 3D screen is slightly different for each eye, giving the illusion that the user is looking out of a real window at a real tree in the close distance. To this end, the 3D screen displays two different views, one for the left eye and the other for the right eye. This can be done, for example, by displaying two views simultaneously, each polarized at a different angle to match the corresponding polarization angle of each lens of the AR headset, or by alternating the display of the two images in synchronization with the LCD shutter on each lens of the AR headset so that only each eye sees the view intended for that eye. As the user moves around the physical room, the 3D image displayed by the 3D screen will change to reflect the new view that will be seen by the real user viewing the real tree through the real window. In order to know what the view should be, the 3D screen receives continuous updates about the position of the AR headset in the room. If the user turns his or her head, but otherwise stays in the same position, the image on the 3D screen 708 will not need to change, except perhaps for small adjustments to the slight change in the position of the user's eyes.

[0071] exist Figure 7 In the example shown in , the AR headset is generating an AR image of object 710. To do this, the AR headset projects the image onto the lenses of the AR headset so that the image is reflected into the eyes of the user wearing the AR headset. The AR images presented to each eye are slightly different from each other, giving the illusion that object 710 is physically present in the room. Figure 7 , object 710 appears closer to 3D screen 708 in the right eye view than in the left eye view, just as real-world objects do. If the user turns his or her head but otherwise stays in the same position, the AR image of object 710 projected onto the lens would have to change to give the illusion that object 710 remains in the same position in the room, unless of course object 710 is intended to remain in a constant position relative to the user rather than relative to the room.

[0072] Figure 8 is a flowchart of an example process performed by an AR device associated with using a display screen to expand the field of view of the AR device according to aspects of the present disclosure. Figure 8 One or more process blocks of may be performed by an AR device (eg, AR headset 100, controller 204). In some implementations, Figure 8 One or more process blocks of may be performed by another device or a group of devices separate from or including the AR device. Additionally or alternatively, Figure 8One or more process blocks of process 800 may be performed by one or more components of a device (such as a processor, memory, transceiver), any or all of which may be means for performing the operations of process 800.

[0073] like Figure 8 As shown, process 800 may include establishing communication between the AR device and the display device having a display screen at block 810. The means for performing the operations of block 810 may include a processor, a memory, or a transceiver of any of the devices described herein. For example, the AR headset 100 may be connected to at least one display device 202 having a display screen 203 using the wireless transceiver 108.

[0074] like Figure 8 As further shown, process 800 may include synchronizing the display device with the AR device time at block 820. Means for performing the operations of block 820 may include a processor, memory, or transceiver of any of the devices described herein. For example, AR headset 100 may use wireless transceiver 108 to synchronize the timestamp with each of the display devices in operation.

[0075] like Figure 8 As further shown in FIG. 8 , process 800 may include determining a position of a display screen relative to an AR device at block 830. In some aspects, the AR headset 100 may also determine a size of a display screen of the display device. A component for performing the operations of block 830 may include a processor, a memory, or a transceiver of any of the devices described herein. For example, the AR headset 100 may use an image captured by the image sensor 110 to determine a position and / or size of a display screen 203 of each of the at least one display device 202.

[0076] like Figure 8 As further shown in FIG. 8 , process 800 may include configuring content to be presented on a display screen of a display device at block 840. Means for performing the operations of block 840 may include a processor, a memory, or a transceiver of any of the devices described herein. For example, the AR headset 100 may use the controller 106 to configure content to be presented on a display screen 203 of the display device 202. In some aspects, the content is configured based at least in part on a position of the display device 202 relative to the AR headset 100.

[0077] like Figure 8As further shown, process 800 may include transmitting the content to a display device at block 850. Means for performing the operations of block 850 may include a processor, memory, or transceiver of any of the devices described herein. For example, the AR headset 100 may use the wireless transceiver 108 to transmit the content to the display device 202.

[0078] In some aspects, establishing communication between the AR device and the display device includes establishing communication between the AR device and multiple display devices, and wherein the multiple display devices operate as a large unified display to provide a single view, as separate displays showing different views, or a combination thereof.

[0079] In some aspects, process 800 may also include transmitting to the display device a time synchronization signal for synchronizing the timing of an image being displayed by the display device with the operation of the AR device.

[0080] In some aspects, process 800 includes determining a location of an AR device, reporting the location of the AR device to a display device, and requesting the display device to display at least one object based on the location of the AR device.

[0081] In some aspects, determining the position of the AR device includes at least one of: determining the position using radio frequency positioning technology, or determining the size of a display screen of a display device, and determining the position of the AR device based at least in part on visual information received by an image sensor while ignoring visual information within the display screen of the display device.

[0082] In some aspects, determining the position of the AR device includes performing six degrees of freedom (6DOF) calculations.

[0083] In some aspects, process 800 includes reporting to a display device a user interaction with at least one of at least one object being displayed by the display device.

[0084] In some aspects, process 800 includes transmitting an object being displayed by a display device to an AR device and displaying the object as an AR image, or transmitting an object being displayed by an AR device as an AR image to the display device and ceasing to display the object as an AR image.

[0085] In some aspects, the AR device includes at least one lens through which a user can view a three-dimensional (3D) display for displaying a pair of images to simulate a 3D image, and wherein the display device includes the 3D display.

[0086] Process 800 may include additional implementations, such as any single implementation or any combination of implementations described below and / or in conjunction with one or more other processes described elsewhere herein. Figure 8 An example block diagram of process 800 is shown, but in some implementations, process 800 may include Figure 8 The process 800 may include more boxes, fewer boxes, different boxes, or boxes arranged in a different manner than those depicted in the process 800. Additionally or alternatively, two or more boxes in the boxes of process 800 may be executed in parallel.

[0087] Fig. 9 is a flow chart of an example process performed by a display device associated with using a display screen of the display device to extend the field of view of an augmented reality (AR) device in accordance with aspects of the present disclosure. Fig. 9 One or more process blocks of may be performed by a display device (eg, display device 202). In some implementations, Fig. 9 One or more process blocks of may be performed by another device or a group of devices separate from or including the display device. Additionally or alternatively, Fig. 9 One or more process blocks of process 900 may be performed by one or more components of a device (such as a processor, memory, transceiver), any or all of which may be means for performing the operations of process 900.

[0088] like Fig. 9 As shown, process 900 may include establishing communication between a display device and an AR device having an image sensor at block 910. The means for performing the operations of block 910 may include a processor, a memory, or a transceiver of any of the devices described herein. For example, display device 202 may be connected to AR headset 100 using wireless transceiver 208.

[0089] like Fig. 9 As further shown, process 900 may include synchronizing the display device with the AR device time at block 920. Means for performing the operations of block 920 may include a processor, memory, or transceiver of any of the devices described herein. For example, display device 202 may synchronize the timestamp with AR headset 100 using wireless transceiver 208.

[0090] like Fig. 9 As further shown in FIG. 9 , process 900 may include determining a position of the AR device relative to the display device at block 930. Means for performing the operations of block 930 may include a processor, a memory, or a transceiver of any of the devices described herein. For example, display device 202 may receive the position of AR headset 100 using wireless transceiver 208.

[0091] like Fig. 9As further shown in FIG. 9 , process 900 may include receiving a request to display at least one object based on the location of the AR device at block 940. Means for performing the operations of block 940 may include a processor, a memory, or a transceiver of any of the devices described herein. For example, display device 202 may receive a request to display at least one object based on the location of AR headset 100 using wireless transceiver 208.

[0092] like Fig. 9 As further shown in FIG. 9 , process 900 may include displaying at least one object based on the position of the AR device at block 950. Means for performing the operations of block 950 may include a processor, a memory, or a transceiver of any of the devices described herein. For example, display device 202 may display at least one object based on the position of AR headset 100, using controller 206 to calculate the position and location of the object within the user's field of view. In aspects where display device 202 is a 3D display, controller 206 may calculate a pair of images required to display the 3D object from the perspective of the current position of AR headset 100.

[0093] In some aspects, a display device includes a three-dimensional (3D) display for displaying a pair of images to simulate a 3D image.

[0094] In some aspects, determining the position of the AR device relative to the display device includes receiving information from the AR device indicating the position of the AR device relative to the display device.

[0095] In some aspects, process 900 includes receiving a report of a user interaction with at least one of at least one object displayed by a display device, and modifying display of at least one of the at least one object being displayed by the display device based on the user interaction.

[0096] In some aspects, modifying the display of at least one of at least one object being displayed by a display device based on user interaction includes at least one of the following: transmitting the object being displayed by the display device to the AR device and stopping display of the object by the display device; or transmitting the object being displayed by the AR device as an AR image to the display device and displaying the object by the display device.

[0097] In some aspects, process 900 includes receiving a time synchronization signal from the AR device and synchronizing the timing of the image being displayed by the display device according to the time synchronization signal. For example, if the display device is a light shutter type 3D display, the time multiplexed image must be displayed in synchronization with the LCD shutter lens of the AR device.

[0098] In some aspects, process 900 includes connecting to at least one other display device and operating the at least one other display device as a large unified display to provide a single view, as a separate display operating different views, or a combination thereof. For example, multiple display devices can be connected together to become a large screen, such as a TV wall, or the screens can be placed individually around a room, etc. Regardless of whether the display devices are grouped or separated, each screen can present the same virtual world, a completely different virtual world, or a combination thereof. When the display devices are grouped, each screen can display its corresponding portion of a larger display including the combined area of ​​the grouped display devices, which can provide a larger apparent depth of field.

[0099] Process 900 may include additional implementations, such as any single implementation or any combination of implementations described below and / or in conjunction with one or more other processes described elsewhere herein. Fig. 9 An example block diagram of process 900 is shown, but in some implementations, process 900 may include Fig. 9 The process 900 may include more boxes, fewer boxes, different boxes, or boxes arranged in a different manner than those depicted in the process 900. Additionally or alternatively, two or more boxes in the boxes of process 900 may be executed in parallel.

[0100] Note that the above examples involve AR devices with cameras or other types of image sensors, but the same principles can be applied to AR devices without image sensors. Such AR devices can benefit from operations that do not require image sensors on the AR headset itself, including but not limited to using a display screen to extend the field of view of the AR device, which can be performed by an AR headset without an image sensor, where the position and orientation of the AR headset can be determined without using the field of view provided by an image sensor mounted to the AR headset itself.

[0101] Now some example use cases are given. The following example use cases are illustrative and not limiting. They are intended to illustrate various uses of the capabilities of the AR system 200.

[0102] Example Use Case 1 A chief fashion designer may need to select a dress, hat, and shoes from a large selection. In his or her office, there are several 3D screens: the first 3D screen displays the dress, the second 3D screen displays the hat, the third 3D screen displays the shoes, and the fourth 3D screen is a "display case" to display the selected dress, hat, and shoes.

[0103] The designer views a virtual fashion model displayed by the AR headset 100 (e.g., projected onto a lens by a projector on the AR headset itself), and the designer can also see the 3D screen through the AR headset 100; the 3D images and scenes displayed on the 3D screen are viewed through the transparent lenses of the AR headset 100, which may be polarized to different angles or may be a light shutter, depending on the type of 3D screen used.

[0104] To help the designer select dresses, the first 3D screen can show a group of dresses on a virtual rack or hanger in rows or columns, which can be zoomed in for a closer look at a smaller number of dresses at a higher resolution or zoomed out to view a larger number of dresses at a lower resolution. Using the rack structure, the designer can place clothes on the racks and stack the racks. The designer can virtually move the racks and rearrange the contents of the racks, and the 3D image presented by the first 3D screen will be updated accordingly. When the designer moves around the first 3D screen, the 3D image on the first 3D screen changes accordingly, for example, to mimic how the view of real clothes on real racks will change when the designer moves around the room.

[0105] In some aspects, the designer can pick a dress and "pull" it closer to see it in more detail (e.g., using a gesture). In some aspects, when a dress is picked from a virtual rack displayed by a first 3D screen, the image of the dress disappears from the first 3D image and is instead generated by the AR headset. By doing so, the dress can be viewed anywhere the designer is viewing through the AR headset, rather than being limited to the area of ​​the first 3D screen. When displayed by the AR headset, the designer can make adjustments to the dress, such as size, length, style, pattern, color, etc., and the AR headset will display the modified dress. In this example, if the designer does not like the dress, the designer can "push back" the dress, such as removing the dress from the AR display and displaying it on the first 3D screen instead. If the designer is satisfied with the dress, the designer can use it to dress the virtual fashion model being displayed by the AR headset. The designer can then walk around the virtual fashion model and see how the dress looks.

[0106] The same action can then be used to virtually try on different hats from a second 3D screen and virtually try on different shoes from a third 3D screen, adjusting size, style, color, accessories, etc., until the virtual fashion model wears a combination of dress, hat, and shoes that satisfies the designer. The designer can then store the combination of dress, hat, and shoes to a fourth 3D screen that operates as a virtual showcase. During this process, details of the object being considered can be transferred between the 3D screen and the AR headset.

[0107] Example Use Case 2 can create an immersive 3D environment for a user of an AR headset using a set of 3D screens that work in coordination to present an extended 3D virtual world. In this use case, all of the 3D screens in the 3D screen appear as windows into the same virtual world, but from different perspectives, just as each window in a window library will show a slightly different view of the real world outside. In one specific implementation, all of the 3D screens in the 3D screen receive a description of the same 3D scene, but each 3D screen only presents the objects that the user will see based on the relative position of the AR headset user to that particular 3D screen.

[0108] For example, the 3D scene could be a garden: as the AR headset moves, the 3D objects displayed by the 3D screen would move in the opposite direction, giving the illusion that the AR headset user is moving through a real window into the real world. The images generated by the 3D screen need not be static: in the garden example, the trees displayed by the 3D screen might sway in the wind, birds might fly between branches, or sing, etc. In some aspects, multiple 3D screens may be grouped to form a large 3D screen, with each 3D screen presenting a view corresponding to its position relative to the AR headset, just as the panes of glasses in a window would show different views of the real world outside.

[0109] The systems and methods disclosed herein are not limited to using only one 3D screen, but can be extended to using multiple 3D screens. Since the virtual image on the 3D screen is generated by the 3D screen and not the AR headset, adding more 3D screens does not increase the rendering overhead of the AR headset, because each 3D screen is handling that aspect. The AR headset will need to know the position of each 3D screen relative to itself in real space, but handling each additional 3D screen is a relatively small incremental processing cost.

[0110] The technology described herein for using a display screen to expand the field of view of an AR device has multiple technical benefits. For example, unlike AR displays that traditionally have a relatively narrow field of view (FOV) (e.g., 90 to 100 degrees horizontally), a display screen can be used to place virtual objects or scenes that can appear anywhere in the user's FOV (typically 180 to 190 degrees horizontally). Another advantage is that the display screen is not power constrained and can produce very bright and realistic images. Another advantage is that existing AR headsets only need to add polarized glasses or electronic shutter lenses to work with 3D displays. Polarized lenses have the following advantages: they are cheap and easy to implement because they do not require any synchronization signals to or from the 3D screen. Electronic shutter glasses have the following advantages: the light seen through the lens will be normal (non-polarized), and when the glasses are powered off, they are still transparent, and the resolution of the electronic shutter glasses will be twice that of the polarized lens.

[0111] In the above specific embodiments, it can be seen that different features are grouped together in each example. This disclosure should not be understood as an intention that the example clauses have more features than the features explicitly mentioned in each clause. On the contrary, various aspects of the present disclosure may include less than all the features of the disclosed individual example clauses. Therefore, the following clauses should be considered to be incorporated into the description accordingly, where each clause itself can be used as a separate example. Although each subordinate clause can refer to a specific combination of one clause with other clauses in a clause, the aspects of the subordinate clause are not limited to specific combinations. It should be understood that other example clauses may also include a combination of the subject matter of the subordinate clause aspect with any other subordinate clause or independent clause or any feature with other subordinate clauses and independent clauses. Various aspects disclosed herein explicitly include these combinations, unless it is clearly expressed or can be easily inferred that a specific combination is not intended to be used (for example, contradictory aspects, such as defining an element as both an electrical insulator and an electrical conductor). In addition, it is also expected that various aspects of the clause can be included in any other independent clause, even if the clause is not directly subordinate to the independent clause.

[0112] Specific implementation examples are described in the following numbered clauses:

[0113] Clause 1. A method for extending the field of view of an augmented reality (AR) device having an image sensor, the method comprising: establishing communication between the AR device and a display device having a display screen; synchronizing time between the display device and the AR device; determining a position of the display screen of the display device relative to the AR device; configuring content to be presented on the display screen of the display device; and transmitting the content to the display device.

[0114] Clause 2. According to the method described in Clause 1, establishing communication between the AR device and the display device includes establishing communication between the AR device and multiple display devices, and wherein the multiple display devices operate as a large unified display to provide a single view, as separate displays displaying different views, or a combination thereof.

[0115] Clause 3. The method of any one of clauses 1 to 2, further comprising: providing a time synchronization signal to the display device for synchronizing the timing of an image being displayed by the display device with the operation of the AR device.

[0116] Clause 4. According to the method described in any one of Clauses 1 to 3, the method also includes: determining the position of the AR device; reporting the position of the AR device to the display device; and requesting the display device to display at least one object based on the position of the AR device.

[0117] Clause 5. A method according to Clause 4, wherein determining the position of the AR device includes at least one of the following: using radio frequency positioning technology to determine the position; or determining the size of the display screen of the display device and determining the position of the AR device based at least in part on visual information received by the image sensor, while ignoring visual information within the display screen of the display device.

[0118] Clause 6. The method of any one of clauses 4 to 5, wherein determining the position of the AR device comprises performing a six degree of freedom (6DOF) calculation.

[0119] Clause 7. The method of any one of clauses 4 to 6, further comprising: reporting to the display device a user interaction with at least one of the at least one object being displayed by the display device.

[0120] Clause 8. The method according to Clause 7 further includes performing at least one of the following in response to the user interaction: transmitting the object being displayed by the display device to the AR device and displaying the object as an AR image; or transmitting the object being displayed by the AR device as an AR image to the display device, and stopping displaying the object as an AR image.

[0121] Clause 9. A method according to any one of clauses 1 to 8, wherein the AR device includes at least one lens, through which the user can view a three-dimensional (3D) display for displaying paired images to simulate a 3D image, and wherein the display device includes a 3D display.

[0122] Clause 10. A method for extending the field of view of an augmented reality (AR) device, performed by a display device having a display screen, the method comprising: establishing communication between the display device and an AR device having an image sensor; synchronizing time between the display device and the AR device; determining a position of the AR device relative to the display device; receiving a request to display at least one object based on the position of the AR device; and displaying the at least one object on the display screen based on the position of the AR device.

[0123] Clause 11. The method of Clause 10, wherein the display device comprises a three-dimensional (3D) display for displaying a pair of images to simulate a 3D image.

[0124] Clause 12. The method of any one of clauses 10 to 11, wherein determining the position of the AR device relative to the display device comprises receiving information from the AR device indicating the position of the AR device relative to the display device.

[0125] Clause 13. A method according to any one of clauses 10 to 12, the method further comprising: receiving a report of a user interaction with at least one of the at least one objects being displayed by the display device; and modifying the display of the at least one of the at least one objects being displayed by the display device based on the user interaction.

[0126] Clause 14. A method according to Clause 13, wherein modifying the display of at least one of the at least one objects being displayed by the display device based on the user interaction includes at least one of the following: transmitting the object being displayed by the display device to the AR device, and stopping the display of the object by the display device; or transmitting the object being displayed by the AR device as an AR image to the display device, and displaying the object by the display device.

[0127] Clause 15. The method of any one of clauses 10 to 14, further comprising: receiving a time synchronization signal from the AR device; and synchronizing the timing of an image being displayed by the display device according to the time synchronization signal.

[0128] Clause 16. A method according to any one of clauses 10 to 15, the method also comprising connecting to at least one other display device and operating the at least one other display device as a large unified display to provide a single view, as separate displays displaying different views, or a combination thereof.

[0129] Clause 17. An augmented reality (AR) device, comprising: at least one lens through which a user can view a display; a projector for projecting an AR image onto the at least one lens for viewing by the user; an image sensor; a memory; at least one transceiver; and at least one processor, the at least one processor being communicatively coupled to the projector, the memory, and the at least one transceiver, the at least one processor being configured to: establish communication with a display device having a display screen; time synchronize the display device with the AR device; determine a position of the display screen of the display device relative to the AR device; configure content to be presented on the display device; and transmit the content to the display device.

[0130] Clause 18. An AR device according to clause 17, wherein the at least one processor is configured to connect to multiple display devices, wherein the multiple display devices operate as a large unified display to provide a single view, as separate displays showing different views, or a combination thereof.

[0131] Clause 19. An AR device according to any one of clauses 17 to 18, wherein the at least one processor is further configured to provide a time synchronization signal to the display device for synchronizing the timing of an image being displayed by the display device with the operation of the AR device.

[0132] Clause 20. An AR device according to any one of clauses 17 to 19, wherein the at least one processor is further configured to: determine the position of the AR device; report the position of the AR device to the display device; and request the display device to display at least one object based on the position of the AR device.

[0133] Clause 21. An AR device according to Clause 20, wherein, in order to determine the position of the AR device, the at least one processor is also configured to perform at least one of the following: determine the position using radio frequency positioning technology, or determine the size of the display screen of the display device and determine the position of the AR device at least in part based on visual information received by the image sensor, while ignoring visual information within the display screen of the display device.

[0134] Clause 22. The AR device of any one of clauses 20 to 21, wherein to determine the position of the AR device, the at least one processor is configured to perform six degrees of freedom (6DOF) calculations.

[0135] Clause 23. The AR device of any one of clauses 20 to 22, wherein the at least one processor is further configured to: report to the display device a user interaction with at least one of the at least one object being displayed by the display device.

[0136] Clause 24. An AR device according to Clause 23, wherein the at least one processor is further configured to: perform at least one of the following in response to the user interaction: transmit the object being displayed by the display device to the AR device and display the object as an AR image; or transmit the object being displayed by the AR device as an AR image to the display device and stop displaying the object as an AR image.

[0137] Clause 25. An AR device according to any one of clauses 17 to 24, wherein the at least one lens includes at least one lens through which a user can view a three-dimensional (3D) display for displaying a pair of images to simulate a 3D image, and wherein the display device includes a 3D display.

[0138] Clause 26. A display device, comprising: a display screen; a memory; at least one transceiver; and at least one processor, the at least one processor being communicatively coupled to the display screen, the memory, and the at least one transceiver, the at least one processor being configured to: establish communication with an augmented reality (AR) device; synchronize a timestamp with the AR device; receive a position of the AR device via the at least one transceiver; receive a request to display at least one object based on the position of the AR device via the at least one transceiver; and display the at least one object on the display screen based on the position of the AR device.

[0139] Clause 27. A display device according to Clause 26, wherein the at least one processor is further configured to: receive, via the at least one transceiver, a report of a user interaction with at least one of the at least one objects being displayed by the display device; and modify the display of the at least one of the at least one objects being displayed by the display device based on the user interaction.

[0140] Clause 28. A display device according to Clause 27, wherein, in order to modify the display of at least one of the at least one objects being displayed by the display device based on the user interaction, the at least one processor is configured to: transmit the object being displayed by the display device to the AR device, and stop displaying the object by the display device; or transmit the object being displayed by the AR device as an AR image to the display device, and display the object by the display device.

[0141] Clause 29. A display device according to any one of clauses 26 to 28, wherein the display screen includes a three-dimensional (3D) display screen for displaying a pair of images to simulate a 3D image, wherein the 3D display screen displays a first image of a pair of images at a first polarization angle and displays a second image of the pair of images at a second polarization angle different from the first polarization angle, or alternately displays a first image of a pair of images and then displays the second image of the pair of images in a time multiplexed manner according to a time synchronization signal received from the AR device.

[0142] Clause 30. A display device according to any one of clauses 26 to 29, wherein the at least one processor is also configured to connect to at least one other display device and operate together with the at least one other display device as a large unified display to provide a single view, as separate displays displaying different views, or a combination thereof.

[0143] Clause 31. An apparatus, comprising: a memory; a transceiver; and a processor, the processor being communicatively coupled to the memory and the transceiver, the memory, the transceiver, and the processor being configured to perform the method of any one of clauses 1 to 16.

[0144] Clause 32. An apparatus comprising means for performing the method of any one of clauses 1 to 16.

[0145] Clause 33. A non-transitory computer-readable medium storing computer-executable instructions, the computer-executable instructions comprising at least one instruction for causing a computer or a processor to perform the method according to any one of clauses 1 to 16.

[0146] It should be understood by those skilled in the art that information and signals may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or optical particles, or any combination thereof.

[0147] In addition, it will be appreciated by those skilled in the art that the various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described in terms of their functionality. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. Those skilled in the art can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0148] The various illustrative logic blocks, modules, and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using a general purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0149] The methods, sequences and / or algorithms described in conjunction with the various aspects disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An example storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integral with the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal (e.g., UE). In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0150] In one or more example aspects, the described functions can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on a computer-readable medium or sent via a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, which include any media that facilitates the transmission of a computer program from one place to another. Storage media can be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, disk storage devices or other magnetic storage devices, or any other media that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if the software is sent from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer readable media.

[0151] Although the foregoing disclosure shows the exemplary aspects of the present disclosure, it should be noted that various changes and modifications may be made herein without departing from the scope of the present disclosure as defined by the appended claims. In addition, the functions, steps and / or actions of the method claims according to the various aspects of the present disclosure as described herein do not need to be performed in any particular order. In addition, although the elements of the present disclosure may be described or claimed in the singular, plural forms may also be considered unless explicitly stated to be limited to the singular.

Claims

1. A method for extending a field of view of an augmented reality (AR) device having an image sensor, the method comprising: Establishing communication between the AR device and a display device having a display screen; Synchronize the time of the display device with the AR device; Determining a position of the display screen of the display device relative to the AR device; configuring content to be presented on the display screen of the display device; as well as The content is transmitted to the display device.

2. According to the method of claim 1, establishing communication between the AR device and the display device includes establishing communication between the AR device and multiple display devices, and wherein the multiple display devices operate as a large unified display to provide a single view, as separate displays displaying different views, or a combination thereof.

3. The method according to claim 1, further comprising: A time synchronization signal for synchronizing the timing of an image being displayed by the display device with the operation of the AR device is provided to the display device.

4. The method according to claim 1, further comprising: Determining a location of the AR device; Reporting the position of the AR device to the display device; as well as The display device is requested to display at least one object according to the position of the AR device.

5. The method of claim 4, wherein determining the position of the AR device comprises at least one of: Use radio frequency positioning technology to determine said location; or A size of the display screen of the display device is determined and the position of the AR device is determined based at least in part on visual information received by the image sensor while ignoring visual information within the display screen of the display device. 6 . The method of claim 4 , wherein determining the position of the AR device comprises performing six degrees of freedom (6DOF) calculations.

7. The method according to claim 4, further comprising: User interactions with at least one of the at least one object being displayed by the display device are reported to the display device.

8. The method of claim 7, further comprising, in response to the user interaction, performing at least one of the following: transmitting the object being displayed by the display device to the AR device, and displaying the object as an AR image; or The object being displayed by the AR device is transmitted to the display device as an AR image, and the display of the object as the AR image is stopped.

9. The method of claim 1 , wherein the AR device comprises at least one lens through which a user of the AR device can view a three-dimensional (3D) display for displaying a pair of images to simulate a 3D image, and wherein the display device comprises a 3D display.

10. A method for extending a field of view of an augmented reality (AR) device, performed by a display device having a display screen, the method comprising: Establishing communication between the display device and an AR device having an image sensor; Synchronize the time of the display device with the AR device; Determining a position of the AR device relative to the display device; receiving a request to display at least one object according to the position of the AR device; as well as The at least one object is displayed on the display screen according to the position of the AR device.

11. The method of claim 10, wherein the display device comprises a three-dimensional (3D) display for displaying the pair of images to simulate a 3D image. 12 . The method of claim 10 , wherein determining the position of the AR device relative to the display device comprises receiving information from the AR device indicating the position of the AR device relative to the display device.

13. The method according to claim 10, further comprising: receiving a report of a user interaction with at least one of the at least one object being displayed by the display device; as well as The display of the at least one of the at least one object being displayed by the display device is modified according to the user interaction.

14. The method of claim 13, wherein modifying the display of the at least one of the at least one object being displayed by the display device according to the user interaction comprises at least one of: transmitting the object being displayed by the display device to the AR device, and stopping the display device from displaying the object; or The object being displayed by the AR device is transmitted to the display device as an AR image, and the object is displayed by the display device.

15. The method according to claim 10, further comprising: receiving a time synchronization signal from the AR device; as well as The timing of the image being displayed by the display device is synchronized according to the time synchronization signal.

16. The method of claim 10, further comprising connecting to at least one other display device and operating the at least one other display device as a large unified display to provide a single view, as separate displays showing different views, or a combination thereof.

17. An augmented reality (AR) device, comprising: at least one lens through which a user can view the display; a projector, the projector being used to project an AR image onto the at least one lens for viewing by the user; Image sensor; Memory; at least one transceiver; as well as at least one processor, the at least one processor being communicatively coupled to the projector, the memory, and the at least one transceiver, the at least one processor being configured to: establishing communication with a display device having a display screen; Synchronize the time of the display device with the AR device; Determining a position of the display screen of the display device relative to the AR device; configuring content to be presented on the display device; as well as The content is transmitted to the display device.

18. The AR device of claim 17, wherein the at least one processor is configured to connect to a plurality of display devices, wherein the plurality of display devices operate as a large unified display to provide a single view, as separate displays displaying different views, or a combination thereof.

19. The AR device of claim 17, wherein the at least one processor is further configured to provide a time synchronization signal to the display device for synchronizing the timing of an image being displayed by the display device with an operation of the AR device.

20. The AR device of claim 17, wherein the at least one processor is further configured to: Determining a location of the AR device; reporting the position of the AR device to the display device; and The display device is requested to display at least one object according to the position of the AR device.

21. An AR device according to claim 20, wherein in order to determine the position of the AR device, the at least one processor is further configured to perform at least one of the following: determine the position using radio frequency positioning technology; or determine the size of the display screen of the display device and determine the position of the AR device at least in part based on visual information received by the image sensor, while ignoring visual information within the display screen of the display device.

22. The AR device of claim 20, wherein to determine the position of the AR device, the at least one processor is configured to perform six degrees of freedom (6DOF) calculations.

23. The AR device of claim 20, wherein the at least one processor is further configured to: User interactions with at least one of the at least one object being displayed by the display device are reported to the display device.

24. The AR device of claim 23, wherein the at least one processor is further configured to: in response to the user interaction, perform at least one of the following: transmitting the object being displayed by the display device to the AR device, and displaying the object as an AR image; or The object being displayed by the AR device is transmitted to the display device as an AR image, and the display of the object as the AR image is stopped.

25. The AR device of claim 17, wherein the at least one lens comprises at least one lens through which a user can view a three-dimensional (3D) display for displaying a pair of images to simulate a 3D image, and wherein the display device comprises a 3D display.

26. A display device, comprising: Display screen; Memory; at least one transceiver; and at least one processor, the at least one processor being communicatively coupled to the display screen, the memory, and the at least one transceiver, the at least one processor being configured to: Establish communication with augmented reality (AR) devices; Synchronizing a timestamp with the AR device; receiving, via the at least one transceiver, a position of the AR device; receiving, via the at least one transceiver, a request to display at least one object based on the location of the AR device; as well as The at least one object is displayed on the display screen according to the position of the AR device.

27. The display device according to claim 26, wherein the at least one processor is further configured to: receiving, via the at least one transceiver, a report of a user interaction with at least one of the at least one object being displayed by the display device; and The display of the at least one of the at least one object being displayed by the display device is modified according to the user interaction.

28. The display device of claim 27, wherein to modify the display of the at least one of the at least one object being displayed by the display device in accordance with the user interaction, the at least one processor is configured to: transmitting the object being displayed by the display device to the AR device, and stopping the display device from displaying the object; or The object being displayed by the AR device is transmitted to the display device as an AR image, and the object is displayed by the display device.

29. A display device according to claim 26, wherein the display screen includes a three-dimensional (3D) display screen for displaying a pair of images to simulate a 3D image, wherein the 3D display screen displays a first image of a pair of images at a first polarization angle and displays a second image of the pair of images at a second polarization angle different from the first polarization angle, or alternately displays the first image of a pair of images and then displays the second image of the pair of images in a time multiplexed manner according to a time synchronization signal received from the AR device.

30. A display device according to claim 26, wherein the at least one processor is also configured to connect to at least one other display device and operate together with the at least one other display device as a large unified display to provide a single view, as separate displays showing different views, or a combination thereof.

Images