Dynamic alignment between perspective camera and eye viewpoint in video perspective (VST) augmented reality (XR)

Abstract

A method includes determining that an inter-pupil distance (IPD) between display lenses of a video perspective (VST) augmented reality (XR) device has been adjusted relative to a default IPD. The method further includes obtaining an image captured using a perspective camera of the VST XR device. The perspective camera is configured to capture an image of a three-dimensional (3D) scene. The method further includes transforming the image according to a change in the IPD relative to the default IPD to match a viewpoint of a corresponding one of the display lenses to generate a transformed image. The method further includes correcting distortion in the transformed image based on one or more lens distortion coefficients corresponding to the change in the IPD to generate a corrected image. Further, the method includes initiating presentation of the corrected image on a display panel of the VST XR device.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to extended reality (XR) systems and processes. More specifically, the present disclosure relates to dynamic alignment between a see-through camera and an eye viewpoint in video see-through (VST) XR. BACKGROUND

[0002] Extended reality (XR) systems are becoming increasingly popular over time, and many applications have been and are being developed for XR systems. Some XR systems, such as augmented reality or “AR” systems and mixed reality or “MR” systems, can augment a user’s view of his or her current environment by overlaying digital content, such as information or virtual objects, on the user’s view of the current environment. For example, some XR systems can generally seamlessly blend virtual objects generated by computer graphics with real-world scenes. SUMMARY

[0003] TECHNICAL PROBLEM SOLUTION

[0004] The present disclosure relates to dynamic alignment between a see-through camera and an eye viewpoint in video see-through (VST) extended reality (XR).

[0005] In a first embodiment, a method includes determining, using at least one processing device, that an interpupillary distance (IPD) between a left display lens and a right display lens of a VST XR device has been adjusted relative to a default IPD of the VST XR device. The method further includes obtaining, using the at least one processing device, an image captured using a see-through camera of the VST XR device, wherein the see-through camera is configured to capture images of a three-dimensional (3D) scene. The method further includes transforming, using the at least one processing device, the image to match a viewpoint of a corresponding one of the display lenses in accordance with the change in the IPD relative to the default IPD, so as to generate a transformed image. The method further includes correcting, using the at least one processing device, distortion in the transformed image based on one or more lens distortion coefficients corresponding to the change in the IPD, so as to generate a corrected image. In addition, the method includes initiating, using the at least one processing device, presentation of the corrected image on a display panel of the VST XR device.

[0006] In a second embodiment, a VST XR device includes left and right perspective cameras configured to capture images of a 3D scene, a display panel configured to present virtual images, and left and right display lenses. The VST XR device further includes at least one processing device configured to determine that an IPD between the left and right display lenses has been adjusted relative to a default IPD of the VST XR device. The at least one processing device is further configured to obtain a designated image of the images captured using a designated one of the perspective cameras. The at least one processing device is further configured to transform the designated image to match a viewpoint of a corresponding one of the display lenses according to the change in the IPD relative to the default IPD, so as to generate a transformed image. The at least one processing device is further configured to correct for distortion in the transformed image based on one or more lens distortion coefficients corresponding to the change in the IPD, so as to generate a corrected image. Moreover, the at least one processing device is configured to initiate presentation of the corrected image on the display panel.

[0007] In a third embodiment, a non-transitory machine-readable medium contains instructions that, when executed, cause at least one processor of a VST XR device to determine that an IPD between left and right display lenses of the VST XR device has been adjusted relative to a default IPD of the VST XR device. The non-transitory machine-readable medium further contains instructions that, when executed, cause the at least one processor to obtain images captured using perspective cameras of the VST XR device, wherein the perspective cameras are configured to capture images of a 3D scene. The non-transitory machine-readable medium further contains instructions that, when executed, cause the at least one processor to transform the images to match a viewpoint of a corresponding one of the display lenses according to the change in the IPD relative to the default IPD, so as to generate transformed images. The non-transitory machine-readable medium further contains instructions that, when executed, cause the at least one processor to correct for distortion in the transformed images based on one or more lens distortion coefficients corresponding to the change in the IPD, so as to generate corrected images. Moreover, the non-transitory machine-readable medium contains instructions that, when executed, cause the at least one processor to initiate presentation of the corrected images on a display panel of the VST XR device.

[0008] Other technical features can be readily apparent to one skilled in the art from the following figures, descriptions, and claims.

[0009] Before undertaking a detailed description of the foregoing, it can be advantageous to set forth definitions of certain words and phrases that have been used throughout this patent document. The terms “transmit,” “receive,” and “communicate,” and variations thereof, encompass both direct and indirect communication. The terms “include” and “comprise,” as well as variations thereof, mean “including but not limited to.” The term “or” is inclusive, meaning and / or. The phrase “associated with,” and variations thereof, means includes, is included in, interconnects with, contains, is contained within, connects to or with, couples to or with, is communicable with, cooperates with, interleave, is proximate to, is bound to or with, has a property of, has a relationship to or with, or the like.

[0010] Moreover, various functions described below can be implemented or supported by one or more computer programs, each of which is formed from computer readable program code and embodied in a computer readable medium. The terms “application” and “program” refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, related data, or a portion thereof applicable for implementation. The phrase “computer readable program code” includes any type of computer code, including source code, object code, and executable code. The phrase “computer readable medium” includes any type of medium capable of being accessed by a computer, such as read only memory (ROM), random access memory (RAM), a hard disk drive, a compact disc (CD), a digital video disc (DVD), or any other type of memory. A “non-transitory” computer readable medium excludes wired, wireless, optical, or other communication links. The non-transitory computer readable medium includes media where the data is permanently stored and media where data is stored and later overwritten, such as a rewritable optical disc or an erasable memory device.

[0011] As used herein, the terms and phrases such as "have," "has," "may have," "include," or "may include" a feature (like a number, a function, an operation, or a component such as a part) indicate the presence of the feature, and do not exclude the presence of other features. Also, as used herein, the phrases "A or B," "at least one of A and / or B," or "one or more of A and / or B" can include all possible combinations of A and B. For example, "A or B," "at least one of A and B," and "at least one of A or B" can indicate all of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B. Also, as used herein, the terms "first" and "second" can modify various components regardless of importance and do not limit the components. The terms are used only to distinguish a component from another component with a same or similar name. For example, a first user device and a second user device can indicate different user devices from each other regardless of sequence or importance of the devices. A first component can be denoted as a second component, and vice versa, without departing from the scope of the disclosure.

[0012] It should be understood that when an element (such as a first element) is referred to as being "coupled with / connected to" or "coupled / connected to" another element (such as a second element) (operatively or communicatively), it can be directly coupled with / connected to, or coupled / connected to, the other element with / without intervening elements (such as a third element). In contrast, it should be understood that when an element (such as a first element) is referred to as being "directly coupled with / connected to" or "directly coupled / connected to" another element (such as a second element), there are no intervening elements (such as a third element) between the element and the other element.

[0013] As used herein, the phrase "configured (or set) to" can be interchangeably used with phrases "suitable for," "have the capacity to," "designed to," "adapted to," "made to," or "capable of" depending on circumstances, as long as the phrase "configured (or set) to" does not mean "specifically designed in hardware to." Rather, the phrase "configured (or set) to" can mean that a device can perform an operation with or without another device or component. For example, the phrase "processor configured (or set) to execute A, B, and C" can mean a general-purpose processor (such as a CPU or an application processor) that can execute the operations A, B, and C by executing one or more software programs stored in a memory device or a special-purpose processor (such as an embedded processor) for performing the operations.

[0014] The terminology and phraseology employed herein are for the purpose of describing some embodiments of the present disclosure only and are not intended to limit the scope of other embodiments of the present disclosure. It will be understood that the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. All terminology and phrases used herein, including technical and scientific terminology and phrases, have the same meaning as commonly understood by one of ordinary skill in the art to which embodiments of the present disclosure belong. It will be further understood that terms and phrases, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. In some instances, terms and phrases defined herein can be interpreted as excluding embodiments of the present disclosure.

[0015] Examples of the "electronic device" according to an embodiment of the present disclosure can include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device such as smart glasses, a head-mounted device (HMD), an electronic apparel, an electronic bracelet, an electronic necklace, an electronic appcessory, an electronic tattoo, a smart mirror, or a smart watch. Other examples of the electronic device include a smart home appliance. Examples of the smart home appliance can include at least one of a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washing machine, a dryer, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box (such as SAMSUNG HOMESYNC, APPLE TV, or GOOGLE TV), a smart speaker or a speaker with an integrated digital assistant (such as SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a game console (such as XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camcorder, or an electronic frame. Other examples of the electronic device include at least one of various medical devices (such as various portable medical measuring devices (such as a blood glucose measuring device, a heart rate measuring device, or a body temperature measuring device), a magnetic resource angiography (MRA) device, a magnetic resource imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), a car infotainment device, a sailing electronic device (such as a sailing navigation device or a gyro compass), avionics, security devices, a head unit for a vehicle, an industrial or home robot, an automatic teller machine (ATM), a point of sale (POS) device, or an Internet of Things (IoT) device (such as a light bulb, various sensors, a gas or water meter, a sprinkler, a fire alarm, a thermostat, a road sticker, a bread maker, a fitness equipment, a hot water tank, a heater, or a boiler). Other examples of the electronic device include at least one of furniture or a portion of a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (such as a device for measuring water, electricity, gas, or electromagnetic waves). Note that the electronic device according to various embodiments of the present disclosure can be one or a combination of the above-listed devices. According to some embodiments of the present disclosure, the electronic device can be a flexible electronic device. The electronic device disclosed herein is not limited to the above-listed devices and can include any other electronic device now known or developed later.

[0016] In the following description, in accordance with various embodiments of the present disclosure, an electronic device is described with reference to the accompanying drawings. As used herein, the term "user" can represent a person or another device (such as an artificial intelligence electronic device) that uses an electronic device.

[0017] Definitions for other certain words and phrases are provided throughout this patent document. Those of ordinary skill in the art will understand that in many, if not most instances, such definitions apply to prior as well as future uses of such defined words and phrases. BRIEF DESCRIPTION OF DRAWINGS

[0018] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings in which:

[0019] Figure 1 shows an example network configuration including an electronic device according to this disclosure;

[0020] Figure 2 , Figure 3 and Figure 4 shows an example arrangement of perspective cameras and eye viewpoints in video see-through (VST) extended reality (XR) according to this disclosure;

[0021] Figure 5 shows an example architecture to support dynamic alignment between perspective cameras and eye viewpoints in VST XR according to this disclosure;

[0022] Figure 6 , Figure 7 and Figure 8 shows an example dynamic alignment between perspective cameras and eye viewpoints in VST XR according to this disclosure;

[0023] Figure 9 , Figure 10 and Figure 11 shows an example arrangement of principal points associated with VST XR according to this disclosure;

[0024] Figure 12 shows an example architecture to support dynamic distortion correction in VST XR according to this disclosure; and

[0025] Figure 13 shows an example method for dynamic alignment between perspective cameras and eye viewpoints in VST XR according to this disclosure. DETAILED DESCRIPTION

[0026] The following discussion describes Figures 1 to 13and various embodiments of the present disclosure. It should be understood, however, that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or alternatives coming within the spirit of the present disclosure are within the scope of the present disclosure. Throughout the specification and drawings like or similar elements can be referred to using like or similar reference numerals.

[0027] As noted above, extended reality (XR) systems are becoming increasingly popular over time, and many applications have been and are being developed for XR systems. Some XR systems, such as augmented reality or “AR” systems and mixed reality or “MR” systems, can augment a user’s view of his or her current environment by overlaying digital content, such as information or virtual objects, on the user’s view of the current environment. For example, some XR systems can seamlessly blend virtual objects generated by computer graphics with real-world scenes.

[0028] Optical see-through (OST) XR systems refer to XR systems in which a user views a real-world scene directly through a head-mounted device (HMD). Unfortunately, OST XR systems face many challenges that can limit their adoption. Some of these challenges include limited field of view, limited use space, such as indoor-only use, inability to display fully opaque black objects, and use of complex optical pipelines that can require projectors, waveguides, and other optical elements. In contrast to OST XR systems, video see-through (VST) XR systems, also referred to as “passthrough” XR systems, present a user with a video sequence of a generated real-world scene. VST XR systems can be constructed using virtual reality (VR) technology and can have various advantages over OST XR systems. For example, VST XR systems can provide a wider field of view and can provide improved contextual augmented reality.

[0029] Many VST XR devices are adjustable to accommodate different interpupillary distances (IPDs). An interpupillary distance indicates a distance between pupils of a user’s eyes, and different users can have different interpupillary distances. Thus, making a VST XR device adjustable allows different users to use the VST XR device without requiring customized design for the VST XR device. However, adjusting a VST XR device to support different interpupillary distances can cause various problems. One problem is that a default interpupillary distance of a VST XR device is typically aligned with a perspective camera of the VST XR device, where the perspective camera is used to capture images of a scene surrounding the VST XR device. Changing the interpupillary distance to accommodate a position of a particular user’s eyes typically causes a misalignment between an actual position of the perspective camera and a virtual camera position, where the virtual camera position represents the position of the particular user’s eyes (also referred to as an eye viewpoint). Without corrective action, this can cause parallax effects or other undesirable visual artifacts to be produced from the user’s perspective.

[0030] The present disclosure provides various techniques that support dynamic alignment between perspective cameras and eye viewpoints in VST XR. As described in more detail below, an interpupillary distance between a left display lens and a right display lens of a VST XR device can be determined to have been adjusted relative to a default IPD of the VST XR device. An image can be captured using a perspective camera of the VST XR device, and the image can be transformed to match a viewpoint of a corresponding one of the display lenses in accordance with a change in the IPD relative to the default IPD in order to generate a transformed image. Distortion in the transformed image can be corrected based on one or more lens distortion coefficients corresponding to the change in the IPD in order to generate a corrected image, and the corrected image can be presented on a display panel of the VST XR device. This can be performed for images captured using different perspective cameras, such as a left perspective camera and a right perspective camera, to generate corrected images that are presented on one or more display panels, such as a left display panel and a right display panel. This can also be repeated for any number of images. In some cases, this can involve dynamically matching a principal point of the perspective camera with a principal point of the display panel.

[0031] In this way, these techniques support the use of adjustable interpupillary distances in a VST XR device. In some cases, for example, one or more positions of one or more see-through cameras are fixed in the VST XR device, and the relative positions between the see-through cameras and one or more virtual cameras (of one or more eyes of the user) can change when the interpupillary distance changes. The described techniques support remapping the perspective of the see-through cameras to the perspective of the virtual cameras based on this changed geometry, and principal point matching (such as between see-through camera viewpoints, display panel viewpoints, and eye viewpoints) can be performed. Moreover, different interpupillary distances can correspond to different distortions produced by display lenses or other components of the VST XR device, and the described techniques can compensate for the distortion based (at least in part) on the interpupillary distance being used. Among other things, the described techniques are able to dynamically align desired viewpoints and generate corrected final views of a scene for the user. For example, with proper alignment, the user can view an image in which virtual objects are properly aligned with real-world objects that have been corrected for parallax.

[0032] Figure 1 An example network configuration 100 including an electronic device according to this disclosure is shown. Figure 1 The embodiment of the network configuration 100 shown in FIG. 1 is for illustration only. Other embodiments of the network configuration 100 could be used without departing from the scope of this disclosure.

[0033] According to an embodiment of this disclosure, an electronic device 101 is included in the network configuration 100. The electronic device 101 can include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, and a sensor 180. In some embodiments, the electronic device 101 can exclude at least one of the components, or can add at least one other component. The bus 110 includes a circuit for connecting the components 120 to 180 to each other and for communicating communications (such as control messages and / or data) between the components.

[0034] The processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). In some embodiments, the processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), a graphics processing unit (GPU), or a neural processing unit (NPU). The processor 120 is capable of performing control of at least one of the other components of the electronic device 101 and / or performing operations or data processing related to communication or other functions. As described below, the processor 120 can be related to one or more functions that perform dynamic alignment between a perspective camera and an eye viewpoint in VST XR.

[0035] The memory 130 can include a volatile and / or nonvolatile memory. For example, the memory 130 can store commands or data related to at least one other component of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 can store software and / or a program 140. The program 140 includes, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or "application") 147. At least a portion of the kernel 141, the middleware 143, or the API 145 can be denoted as an operating system (OS).

[0036] The kernel 141 can control or manage system resources (such as the bus 110, the processor 120, or the memory 130) used to execute operations or functions implemented in other programs (such as the middleware 143, the API 145, or the application 147). The kernel 141 provides an interface that allows the middleware 143, the API 145, or the application 147 to access individual components of the electronic device 101 to control or manage system resources. The application 147 can include one or more applications that, among other things, perform dynamic alignment between a perspective camera and an eye viewpoint in VST XR. These functions can be performed by a single application or multiple applications, with each application performing one or more of these functions. For example, the middleware 143 can act as a relay to allow the API 145 or the application 147 to communicate data with the kernel 141. A plurality of applications 147 can be provided. The middleware 143 is capable of controlling work requests received from the application 147, such as by assigning priorities for use of system resources (such as the bus 110, the processor 120, or the memory 130) of the electronic device 101 to at least one of the plurality of applications 147. The API 145 is an interface that allows the application 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for file control, window control, image processing, or text control.

[0037] The I / O interface 150 serves as an interface that can, for example, deliver a command or data input from a user or other external device to other components of the electronic device 101. The I / O interface 150 can also output a command or data received from other components of the electronic device 101 to the user or other external device.

[0038] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. The display 160 can also be a depth-aware display such as a multi-focal display. The display 160 is capable of displaying, for example, a variety of contents (such as text, images, videos, icons, or symbols) to a user. The display 160 can include a touch screen and can receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of a user.

[0039] The communication interface 170 is capable of establishing communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106), for example. For example, the communication interface 170 can be connected with the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for transmitting and receiving signals.

[0040] Wireless communication can use, for example, at least one of WiFi, long term evolution (LTE), long term evolution-advanced (LTE-A), fifth generation wireless system (5G), millimeter wave or 60 GHz wireless communication, wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system for mobile communications (GSM) as a communication protocol. Wired connection can include, for example, at least one of a universal serial bus (USB), a high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS). The network 162 or 164 includes at least one communication network such as a computer network (like a local area network (LAN) or a wide area network (WAN)), the Internet, or a telephone network.

[0041] The electronic device 101 also includes one or more sensors 180 that can measure physical quantities or detect an activation state of the electronic device 101 and convert the measured or detected information into an electrical signal. For example, the sensors 180 can include a camera or other imaging sensor that can be used to capture images of a scene. The sensors 180 can also include one or more buttons for touch input, one or more microphones, a depth sensor, a gesture sensor, a gyroscope or gyro sensor, a barometric sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor such as a red, green, blue (RGB) sensor, a biophysical sensor, a temperature sensor, a humidity sensor, an illuminance sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasonic sensor, an iris sensor, or a fingerprint sensor. In addition, the sensors 180 can include one or more position sensors such as an inertial measurement unit that can include one or more accelerometers, gyroscopes, and other components. In addition, the sensors 180 can include a control circuit for controlling at least one of the sensors included herein. Any of these sensors 180 can be located within the electronic device 101.

[0042] In some embodiments, the electronic device 101 can be a wearable device or an electronic device-mountable wearable device such as an HMD. For example, the electronic device 101 can represent an XR wearable device such as headphones or smart glasses. In other embodiments, the first external electronic device 102 or the second external electronic device 104 can be a wearable device or an electronic device-mountable wearable device such as an HMD. In other embodiments, when the electronic device 101 is mounted in the electronic device 102 such as an HMD, the electronic device 101 can communicate with the electronic device 102 through the communication interface 170. The electronic device 101 can be directly connected with the electronic device 102 to communicate with the electronic device 102 without involving a separate network.

[0043] The first and second external electronic devices 102 and 104 and the server 106 can each be devices of the same or different type as the electronic device 101. According to certain embodiments of the present disclosure, the server 106 includes a set of one or more servers. Also, according to certain embodiments of the present disclosure, all or some of the operations performed on the electronic device 101 can be performed on another or multiple other electronic devices, such as the electronic devices 102 and 104 or the server 106. Furthermore, according to certain embodiments of the present disclosure, when the electronic device 101 is supposed to automatically, or on request, perform some function or service, the electronic device 101, instead of performing by itself, or in addition to that, the function or service, can request another device (such as the electronic devices 102 and 104 or the server 106) to perform at least some of the functions associated therewith. The other electronic device (such as the electronic devices 102 and 104 or the server 106) is capable of performing the requested function or additional function, and transferring the result of the performance to the electronic device 101. The electronic device 101 can provide the requested function or service by processing the received result as it is or in addition. To this end, for example, cloud computing, distributed computing, or client-server computing technology can be used. Although Figure 1 The electronic device 101 is shown to include a communication interface 170 that communicates with the external electronic device 104 or the server 106 via the network 162 or 164, but according to some embodiments of the present disclosure, the electronic device 101 can operate independently without a separate communication function.

[0044] The server 106 can include the same or similar components (or suitable subsets thereof) as the electronic device 101. The server 106 can support driving the electronic device 101 by performing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 can include a processing module or a processor that can support the processor 120 implemented in the electronic device 101. As described below, the server 106 can perform one or more functions related to dynamic alignment between a perspective camera and an eye viewpoint in VST XR.

[0045] Although Figure 1 One example of a network configuration 100 including the electronic device 101 is shown, but various changes can be made to Figure 1 For example, the network configuration 100 can include any number of each component in any suitable arrangement. In general, computing and communication systems have a wide variety of configurations, and Figure 1 The scope of the present disclosure is not limited to any particular configuration. Furthermore, although Figure 1 One operational environment in which various features disclosed in this patent document can be used is shown, but these features can be used in any other suitable system.

[0046] Figures 2 to 4 An example arrangement of a perspective camera and eye viewpoint in a VST XR according to this disclosure is shown. For ease of explanation, regarding... Figure 1 The network configuration 100 is described by electronic device 101. Figures 2 to 4 The arrangement shown. However, Figures 2 to 4 The arrangement shown can involve any other suitable equipment and any other suitable system.

[0047] like Figures 2 to 4 As shown, a user of electronic device 101 is viewing a three-dimensional (3D) scene. In this example, the scene is represented using plane 202, which is associated with a specific point P within the 3D scene. Electronic device 101 includes a left perspective camera and right perspective cameras 204a to 204b, which can be used to capture images of the 3D scene. Perspective cameras 204a to 204b can, for example, represent an imaging sensor 180 of electronic device 101. Each of the perspective cameras 204a to 204b can be used to capture perspective image frames 206a to 206b respectively, wherein perspective image frames 206a to 206b represent images of the 3D scene captured from the viewpoint of perspective cameras 204a to 204b.

[0048] Perspective image frames 206a to 206b can be used to generate images presented on the left and right display panels 208a to 208b of the electronic device 101. Display panels 208a to 208b can, for example, represent one or more displays 160 of the electronic device 101. In some cases, display panels 208a to 208b can represent a single display 160. In other cases, display panels 208a to 208b can represent different portions of the same display 160. Left and right display lenses 210a to 210b are used to focus the images presented on the display panels 208a to 208b, wherein the left and right display lenses 210a to 210b can represent convex-convex lenses or other suitable lenses used in the electronic device 101. The displayed images are used to generate left and right virtual image frames 212a to 212b, which can be viewed by the user's left and right eyes 214a to 214b.

[0049] exist Figure 2 In the illustrated arrangement 200, the default distance between the user's left and right eyes 214a and 214b (or between the optical centers of the left and right display lenses 210a and 210b) is referred to as the default interpupillary distance (ipd). In some embodiments, the default interpupillary distance (ipd) may be approximately 63.5 mm, but other default interpupillary distances may be used. Figure 2As shown, the perspective camera 204a, the display panel 208a, and the display lens 210a are all aligned along a common optical axis on the left side. Similarly, the perspective camera 204b, the display panel 208b, and the display lens 210b are all aligned along a common optical axis on the right side. Thus, the viewpoints of the perspective cameras, the viewpoints of the eyes, and the centers of the display panels are aligned on the left side and also on the right side. Accordingly, each perspective image frame 206a to 206b can be processed and rendered onto the corresponding display panel 208a to 208b, and each eye 214a to 214b can view the rendered frames on the corresponding display panel 208a to 208b through the corresponding display lens 210a to 210b.

[0050] In Figure 2 the arrangement 200, a point P within the scene is projected onto the left perspective camera 204a at point and is projected onto the left virtual camera (left eye 214a) at point Since the left virtual camera cannot directly view the point P, the point on the left perspective image frame 206a is transformed into a point on the left virtual image frame 212a. Similarly, the point P within the scene is projected onto the right perspective camera 204b at point and is projected onto the right virtual camera (right eye 214b) at point Since the right virtual camera cannot directly view the point P, the point on the right perspective image frame 206b is transformed into a point on the right virtual image frame 212b.

[0051] In Figure 3 the arrangement 300 shown, the display panels 208a to 208b and the display lenses 210a to 210b have been moved inward, which reduces the distance between the user's left and right eyes 214a to 214b and creates an interpupillary distance ipd1. Here, the display panel 208a and the display lens 210a can be moved inward (to the left) by a distance , and the display panel 208b and the display lens 210b can be moved inward (to the right) by a distance . Accordingly, it can be said that ipd1 < ipd, and the range of ipd1 can be from ipd - 2 up to just below ipd.

[0052] As Figure 3As shown, the perspective camera 204a is not aligned with the display panel 208a and the display lens 210a, meaning that the optical axis of the perspective camera 204a is not the same as the optical axis of the display panel 208a and the display lens 210a. Similarly, the perspective camera 204b is not aligned with the display panel 208b and the display lens 210b, meaning that the optical axis of the perspective camera 204b is not the same as the optical axis of the display panel 208b and the display lens 210b. Therefore, the viewpoint of the perspective camera, the viewpoint of the eye, and the center of the display panel are misaligned on the left and on the right. As a result, each perspective image frame 206a to 206b can be transformed and rendered onto the corresponding display panels 208a to 208b as described below, and each eye 214a to 214b can view the rendered frame on the corresponding display panels 208a to 208b through the corresponding display lenses 210a to 210b.

[0053] exist Figure 3 In the arrangement of 300, point P in the scene is at point... The point is projected onto the left perspective camera 204a and at point Point P is projected onto the left virtual camera (left eye 214a). Since the left virtual camera cannot directly view point P, the point on the left perspective image frame 206a is... Points transformed into left virtual image frame 212a Similarly, point P in the scene is at point... The point is projected onto the right perspective camera 204b and at point Point P is projected onto the right virtual camera (right eye 214b). Since the right virtual camera cannot directly view point P, the point on the right perspective image frame 206b is... Points transformed into right virtual image frame 212b However, with Figure 2 Compared to the arrangement 200 shown, the projection points on the left virtual image frame 212a and the projection point on the right virtual image frame 212b The position changes due to the altered geometric relationships, and new transformations can be used to generate left and right virtual views.

[0054] exist Figure 4 In the arrangement 400 shown, the display panels 208a to 208b and the display lenses 210a to 210b have been moved outwards, which increases the distance between the user's left and right eyes 214a to 214b and generates the interpupillary distance ipd2. Here, the display panels 208a and the display lenses 210a can be moved outwards (to the left) by a certain distance. Furthermore, the display panel 208b and the display lens 210b can move outward (to the right) a certain distance. Therefore, it can be said that ipd < ipd2, and the range of ipd2 can be from just greater than ipd up to ipd + 2 .

[0055] As Figure 4 shown, the perspective camera 204a is not aligned with the display panel 208a and the display lens 210a, meaning that the optical axis of the perspective camera 204a is not the same as the optical axes of the display panel 208a and the display lens 210a. Similarly, the perspective camera 204b is not aligned with the display panel 208b and the display lens 210b, meaning that the optical axis of the perspective camera 204b is not the same as the optical axes of the display panel 208b and the display lens 210b. Therefore, the viewpoints of the perspective cameras, the viewpoints of the eyes, and the centers of the display panels are not aligned on the left side and are not aligned on the right side. As a result, each perspective image frame 206a to 206b can be transformed and rendered onto the corresponding display panels 208a to 208b as described below, and each eye 214a to 214b can view the rendered frames on the corresponding display panels 208a to 208b through the corresponding display lenses 210a to 210b.

[0056] In Figure 4 the arrangement 400, a point P within the scene is projected onto the left perspective camera 204a at a point and is projected onto the left virtual camera (left eye 214a) at a point . Since the left virtual camera cannot directly view the point P, the point on the left perspective image frame 206a is transformed into a point on the left virtual image frame 212a. Similarly, the point P within the scene is projected onto the right perspective camera 204b at a point and is projected onto the right virtual camera (right eye 214b) at a point . Since the right virtual camera cannot directly view the point P, the point on the right perspective image frame 206b is transformed into a point on the right virtual image frame 212b. However, compared to the arrangement 200 shown in Figure 2 , the positions of the projected points on the left virtual image frame 212a and the projected points on the right virtual image frame 212b change due to the change in geometric relationships, and new transformations can be used to generate the left virtual view and the right virtual view.

[0057] Based on this, the electronic device 101 can be configured to allow for dynamic adjustment of the interpupillary distance within the range from ipd1 to ipd2, which can be expressed as [ipd - 2 , ipd + 2 Assuming perspective cameras 204a to 204b have fixed positions, adjusting the interpupillary distance to any value other than the default ipd can cause misalignment between perspective cameras 204a to 204b and the corresponding display panels 208a to 208b and display lenses 212a to 212b. The electronic device 101 can use the techniques described below to support remapping of image data to compensate for this lack of alignment when the interpupillary distance is dynamically adjusted away from the default ipd. For example, the projection point on the left perspective image frame 206a... Projection points on right perspective image frame 206b It can remain unchanged regardless of the adjustment of the interpupillary distance. However, the projection point on the left virtual image frame 212a and the projection point on the right virtual image frame 212b It can be changed. Electronic device 101 can use the techniques described below to dynamically adjust the point based on the geometric relationship associated with the current value of the interpupillary distance. Transform into points and point Transform into points .

[0058] The techniques described below can also be used to support adaptive geometric distortion and chromatic aberration models when the interpupillary distance is dynamically adjusted. Changes in the interpupillary distance can affect the geometric distortion, chromatic aberration, or other distortions produced by the electronic device 101 on the images presented on the display panels 208a to 208b. Therefore, changing the interpupillary distance can change how geometric distortion, chromatic aberration, or other distortions are produced, thereby affecting how distortion compensation is performed to reduce or eliminate those distortions. The electronic device 101 can use the techniques described below to adjust one or more models used to perform distortion compensation based on the geometric relationship associated with the current value of the interpupillary distance. Note that this can be particularly useful when each display panel 208a to 208b is movable relative to its associated display lenses 210a to 210b, meaning that each display panel 208a to 208b and its corresponding display lenses 210a to 210b do not need to be fixed relative to each other.

[0059] although Figures 2 to 4 This shows an example of the arrangement of the perspective camera and eye viewpoint in a VST XR, but it is possible to modify... Figures 2 to 4 Various changes were made. For example, although it is assumed here that the range of interpupillary distance can be from IPD-2. to ipd+2 However, it is not required that the range of interpupillary distance values ​​be symmetrical. As a specific example, the default interpupillary distance (ipd) can be closer to one end of the range of interpupillary distance values ​​than the other end.

[0060] Figure 5 An example architecture 500 that supports dynamic alignment between a see-through camera and an eye point in VST XR according to this disclosure is shown. To facilitate explanation, Figure 5 The architecture 500 is described as being implemented using the electronic device 101 in the network configuration 100 of Figure 1 , where the architecture 500 can be used with any arrangement of components shown in Figures 2 to 4 . However, the architecture 500 can be implemented using any other suitable device and in any other suitable system, and the architecture 500 can be used with any other suitable arrangement of components.

[0061] As shown in Figure 5 , the architecture 500 receives and processes the see-through image frame 206a or 206b. As described above, the see-through image frame 206a or 206b is captured using one of the see-through cameras 204a or 204b. The camera lens undistortion operation 502 generally operates to pre-process the see-through image frame 206a or 206b in order to reduce camera lens distortions in the see-through image frame 206a or 206b. The camera lens undistortion operation 502 can reduce any suitable camera lens-based distortions from the see-through image frame 206a or 206b, such as radial distortions or tangential distortions.

[0062] The principal point matching operation 504 generally operates to match a principal point of the see-through camera 204a or 204b with a center of the corresponding display panel 208a or 208b. For example, the principal point matching operation 504 can identify a principal point of the see-through camera 204a or 204b and identify a center point of the corresponding display panel 208a or 208b, which allows the principal point matching operation 504 to indicate whether the see-through camera 204a or 204b and the corresponding display panel 208a or 208b are aligned.

[0063] In some embodiments, one or both of the camera lens de-distortion operation 502 and the principal point matching operation 504 can operate using a perspective camera matrix and lens distortion model 506, where the perspective camera matrix and lens distortion model 506 represents camera calibration data or other data defining known characteristics of the electronic device 101. The camera matrix is typically defined as a three-by-three matrix that includes two focal lengths in the x and y directions and a principal point of the camera defined using x and y coordinates. The lens distortion model is typically defined as a mathematical model that indicates how an image can be de-distorted, which can be derived based on the particular lens or other optical components used. In some cases, multiple lens distortion models can be used, such as when one model is used to correct for lens geometric distortion and another model is used to correct for chromatic aberration. Note that the principal point matching can be static since the location of the perspective camera 204a or 204b and the camera calibration parameters are fixed, and can only need to be performed once (regardless of how many perspective image frames 206a or 206b are captured using the current interpupillary distance of the electronic device 101).

[0064] The IPD change detection operation 508 generally operates to detect a change in the interpupillary distance of the electronic device 101, such as to a non-default interpupillary distance value (e.g., a value other than ipd). In some cases, for example, the IPD input 510 can represent input from a sensor or other device that can sense or measure the current interpupillary distance of the electronic device 101. If the interpupillary distance of the electronic device 101 has not changed (such as when the current interpupillary distance of the electronic device 101 matches the default interpupillary distance ipd), the electronic device 101 can already know how to map the image frames 206a or 206b to the virtual image frames 212a or 212b. This is because the alignment of the perspective camera 204a or 204b with the corresponding display panel 208a or 208b can be known. The camera frame to aligned virtual frame mapping operation 512 generally operates to map the image frames 206a or 206b into the corresponding virtual image frames 212a or 212b. Using the known geometric relationship between the image frames 206a or 206b and the virtual image frames 212a or 212b (which is known due to the alignment of the perspective camera 204a or 204b with the corresponding display panel 208a or 208b), the camera frame to aligned virtual frame mapping operation 512 can map each point in the image frames 206a or 206b to a corresponding point in the corresponding virtual image frames 212a or 212b. Figure 2 As an example, the camera frame to aligned virtual frame mapping operation 512 can map each point in the image frames 206a or 206b to a corresponding point in the corresponding virtual image frames 212a or 212b based on the known geometric relationship between the image frames 206a or 206b and the virtual image frames 212a or 212b (which is known due to the alignment of the perspective camera 204a or 204b with the corresponding display panel 208a or 208b). or This can be done for all points in the image frames 206a or 206b. or

[0065] ​If the interpupillary distance of electronic device 101 has changed (e.g., when the current interpupillary distance of electronic device 101 no longer matches the default interpupillary distance ipd), then electronic device 101 learns how to map image frame 206a or 206b to virtual image frame 212a or 212b. In this example, alignment operation 514 typically operates to dynamically align the viewpoint of the perspective camera, the viewpoint of the eye, and the center of the display panel in order to establish the geometric relationship between those points. Here it is assumed that display panel 208a or 208b and the associated display lens 210a or 210b have moved inward or outward such that the current interpupillary distance of electronic device 101 no longer matches the default interpupillary distance ipd. Given this, alignment operation 514 can identify the current viewpoint of perspective camera 204a or 204b, the current viewpoint of user eye 214a or 214b, and the current position of the center of display panel 208a or 208b.

[0066] The misaligned camera frame to misaligned virtual frame mapping operation 516 typically operates by mapping image frame 206a or 206b to corresponding virtual image frame 212a or 212b based on the currently learned geometric relationship between the current viewpoint of perspective camera 204a or 204b, the current viewpoint of user eye 214a or 214b, and the current position of the center of display panel 208a or 208b. Figure 3 or Figure 4 As an example, the misaligned camera frame to misaligned virtual frame mapping operation 516 can map individual points from image frame 206a or 206b based on the learned geometric relationship between image frame 206a or 206b and virtual image frame 212a or 212b. or Mapped to the corresponding point in the corresponding virtual image frame 212a or 212b or Similarly, this can be done for all points in image frames 206a or 206b. This dynamically maps the misaligned perspective image frames 206a or 206b to the misaligned virtual image frames 212a or 212b by matching different viewpoints and applying adjusted geometric relations. Among other things, this helps to dynamically correct parallax errors while changing interpupillary distances by mapping perspective image frames 206a or 206b to virtual image frames 212a or 212b.

[0067] Dynamic principal point matching operation 518 generally operates to dynamically match the principal point of perspective camera 204a or 204b with the center of display panel 208a or 208b. For example, dynamic principal point matching operation 518 can identify the principal point of perspective camera 204a or 204b and identify the center point of the corresponding display panel 208a or 208b, which can now be different due to the misalignment of perspective camera 204a or 204b and the corresponding display panel 208a or 208b. The end result of this sequence of operations is to generate a transformed image.

[0068] Model adaptation operation 520 generally operates to adjust the lens distortion models to account for the now misaligned nature of the various components in electronic device 101. For example, model adaptation operation 520 can adjust the model used to correct for lens geometric distortion and the model used to correct for chromatic aberration. One or more of the models can be adjusted here based on the change in the interpupillary distance of electronic device 101. For example, different interpupillary distances can have different parameters for the lens distortion model and the chromatic aberration model. Using the current value of the interpupillary distance allows electronic device 101 to identify the appropriate model to use to correct virtual image frame 212a or 212b.

[0069] Distortion correction operation 522 generally operates to reduce or eliminate distortion in virtual image frame 212a or 212b (transformed image). For example, distortion correction operation 522 can perform display lens geometric distortion correction and chromatic aberration correction using the appropriate distortion models based on the current interpupillary distance. Note that distortion correction operation 522 can be performed regardless of whether virtual image frame 212a or 212b is generated using aligned camera frame to aligned virtual frame mapping operation 512 or misaligned camera frame to misaligned virtual frame mapping operation 516. This results in the generation of a corrected image.

[0070] One or more post-processing operations can be performed using the corrected image. For example, a head pose change compensation operation 524 can be used to sense a change in a head pose of a user that is using the electronic device 101. In some cases, this can be based on IMU sensor data or other sensor data generated by the electronic device 101. The change in head pose can be used to adjust the corrected image, such as for changes in user head pose that occur between when the image frame 206a or 206b is captured and when the resulting virtual image frame 212a or 212b is being rendered and presented. A virtual image frame rendering operation 526 generally operates to render the virtual image frame 212a or 212b (possibly modified by the head pose change compensation operation 524). The virtual image frame rendering operation 526 generates a final image of the scene that can be presented to the user, such as on the corresponding display panel 208a-b. A virtual image frame presentation operation 528 generally operates to initiate presentation of the rendered final image on the corresponding display panel 208a-b, such as by providing suitable image data to the corresponding display panel 208a-b.

[0071] Although Figure 5 One example of an architecture 500 that supports dynamic alignment between perspective cameras and eye viewpoints in VST XR is shown, but various changes can be made Figure 5 For example, Figure 5 Various components or functions in can be combined, further subdivided, omitted, replicated, or rearranged, and additional components or functions can be added in accordance with particular needs. Also, note that the same processes described above as performed by the architecture 500 can be repeated for any number of perspective image frames 206a-b. For example, the architecture 500 can be used to repeatedly process perspective image frames 206a-b captured using left and right perspective cameras 204a-b for presenting rendered images on display panels 208a-b.

[0072] Figures 6 to 8 An example dynamic alignment between perspective cameras and eye viewpoints in VST XR in accordance with the present disclosure is shown. To facilitate explanation, Figures 6 to 8 The dynamic alignment shown in is described as being provided by Figure 5 the architecture 500 of which can be implemented using the electronic device 101 in the network configuration 100 of Figure 1 However, the architecture 500 can be used in any other suitable manner.

[0073] As described above, alignment operation 514 typically operates to dynamically align the viewpoint of the perspective camera, the viewpoint of the user's eye, and the center of the display panel in order to establish the geometric relationship between those points. Furthermore, the misaligned camera frame to misaligned virtual frame mapping operation 516 typically operates to map perspective image frame 206a or 206b to the corresponding virtual image frame 212a or 212b based on the geometric relationship between the current viewpoint of perspective camera 204a or 204b, the current viewpoint of the user's eye 214a or 214b, and the current position of the center of display panel 208a or 208b.

[0074] like Figure 6 As shown, the perspective camera 204a, display panel 208a, and display lens 210a are aligned such that a common optical axis passes through all three components. Although not shown here, the perspective camera 204b, display panel 208b, and display lens 210b are also aligned. This is consistent with... Figure 2 The arrangement shown in 200 is consistent. In this example, point P in the 3D scene is projected onto point p1 in perspective image frame 206a, where point p1 has coordinates (x, y, y). s y s Point p1 is projected onto point p2 in virtual image frame 212a, where point p2 has coordinates (x, y). e y e The center of the perspective camera lens is represented as o. s And the center of the virtual camera at the user's eye 214a is represented as o. e The principal point of the perspective camera 204a is represented as o1, and the principal point of the virtual camera is represented as o2 (note that in the alignment configuration, o2=o). s Point o represents a point on the optical axis of the perspective camera 204a at plane 202 of the 3D scene. The distance between the user's eye 214a and the imaging plane of the perspective camera 204a is represented as d. es And the depth of point P is denoted as d. The focal length of the user's eye 214a is denoted as f. e Furthermore, the focal length of the perspective camera 204a is expressed as f. s (in some cases, f) e =f s =f).

[0075] In some embodiments, Figure 6 In order to map the perspective image frame 206a to the virtual image frame 212a, it can be shown that:

[0076] [Mathematical Expression 1]

[0077]

[0078] [Mathematical Expression 2]

[0079]

[0080] Here, x s p1(x) s y s The x-coordinate of point P(X, Y, Z) is given by f, where X represents the x-coordinate of the 3D point P(X, Y, Z), f represents the focal length of the perspective camera 204a, and d represents the depth of the 3D point P(X, Y, Z). It can also be shown that:

[0081] [Mathematical Expression 3]

[0082]

[0083] [Mathematical Expression 4]

[0084]

[0085] Here, x e p2(x) e y e The x-coordinate of ) and d es This represents the distance between the user's eye 214a and the imaging plane of the perspective camera 204a. The following can be obtained by removing X from equations 2 and 4.

[0086] [Mathematical Expression 5]

[0087]

[0088] Exporting y e Following similar equations, the following set of equations can be used to map perspective image frame 206a to virtual image frame 212a.

[0089] [Mathematical Expression 6]

[0090]

[0091] like Figure 7 As shown, the perspective camera 204a, display panel 208a, and display lens 210a are not aligned. Instead, the display panel 208a and display lens 210a have been moved inward by a distance. Therefore, there exists an optical axis o passing through the perspective camera 204a and an optical axis o' passing through the display panel 208a and the display lens 210a. Although not shown here, the perspective camera 204b, the display panel 208b, and the display lens 210b have the same arrangement, wherein the display panel 208b and the display lens 210b have been moved inward by a distance. This is related to Figure 3the arrangement 300 shown in FIG. 3. The principal point of the virtual camera in its new position is denoted as o2’ (note that in the misaligned configuration, ). The point o’ denotes the point on the optical axis of the virtual camera at the plane 202 of the 3D scene, which is now different from the point o.

[0092] In some embodiments, in Figure 7 , to map the perspective image frame 206a to the virtual image frame 212a, one can show:

[0093] [math. 7]

[0094]

[0095] [math. 8]

[0096]

[0097] These match the above math. 1 and math. 2. One can also show:

[0098] [math. 9]

[0099]

[0100] [math. 10]

[0101]

[0102] By removing X in math. 8 and math. 10, one can obtain the following.

[0103] [math. 11]

[0104]

[0105] After deriving the similar equations for y e , one can use the following set of equations to map the perspective image frame 206a to the virtual image frame 212a.

[0106] [math. 12]

[0107]

[0108] As shown in Figure 8 , the perspective camera 204a, the display panel 208a, and the display lens 210a are misaligned. Instead, the display panel 208a and the display lens 210a have moved outward by a distance Thus there is an optical axis o through the perspective camera 204a and an optical axis o' through the display panel 208a and the display lens 210a. Although not shown here, the perspective camera 204b, the display panel 208b and the display lens 210b have the same arrangement, where the display panel 208b and the display lens 210b have been moved outward by a distance This is in line with the arrangement 400 shown in Figure 4 .

[0109] In some embodiments, in Figure 8 order to map the perspective image frame 206a to the virtual image frame 212a, it can be shown that:

[0110] [Equation 13]

[0111]

[0112] [Equation 14]

[0113]

[0114] These match the above Equation 1 and Equation 2. It can also be shown that:

[0115] [Equation 15]

[0116]

[0117] [Equation 16]

[0118]

[0119] By removing X in Equation 14 and Equation 16, the following can be obtained.

[0120] [Equation 17]

[0121]

[0122] After deriving similar equations for y e , the following set of equations can be used to map the perspective image frame 206a to the virtual image frame 212a.

[0123] [Equation 18]

[0124]

[0125] As can be seen here, Equation 6, 12 and 18 provide a way to transform the perspective image frame 206a to match the viewpoint of the corresponding display lens 210a. Whether the current interpupillary distance matches the default interpupillary distance, is 2 less than the default interpupillary distance or is 2 , the mapping can be implemented. Thus, the aligned operation 514 and the misaligned camera frame to misaligned virtual frame mapping operation 516 can dynamically align the viewpoint in the presence of a changing interpupillary distance.

[0126] Although Figures 6 to 8 an example of dynamic alignment between a perspective camera and an eye viewpoint in VST XR is shown, various changes can be made. Figures 6 to 8 For example, for components associated with the user’s right eye 214b, the same or similar processes described with reference to the user’s left eye 214a can occur. Further, while it is assumed here that each display panel 208a-b is fixed relative to its associated display lens 210a-b, other embodiments can allow each display panel 208a-b to be movable relative to its associated display lens 210a-b. Figures 6 to 8 The same or similar processes described with reference to the network configuration 100 of FIG. 1 can be performed with respect to the arrangement shown in FIG. 2.

[0127] Figures 9 to 11 An example arrangement of a principal point associated with VST XR according to the present disclosure is shown. To facilitate explanation, the arrangement shown in FIG. 2 is described with respect to the electronic device 101 in the network configuration 100 of FIG. 1. However, the arrangement shown in FIG. 2 can involve any other suitable device and any other suitable system. Figure 1 Figures 9 to 11 The same or similar processes described with reference to the network configuration 100 of FIG. 1 can be performed with respect to the arrangement shown in FIG. 2. Figures 9 to 11 The arrangement shown in FIG. 2 can involve any other suitable device and any other suitable system.

[0128] As described above, the dynamic principal point matching operation 518 generally operates to dynamically match a principal point of the perspective camera 204a or 204b with a center of the display panel 208a or 208b. As shown in FIG. 3, a point is identified with respect to each of the perspective camera 204a, the perspective image frame 206a, the display panel 208a, the display lens 210a, and the user eye 214a. Incident illumination 902 is received from a scene and the perspective camera 204a generates the perspective image frame 206a using the incident illumination 902. The identified point of the perspective camera 204a can represent a principal point of the perspective camera 204a, where the principal point of the perspective camera 204a indicates a location where an optical axis intersects an image plane of the perspective camera 204a. The identified point of the display panel 208a can represent a center of the display panel 208a. The identified point of the eye 214a can represent a viewpoint of the user eye 214a. While not shown here, the same arrangement of elements can be associated with the user’s right eye 214b. Figure 9 In the example shown in FIG. 3, the respective points are aligned, indicating

[0129] Figure 9 Figure 2 ​​​The arrangement 200 shown is currently in use. Therefore, the principal point of the perspective camera, the eye's viewpoint, and the center of the display panel are aligned, which typically occurs when the interpupillary distance used matches the default interpupillary distance (ipd). Thus, the perspective image frame 206a can be mapped to a virtual image frame 212a using alignment geometry for parallax correction. The virtual image frame 212a can be rendered and displayed on the corresponding display panel 208a, and the user can obtain a corrected view through the corresponding display lens 210a.

[0130] like Figure 10 and Figure 11 As shown, the points are no longer aligned, indicating Figure 3 The arrangement shown is 300 or Figure 4 The arrangement 400 shown is currently in use. Therefore, the geometric relationship between the principal point of the perspective camera, the eye's viewpoint, and the center of the display panel has changed. Therefore, in order to map the perspective image frame 206a to the virtual image frame 212a, the principal point of the perspective camera is matched with the eye's viewpoint and the center of the display panel. Furthermore, distortion correction can be performed based on the change in interpupillary distance relative to the default interpupillary distance ipd.

[0131] When the principal point of the perspective camera 204a does not match the center of the display panel 208a, the perspective image frame 206a can be transformed by moving the principal point of the perspective camera 204a to the center of the virtual image frame 212a. In some embodiments, this can be accomplished using a camera matrix. An example of a camera matrix is ​​shown below.

[0132] [Mathematical Expression 19]

[0133]

[0134] here, This indicates that the perspective camera 204a is in and The focal length in the direction, and The principal point of the perspective camera 204a is indicated. In some cases, the camera matrix can be obtained through camera calibration. After mapping, the virtual image frame 212a can be rendered and displayed on the corresponding display panel 208a, and the user can obtain a corrected view through the corresponding display lens 210a. Figure 10 In the example shown, it can be shown As shown in the reference above Figure 3 As stated above. Figure 11 In the example shown, it can be shown As shown in the reference above Figure 4The. In either case, appropriate geometric relationships based on decreasing or increasing interpupillary distance can be identified, and the perspective image frame 206a can be mapped to a virtual image frame 212a. The same type of process can be used to map the perspective image frame 206b, where the perspective image frame 206b can be mapped to a virtual image frame 212b.

[0135] Although Figures 9 to 11 examples are shown of arrangements of principal points associated with VST XR, various changes can be made Figures 9 to 11 For example, while it is assumed here that the range of interpupillary distances can be from ipd-2 to ipd+2 , it is not required that the range of interpupillary distance values need to be symmetric.

[0136] Figure 12 An example architecture 1200 that supports dynamic distortion correction in VST XR according to the present disclosure is shown. For ease of explanation, Figure 12 the architecture 1200 is described as being implemented using the electronic device 101 in the network configuration 100 of Figure 1 However, the architecture 1200 can be implemented using any other suitable device and in any other suitable system.

[0137] As shown in Figure 12 the architecture 1200 includes Figure 5 various operations or in conjunction with various operations of Figure 5 Thus, the architecture 1200 can be used as part of or with the architecture 500. In this example, the perspective image frame 206a or 206b can be mapped using the misaligned camera frame to misaligned virtual frame mapping operation 516, and the distortion correction operation 522 can reduce or eliminate distortion in the virtual image frame 212a or 212b. For brevity, other elements from Figure 12 are omitted from Figure 5

[0138] As part of the distortion correction operation 522, one or more models 1202 can be applied to the virtual image frame 212a or 212b in order to generate a corrected image. The one or more models 1202 can be used to correct any suitable distortion contained in the virtual image frame 212a or 212b, such as lens geometric distortion and / or chromatic aberration. In some cases, separate models 1202 can be used to correct for lens geometric distortion and chromatic aberration.

[0139] ​The model 1202 used for distortion correction here can be adapted by the model adaptation operation 520 in order to account for the current interpupillary distance used by the electronic device 101. For example, in some cases, the model 1202 can include different parameters that are applied when different interpupillary distances are used, and the model adaptation operation 520 can select the appropriate parameters based on the current interpupillary distance. In some cases, the parameters of the model 1202 can be determined for different interpupillary distances using known calibrations of the distortion 1204, where the known calibrations of the distortion 1204 can represent calibration data associated with lens geometric distortion, chromatic aberration, or other distortions associated with different interpupillary distances within a design range 1206 of the IPD. Thus, the model 1202 can have different parameters for correcting distortions associated with different interpupillary distances.

[0140] As a specific example of this, a display lens distortion model can be represented as follows.

[0141] [mathematical expression 20]

[0142]

[0143] Here, represents one or more lens distortion coefficients, x and y represent coordinates on a corresponding display panel, and Different interpupillary distances can correspond to different distortion coefficients, which in some cases can be the obtained frame display lens calibration. Thus, the values of the lens distortion coefficients can be selected based on the current interpupillary distance used. Once corrected, the final image of the scene can be presented to the user. Thus, different users with different interpupillary distances can receive substantially the same view based on using the appropriate mapping and the appropriate distortion correction.

[0144] Although Figure 12 the example architecture 1200 is shown to support dynamic distortion correction in VST XR, various changes can be made Figure 12 . For example, Figure 12 various components or functions in can be combined, further subdivided, replicated, omitted, or rearranged, and additional components or functions can be added in accordance with particular needs. Also, note that the same processes described above as performed by the architecture 1200 can be repeated for any number of perspective image frames 206a-b. For example, the architecture 1200 can be used to repeatedly process perspective image frames 206a-b captured using left and right perspective cameras 204a-b for rendering images for presentation on display panels 208a-b.

[0145] Figure 13An example method 1300 for dynamic alignment between a perspective camera and an eye viewpoint in a VST XR, according to this disclosure, is shown. For ease of explanation, Figure 13 The method 1300 shown is described as being by Figure 1 The network configuration 100 is performed by electronic device 101, wherein electronic device 101 can use Figure 5 and Figure 12 The architectures 500 and 1200 are shown. However, they can be implemented in any other suitable system using any other suitable equipment and architecture. Figure 13 Method 1300 is shown.

[0146] like Figure 13 As shown, at step 1302, it is determined that the interpupillary distance (IPD) between the left and right display lenses of the VST XR device has been adjusted relative to the default IPD of the VST XR device. In some embodiments, a user can manually adjust the IPD by interacting with user control elements, such as physical buttons on the VST XR device or a virtual control panel displayed on the VST XR device, to improve alignment between the user's eyes and the display panel of the VST XR device. In other embodiments, a camera or other sensor of the VST XR device can be used to capture an image of the user's eyes, and this image can be used to determine the position of the user's eyes. Based on the determined position, the IPD can be automatically adjusted for the user. In response to the adjustment of the IPD, the processor 120 of the electronic device 101 can receive sensor input or other input indicating that the IPD between the left and right display lenses 210a to 210b has been adjusted and is no longer at the default IPD. The input can be represented in any suitable manner, such as a change in the IPD relative to the default IPD or a setting associated with a known change in distance.

[0147] At step 1304, one or more perspective cameras of the VST XR device are used to capture one or more images. This may include, for example, the processor 120 of the electronic device 101 obtaining one or more perspective image frames 206a-206 captured using one or more perspective cameras 204a-204b of the electronic device 101. In some cases, this may include obtaining one or more left perspective image frames 206a captured using the left perspective camera 204a and one or more right perspective image frames 206b captured using the right perspective camera 204b. Each perspective image frame 206a-206b may represent a captured image of a 3D scene, such as the scene around the user / electronic device 101. Each perspective image frame 206a-206b may optionally be preprocessed, such as by performing operation 502.

[0148] At step 1306, each image is transformed to match the viewpoint of the corresponding display lens according to the change in interpupillary distance relative to the default interpupillary distance. This can include, for example, the processor 120 of the electronic device 101 performing operations 514, 516, 518 to transform each perspective image frame 206a-b to a corresponding virtual image frame 212a-b. In some embodiments, this can involve performing the mapping using mathematical equation 12 if the interpupillary distance is less than the default interpupillary distance, or using mathematical equation 18 if the interpupillary distance is greater than the default interpupillary distance. Among other things, the mapping can help dynamically correct parallax errors. Part of the transformation here can involve dynamically matching the principal point of each perspective camera 204a-b to the principal point (center) of the corresponding display panel 208a-b. This can result in generating one or more transformed images.

[0149] At step 1308, distortion in the one or more transformed images can be at least partially corrected in order to generate one or more corrected images. This can include, for example, the processor 120 of the electronic device 101 performing operations 520, 522 to correct lens geometric distortion and / or chromatic aberration using one or more models 1202, where the correction can be based on the change in interpupillary distance relative to the default interpupillary distance. In some embodiments, this can involve performing distortion correction using equation (20). Thus, for example, one or more display lens geometric distortion and chromatic aberration models 1202 can be dynamically adapted based on the change in interpupillary distance, and the one or more adapted display lens geometric distortion and chromatic aberration models 1202 can be used to correct display lens geometric distortion and chromatic aberration.

[0150] At step 1310, each corrected image can be rendered, and presentation of each rendered image can be initiated at step 1312. This can include, for example, the processor 120 of the electronic device 101 performing operations 526, 528 (possibly prior to operation 524) to generate one or more rendered images that can be presented on one or more appropriate display panels 208a-b. As noted above, the corrected image can be rendered and presented on the left display panel 208a associated with the user’s left eye 214a when the associated perspective image frame 206a was captured using the left perspective camera 204a, and the corrected image can be rendered and presented on the right display panel 208b associated with the user’s right eye 214b when the associated perspective image frame 206b was captured using the right perspective camera 204b.

[0151] Although Figure 13 One example of a method 1300 for dynamic alignment between perspective cameras and eye viewpoints in VST XR is shown, but Figure 13Various changes can be made. For example, while shown as a series of steps, various steps can overlap, occur in parallel, occur in a different order, or occur any number of times (including zero times). Figure 13

[0152] It should be noted that the perspective cameras 204a-b are generally assumed to be pointing forward, and this orientation of the perspective cameras 204a-b is shown in the various figures. However, this need not be the case, and other orientations of the perspective cameras 204a-b can be used. For example, the perspective cameras 204a-b can be angled outward to provide a wider field of view. As long as the geometric relationship between each perspective camera 204a-b and the display panel 208a-b associated therewith is known, the perspective image frames 206a-b can be mapped to the virtual image frames 212a-b.

[0153] It should also be noted that the display panels 208a-b and their display lenses 210a-b need not have a fixed relationship. As noted above, each display panel 208a-b and its corresponding display lens 210a-b need not be fixed relative to one another. This can be accommodated by producing the mapping between each perspective camera and the virtual camera associated therewith, as well as between the virtual camera and the associated display panel.

[0154] It should also be noted that the functionality shown in or described with respect to Figures 2 to 13 may be implemented in any suitable manner in the electronic device 101, 102, 104, the server 106, or other device. For example, in some embodiments, at least some of the functionality shown in or described with respect to Figures 2 to 13 may be implemented or supported by other software instructions or one or more software applications executed by the processor(s) 120 of the other device, the server 106, or the electronic device 101, 102, 104. In other embodiments, at least some of the functionality shown in or described with respect to Figures 2 to 13 may be implemented or supported by dedicated hardware components that are not software instructions or software applications. Figures 2 to 13 In other embodiments, at least some of the functionality shown in or described with respect to Figures 2 to 13 may be implemented or supported by dedicated hardware components that are not software instructions or software applications. In general, the functionality shown in or described with respect to Figures 2 to 13 may be implemented using any suitable hardware or any suitable combination of hardware and software / firmware instructions. Moreover, Figures 2 to 13 the functionality shown in or described with respect to Figures 2 to 13 may be performed by a single device or by multiple devices. Figures 2 to 13 the functionality shown in or described with respect to Figures 2 to 13 may be performed by a single device or by multiple devices.

[0155] ​While the present disclosure has been described with example embodiments, various changes and modifications can be suggested to one skilled in the art. It is intended that the present disclosure encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method comprising: determining, using at least one processing device, that an interpupillary distance (IPD) between left and right display lenses of a video see-through (VST) extended reality (XR) device has been adjusted relative to a default IPD of the VST XR device; obtaining, using the at least one processing device, an image captured using a see-through camera of the VST XR device, the see-through camera configured to capture images of a three-dimensional (3D) scene; transforming, using the at least one processing device, the image to match a viewpoint of a corresponding one of the display lenses in accordance with the change in the IPD relative to the default IPD so as to generate a transformed image; correcting, using the at least one processing device, distortion in the transformed image based on one or more lens distortion coefficients corresponding to the change in the IPD so as to generate a corrected image; and initiating, using the at least one processing device, presentation of the corrected image on a display panel of the VST XR device.

2. The method of claim 1, wherein: when the image is captured using the left see-through camera, the corrected image is presented on a left display panel associated with a left eye of a user; and when the image is captured using the right see-through camera, the corrected image is presented on a right display panel associated with a right eye of the user.

3. The method of claim 1, wherein: the see-through camera represents a left see-through camera; the viewpoint corresponds to the left display lens; the display panel represents a left display panel associated with a left eye of a user; and the method further comprises: obtaining, using the at least one processing device, a second image captured using a right see-through camera of the VST XR device; transforming, using the at least one processing device, the second image to match a viewpoint of the right display lens so as to generate a second transformed image; correcting, using the at least one processing device, distortion in the second transformed image so as to generate a second corrected image; and initiating, using the at least one processing device, presentation of the second corrected image on the right display panel of the VST XR device, the right display panel associated with a right eye of the user.

4. The method of claim 1, wherein: the image is transformed to match the viewpoint of the corresponding one of the display lenses using the following equation:

5. The method of claim 1, wherein: IPD is adjusted to be 2 smaller than the default IPD ; and the image is transformed to match the viewpoint of the corresponding one of the display lenses using the following equation: wherein, and denotes the coordinates of a point of the image associated with a point in the 3D scene, and denotes the coordinates of a point projected onto the display panel, denotes the distance between a point in the 3D scene and the perspective camera, denotes the focal length of the perspective camera, and denotes the distance between the user’s eye and the perspective camera. the distortion in the transformed image is corrected using the following equation: IPD is adjusted to be 2 larger than default IPD ; and 7. The method of claim 1, wherein: wherein, and denote coordinates of points of the image associated with points in the 3D scene, and denote coordinates of points projected onto the display panel, denote the distance between points in the 3D scene and the perspective camera, denote the focal length of the perspective camera, and denote the distance between the user’s eyes and the perspective camera.

6. The method of claim 1, wherein, the display panel and the corresponding one of the display lenses are associated with one eye of a user; and wherein, represents one or more lens distortion coefficients, x and y represent coordinates on the display panel, and . transforming the image to match the viewpoint of the corresponding one of the display lenses comprises dynamically matching a principal point of the see-through camera to a principal point of the display panel. transforming the image to match the viewpoint of the corresponding one of the display lenses comprises mapping a see-through camera frame to a virtual camera frame so as to dynamically correct parallax errors.

9. The method of claim 1, wherein: the display panel and the corresponding one of the display lenses are associated with one eye of a user; and 8. The method of claim 1, wherein, correcting the distortion in the transformed image comprises: ​ ​ ​ ​ dynamically adapting one or more display lens geometric distortion and chromatic aberration models based on the change in IPD; and correcting display lens geometric distortion and chromatic aberration using the one or more adapted display lens geometric distortion and chromatic aberration models.

10. A video see-through (VST) extended reality (XR) device, comprising: left and right perspective cameras configured to capture images of a three-dimensional (3D) scene; a display panel configured to present virtual images; left and right display lenses; and at least one processing device configured to: determine that an interpupillary distance (IPD) between the left and right display lenses has been adjusted relative to a default IPD of the VST XR device; obtain a specified image among images captured using a specified one of the perspective cameras; transform the specified image to match a viewpoint of a corresponding one of the display lenses in accordance with the change in IPD relative to the default IPD so as to generate a transformed image; correct distortion in the transformed image based on one or more lens distortion coefficients corresponding to the change in IPD so as to generate a corrected image; and initiate presentation of the corrected image on the display panel. the at least one processing device is configured to:

11. The VST XR device of claim 10, wherein, initiate presentation of the corrected image on the left display panel associated with the user’s left eye when the specified image is captured using the left perspective camera; and initiate presentation of the corrected image on the right display panel associated with the user’s right eye when the specified image is captured using the right perspective camera.

12. The VST XR device of claim 10, wherein: the specified perspective camera represents the left perspective camera; the viewpoint corresponds to the left display lens; the display panel represents the left display panel associated with the user’s left eye; and the at least one processing device is further configured to: obtain a second image captured using the right perspective camera; transform the second image to match the viewpoint of the right display lens so as to generate a second transformed image; correct distortion in the second transformed image so as to generate a second corrected image; and initiate presentation of the second corrected image on the right display panel of the VST XR device, the right display panel being associated with the user’s right eye. To transform the image to match the viewpoint of the corresponding one of the display lenses, the at least one processing device is configured to dynamically match a principal point of the perspective camera to a principal point of the display panel.

13. The VST XR device of claim 10, wherein, 14. The VST XR device of claim 10, wherein: the display panel and the corresponding one of the display lenses are associated with one of the user’s eyes; and To correct distortion in the transformed image, the at least one processing device is configured to: dynamically adapt one or more display lens geometric distortion and chromatic aberration models based on the change in IPD; and correct display lens geometric distortion and chromatic aberration using the one or more adapted display lens geometric distortion and chromatic aberration models.

15. A non-transitory machine-readable medium containing instructions that, when executed, cause at least one processor of a video see-through (VST) extended reality (XR) device to: ​ determining that an interpupillary distance (IPD) between a left display lens and a right display lens of a VST XR device has been adjusted relative to a default IPD of the VST XR device; obtaining an image captured using a see-through camera of the VST XR device, the see-through camera configured to capture images of a three-dimensional (3D) scene; transforming the image to match a viewpoint of a corresponding one of the display lenses in accordance with the change in the IPD relative to the default IPD, so as to generate a transformed image; correcting for distortion in the transformed image based on one or more lens distortion coefficients corresponding to the change in the IPD, so as to generate a corrected image; and initiating presentation of the corrected image on a display panel of the VST XR device.

Images