Distortion combination and correction of final views in video perspective (VST) augmented reality (AR)

Abstract

A method may include obtaining an image captured by a perspective camera of a VST AR device. The method may include identifying a first distortion generated by at least one see-through camera lens, and identifying a second distortion generated by at least one display lens of the VST AR device. The method may include determining a combined distortion based on the first distortion and the second distortion. The method may include pre-warping the image to counteract the combined distortion such that the image is undistorted when a user views the pre-warped image through at least one display lens of the VST AR device. Further, the method may include presenting the pre-warped image to a user on at least one display of the VST AR device.

Description

Technical Field

[0001] The present disclosure generally relates to augmented reality (AR) systems and processing. More specifically, the present disclosure relates to distortion combination and correction of a final view in video see-through (VST) AR. Background Art

[0002] A video see-through (VST) augmented reality (AR) system can utilize a see-through camera mounted on a VST head-mounted device to access the environment around a user. The user can view mixed reality information of an image frame captured by the see-through camera and virtual objects generated by a graphics pipeline through at least one display lens. In a VST AR pipeline, one or more see-through camera lenses may introduce distortion in the captured image frame, and the display lens or lenses may introduce distortion when the user views the view rendered on the display through the display lens or lenses. Summary of the Invention

[0003] Technical Solution The present disclosure provides distortion combination and correction of a final view in video see-through (VST) augmented reality (AR).

[0004] In an embodiment, a method may include obtaining an image captured by a see-through camera of a VST AR device. The method may include identifying a first distortion introduced by at least one see-through camera lens, and identifying a second distortion introduced by at least one display lens of the VST AR device. The method may include determining a combined distortion based on the first distortion and the second distortion. The method may include pre-distorting the image to counteract the combined distortion such that the image is distortion-free when the user views the pre-distorted image through the at least one display lens of the VST AR device. Additionally, the method may include presenting the pre-distorted image to the user on at least one display of the VST AR device.

[0005] In an embodiment, a VST AR device may include at least one display, at least one see-through camera, and at least one processing device. The at least one display may include at least one display lens. The at least one see-through camera may include at least one see-through camera lens and may be configured to capture an image. The at least one processing device may be configured to obtain the image captured by the at least one see-through camera, identify a first distortion generated by the at least one see-through camera lens, and identify a second distortion generated by the at least one display lens. The at least one processing device may be configured to determine a combined distortion based on the first distortion and the second distortion. The at least one processing device may be configured to pre-distort the image to cancel out the combined distortion such that when a user views the pre-distorted image through the at least one display lens, the image is not distorted. Additionally, the at least one processing device may be configured to present the pre-distorted image to the user on the at least one display.

[0006] In an embodiment, a non-transitory machine-readable medium may contain instructions which, when executed, cause at least one processor to obtain an image captured by a see-through camera of a VST AR device. The non-transitory machine-readable medium may contain instructions which, when executed, cause the at least one processor to identify a first distortion generated by at least one see-through camera lens and identify a second distortion generated by at least one display lens of the VST AR device. The non-transitory machine-readable medium may contain instructions which, when executed, cause the at least one processor to determine a combined distortion based on the first distortion and the second distortion. The non-transitory machine-readable medium may contain instructions which, when executed, further cause the at least one processor to pre-distort the image to cancel out the combined distortion such that when a user views the pre-distorted image through the at least one display lens of the VST AR device, the image is not distorted. Additionally, the non-transitory machine-readable medium may contain instructions which, when executed, cause the at least one processor to present the pre-distorted image to the user on at least one display of the VST AR device. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 Shows an example network configuration including an electronic device according to the present disclosure; Figure 2A Shows an example process for distortion combination and correction of a final view in video see-through (VST) augmented reality (AR) according to the present disclosure; Figure 2B Shows an example process for distortion combination and correction of a final view in video see-through (VST) augmented reality (AR) according to the present disclosure; Figure 2CIllustrates an example process for distortion combination and correction of a final view in video see-through (VST) augmented reality (AR) according to the present disclosure; Figure 2D Illustrates an example process for distortion combination and correction of a final view in video see-through (VST) augmented reality (AR) according to the present disclosure; Figure 2E Illustrates an example process for distortion combination and correction of a final view in video see-through (VST) augmented reality (AR) according to the present disclosure; Figure 2F Illustrates an example process for distortion combination and correction of a final view in video see-through (VST) augmented reality (AR) according to the present disclosure; Figure 3 Illustrates an example camera matrix and lens distortion model according to the present disclosure; Figure 4 Illustrates an example distortion combination for displaying lens geometric distortion and chromatic aberration correction; and Figure 5 Illustrates an example method for distortion combination and correction of a final view in VST AR according to the present disclosure. DETAILED DESCRIPTION

[0008] The following description is described with reference to the accompanying drawings Figures 1 to 5 and various embodiments of the present disclosure. However, it should be understood that the present disclosure is not limited to these embodiments, and all changes and / or equivalents or substitutions thereof also fall within the scope of the present disclosure.

[0009] As described above, a video see-through (VST) augmented reality (AR) system can utilize a perspective camera mounted on a VST head-mounted device to access the environment around the user. The user can view mixed reality information of an image frame captured by the perspective camera and virtual objects generated by a graphics pipeline through at least one display lens. In a VST AR pipeline, one perspective camera lens or multiple perspective camera lenses may introduce distortion in the captured image frame, and the display lens or multiple display lenses may introduce distortion when the user views the view rendered on the display through the display lens(s). For example, geometric distortion and chromatic aberration from the perspective camera and the display lens may distort the image presented to the user. Unfortunately, distortion correction typically results in latency issues in the VST AR pipeline.

[0010] The present disclosure provides techniques for distortion combination and correction for a final view in VST AR. As described in more detail below, an image captured by a perspective camera of a VST AR device may be obtained. A first distortion generated by at least one perspective camera lens may be identified, and a second distortion generated by at least one display lens of the VST AR device may be identified. A combined distortion may be determined based on the first distortion and the second distortion. The image may be pre-distorted to cancel the combined distortion such that when a user views the pre-distorted image through at least one display lens of the VST AR device, the image is not distorted. The pre-distorted image may be presented to the user on at least one display of the VST AR device.

[0011] In this way, the distortion from the perspective camera lens and the distortion from the display lens may be combined to help compensate for geometric distortion, chromatic aberration, or other distortion in the final view of the image. This may be performed while reducing latency issues compared to other methods. Additionally, in an embodiment, a distortion grid configured to compensate for the combined distortion may be used, such as when geometric distortion and chromatic aberration correction may be performed during an initialization phase or other phase of a VST AR pipeline based on the distortion grid. This may help simplify and accelerate the correction process. Further, the final view of the scene may be rendered directly from the image captured by the perspective camera based on the distortion grid, which may again accelerate the process.

[0012] Figure 1 An example network configuration 100 including an electronic device is shown in accordance with the present disclosure. In accordance with an embodiment of the present disclosure, an electronic device 101 is included in the network configuration 100. The electronic device 101 may include at least one of a bus 110, a processor 120, a memory 130, an input / output (I / O) interface 150, a display 160, a communication interface 170, and a sensor 180. In an embodiment, the electronic device 101 may exclude at least one of these components, or may add at least one other component. The bus 110 includes circuitry for connecting the components 120 - 180 to each other and for transferring communications (such as control messages and / or data) between the components.

[0013] The processor 120 includes one or more processing devices, such as one or more microprocessors, microcontrollers, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs). In an embodiment, the processor 120 includes one or more of a central processing unit (CPU), an application processor (AP), a communication processor (CP), or a graphics processing unit (GPU). The processor 120 is capable of performing control on at least one of the other components of the electronic device 101 and / or performing operations or data processing related to communication or other functions. As described below, the processor 120 may obtain an input image and process the input image to perform distortion combination and correction of a final view to support at least one VST AR application, as described in more detail below.

[0014] The memory 130 may include volatile memory and / or non-volatile memory. For example, the memory 130 may store commands or data related to at least one of the other components of the electronic device 101. According to an embodiment of the present disclosure, the memory 130 may store software and / or a program 140. The program 140 includes, for example, a kernel 141, middleware 143, an application programming interface (API) 145, and / or an application program (or “app”) 147. At least a part of the kernel 141, middleware 143, or API 145 may be represented as an operating system (OS).

[0015] The kernel 141 may control or manage system resources (such as the bus 110, the processor 120, or the memory 130) for performing operations or functions implemented in other programs, such as the middleware 143, the API 145, or the app 147. The kernel 141 provides an interface that allows the middleware 143, the API 145, or the app 147 to access various components of the electronic device 101 to control or manage system resources. The app 147 may include one or more apps, where the one or more apps, among other things, obtain an input image and process the input image to perform distortion combination and correction of a final view to support at least one VST AR application. These functions may be performed by a single app or multiple apps, where each app performs one or more of these functions. For example, the middleware 143 may act as a forwarder to allow the API 145 or the app 147 to communicate data with the kernel 141. Multiple apps 147 may be provided. The middleware 143 is capable of controlling work requests received from the app 147, such as by assigning priorities for using system resources of the electronic device 101, such as the bus 110, the processor 120, or the memory 130, to at least one of the multiple apps 147. The API 145 is an interface that allows the app 147 to control functions provided from the kernel 141 or the middleware 143. For example, the API 145 includes at least one interface or function (such as a command) for file control, window control, image processing, or text control.

[0016] The I / O interface 150 serves as an interface that can transfer, for example, commands or data input from a user or other external devices to other components of the electronic device 101. The I / O interface 150 can also output commands or data received from other components of the electronic device 101 to the user or other external devices.

[0017] The display 160 includes, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum dot light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display 160 can also be a depth perception display, such as a multi-focus display. The display 160 is capable of displaying various contents (such as text, images, videos, icons, or symbols) to the user. The display 160 can include a touch screen and can receive, for example, touch, gesture, proximity, or hover inputs using an electronic pen or a body part of the user.

[0018] For example, the communication interface 170 is capable of establishing communication between the electronic device 101 and an external electronic device (such as the first electronic device 102, the second electronic device 104, or the server 106). For example, the communication interface 170 can be connected to the network 162 or 164 through wireless or wired communication to communicate with the external electronic device. The communication interface 170 can be a wired or wireless transceiver or any other component for transmitting and receiving signals.

[0019] Wireless communication can use, for example, at least one of WiFi, Long Term Evolution (LTE), Long Term Evolution - Advanced (LTE-A), Fifth Generation Wireless System (5G), millimeter wave or 60 GHz wireless communication, Wireless USB, Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Wireless Broadband (WiBro), or Global System for Mobile Communications (GSM) as a cellular communication protocol. The wired connection can include, for example, at least one of Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI), Recommended Standard 232 (RS-232), or Plain Old Telephone Service (POTS). The network 162 or 164 includes at least one communication network, such as a computer network (such as a Local Area Network (LAN) or a Wide Area Network (WAN)), the Internet, or a telephone network.

[0020] The electronic device 101 also includes one or more sensors 180, which can measure a physical quantity or detect an activation state of the electronic device 101 and convert the measured or detected information into an electrical signal. For example, the sensor(s) 180 includes one or more cameras or other imaging sensors that can be used to capture an image of a scene. The sensor(s) 180 may also include one or more buttons for touch input, one or more microphones, a gesture sensor, a gyroscope or gyro sensor, a barometric pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (such as a red green blue (RGB) sensor), a biophysical sensor, a temperature sensor, a humidity sensor, an illuminance sensor, an ultraviolet (UV) sensor, an electromyogram (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an infrared (IR) sensor, an ultrasonic sensor, an iris sensor, or a fingerprint sensor. In addition, the sensor(s) 180 may include an inertial measurement unit, which may include one or more accelerometers, gyroscopes, and other components. In addition, the sensor(s) 180 may include a control circuit for controlling at least one of the sensors included herein. Any one of these sensors 180 may be located within the electronic device 101.

[0021] In an embodiment, the electronic device 101 may be a wearable device or a wearable device (such as an HMD) on which an electronic device can be mounted. For example, the electronic device 101 may represent an AR wearable device, such as a head-mounted device or smart glasses. In an embodiment, the first external electronic device 102 or the second external electronic device 104 may be a wearable device or a wearable device (such as an HMD) on which an electronic device can be mounted. In an embodiment, when the electronic device 101 is mounted in the electronic device 102 (such as an HMD), the electronic device 101 may communicate with the electronic device 102 through the communication interface 170. The electronic device 101 may be directly connected to the electronic device 102 to communicate with the electronic device 102 without involving a separate network.

[0022] The first external electronic device 102, the second external electronic device 104, and the server 106 may each be a device of the same type or a different type from the electronic device 101. According to an embodiment of the present disclosure, the server 106 may include a group of one or more servers. In addition, according to an embodiment of the present disclosure, all operations or some operations performed on the electronic device 101 may be performed on another electronic device or multiple other electronic devices (such as the electronic device 102 and the electronic device 104 or the server 106). In addition, according to an embodiment of the present disclosure, when the electronic device 101 is supposed to automatically perform some functions or services or perform some functions or services upon request, the electronic device 101 may request another device (such as the electronic device 102 and the electronic device 104 or the server 106) to perform at least some functions associated therewith, rather than performing the function or service by itself, or in addition to performing the function or service by itself, the electronic device 101 may also request another device (such as the electronic device 102 and the electronic device 104 or the server 106) to perform at least some functions associated therewith. Another electronic device (such as the electronic device 102 and the electronic device 104 or the server 106) is capable of performing the requested function or additional functions and transmitting the execution result to the electronic device 101. The electronic device 101 may provide the requested function or service by processing the received result as it is or additionally processing the received result. For this purpose, for example, cloud computing technology, distributed computing technology, or client-server computing technology may be used. Although Figure 1 FIG. shows that the electronic device 101 includes a communication interface 170 that communicates with the external electronic device 104 or the server 106 via the network 162 or the network 164, but according to an embodiment of the present disclosure, the electronic device 101 may be independently operated without a separate communication function.

[0023] The server 106 may include components (or a suitable subset of components) that are the same as or similar to those of the electronic device 101. The server 106 may support driving the electronic device 101 by executing at least one of the operations (or functions) implemented on the electronic device 101. For example, the server 106 may include a processing module or a processor that can support the processor 120 implemented in the electronic device 101. As described below, the server 106 may obtain an input image and process the input image to perform distortion combination and correction of the final view in order to support at least one VST AR application, as described in more detail below.

[0024] Although Figure 1 FIG. shows an example of the network configuration 100 including the electronic device 101, but various changes may be made to Figure 1 it. For example, the network configuration 100 may include any number of each component in any suitable arrangement. Generally, computing and communication systems have a wide variety of configurations, and Figure 1Do not limit the scope of the present disclosure to any particular configuration. Additionally, while Figure 1 an operating environment is shown in which various features disclosed in this patent document can be used, these features can be used in any other suitable system.

[0025] Figures 2A to 2F An example process 200 for distortion combination and correction of a final view in VST AR according to the present disclosure is shown. For ease of explanation, Figure 2A process 200 is described as being performed using Figure 1 electronic device 101. However, process 200 can be performed using any other suitable device(s) and can be used in any other suitable system(s).

[0026] As Figure 2A shown, process 200 can include an initialization phase 201 and an online rendering phase 202. The initialization phase 201 generally operates to determine one or more transformations to be applied to pre-distort an image to be displayed to a user, and the online rendering phase 202 generally operates to apply the transformation(s) to the image and present the transformed image on at least one display panel 220. The at least one display panel 220 can represent one or more displays 160 of the electronic device 101. Note that multiple separate display panels 220 (such as a left display panel and a right display panel that can be separately viewed by a user's eyes) or a single display panel 220 (such as a display panel where the left and right portions of the display panel can be separately viewed by a user's eyes) can be used to present the rendered view.

[0027] The initialization phase 201 includes a camera model operation 203 and a display lens model operation 204 for estimating the distortion produced by at least one perspective camera 221 and at least one display lens 222, respectively. The at least one perspective camera 221 can represent one or more imaging sensors 180 of the electronic device 101. The at least one display lens 222 can represent one or more lenses that form part of the at least one display panel 220.

[0028] During the camera model operation 203, the electronic device 101 can perform a perspective camera calibration operation 205 to determine the intrinsic and extrinsic parameters of one or more perspective camera lenses 223 of the perspective camera(s) 221. For example, the perspective camera calibration operation 205 can be used to generate a camera matrix and a distortion model, where the camera matrix and the distortion model can be used to determine camera distortion. The perspective camera calibration operation 205 can use any suitable technique(s) to identify the intrinsic and extrinsic parameters of the perspective camera lens(es) 223.

[0029] As part of the camera model operation 203, the electronic device 101 may perform a camera lens distortion calculation operation 206 using the intrinsic and extrinsic parameters of the perspective camera lens(es) 223. For example, the camera lens distortion calculation operation 206 may identify perspective camera lens distortion 224 associated with at least one perspective camera lens 223, such as barrel distortion and pincushion distortion. As shown in Figure 2C , the perspective camera lens distortion 224 may transform the scene 225 into a first distorted image 226. The first distorted image 226 is how the scene appears to the optical sensor of the perspective camera 221 when viewed through the perspective camera lens 223.

[0030] During the display lens model operation 204, the electronic device 101 may perform a display lens calibration operation 207 to determine the intrinsic and extrinsic parameters of the display lens(es) 222. The display lens calibration operation 207 may use any suitable technique(s) to identify the intrinsic and extrinsic parameters of the display lens(es) 222. As part of the display lens model operation 204, the electronic device 101 may perform a display lens distortion calculation operation 208 using the intrinsic and extrinsic parameters of the display lens(es) 222. For example, the display lens distortion calculation operation 208 may identify display lens distortion 227 associated with at least one display lens 222, such as barrel distortion and pincushion distortion. In an embodiment, the display lens distortion calculation operation 208 may generate an inverse display lens distortion 228, where the inverse display lens distortion 228 may be used to correct the display lens distortion 227 of the display lens(es) 222.

[0031] The initialization phase 201 also includes various operations 209 - 215 that operate based on the outputs of the camera model operation 203 and the display lens model operation 204. For example, the electronic device 101 may perform a render mesh generation operation 209 to generate a render mesh. The render mesh provides the ability to map an image and simplify the transformation applied to the image. A camera lens distortion application operation 210 may be performed by the electronic device 101 to apply the camera lens distortion to the render mesh to generate a first distorted mesh 229. As shown in Figure 2D , the viewpoint matching and disparity correction transformation operation 211 may transform the first distorted mesh 229 into a second distorted mesh 232.

[0032] The first distorted mesh 229 may be provided to the viewpoint matching and disparity correction transformation operation 211 that may modify the first distorted mesh 229. For example, as shown in Figure 2DAs shown, the view point matching and disparity correction transformation operation 211 can be performed to match one or more camera viewpoints 230 and one or more rendering viewpoints 231. As an example, the view point matching and disparity correction transformation operation 211 can generate one or more transformations that help match each camera viewpoint 230 with the corresponding rendering viewpoint 231. These transformations can be based on one or more transformations identified by the view point matching and disparity correction transformation operation 211. Since the camera and the user's eyes cannot occupy the same physical space, the images captured by the camera can be modified to correct for depth and / or field of view differences with the rendering viewpoint 231.

[0033] The view point matching and disparity correction transformation operation 211 can also be performed to identify adjustments that may be needed to remove view artifacts in the overlapping regions between multiple perspective cameras 221. For example, since the camera viewpoints 230 are not the same as the rendering viewpoints 231, objects at different depths may be distorted if equally adjusted. As an example, objects that are closer to the viewpoint in depth can be adjusted at a greater rate than objects at a greater distance or the background of the real-world scene. The view point matching and disparity correction transformation operation 211 can determine the differences in viewing objects in the 3D space based on the line of sight, and the view point matching and disparity correction transformation operation 211 can be used to correct these differences. For example, the view point matching and disparity correction transformation operation 211 can correct objects in the rendered image that are affected by disparity in the real-world scene.

[0034] The electronic device 101 can perform a combined distortion operation 212 to combine the distortion generated by at least one perspective camera lens 223 and the distortion generated by at least one display lens 222. For example, the combined distortion operation 212 can combine the second distortion grid 232, the perspective camera lens distortion 224, and the reverse display lens distortion 228 into a combined distortion grid 233. The combined distortion grid 233 provides a single set of transformations to be applied to the initial image of the scene captured by the perspective camera (instead of separate sets of transformations for separately considering the distortion generated by the perspective camera lens 223 and the display lens 222).

[0035] The electronic device 101 can perform pre-warped grid generation 213 by applying the combined distortion grid 233 to the rendering grid in order to generate, as Figure 2BThe pre-distortion 234 shown in [Fig.]. The pre-distortion 234 can be applied to an image to transform the image into a final view 235 for viewing through an associated display lens 222. Among other things, the pre-distortion 234 can reduce the amount of image processing, thereby reducing latency. In an example, the electronic device 101 can perform a distortion calculation operation 214 to determine the distortion in each of a plurality of color channels (such as in each of a red color channel, a green color channel, and a blue color channel). For example, the distortion grid can be pre-distorted with a combined distortion to compensate for the display lens geometric distortion in the green color channel, which is a geometric distortion correction for the final view 235. The distortions in the red and blue color channels can be combined to generate a pre-distortion grid as a compensation for the distortions in the red and blue color channels.

[0036] In the chromatic aberration calculation operation 215, the chromatic aberration can be determined by using the distortion differences between, for example, the red and green color channels and between the blue and green color channels. Here, for the convenience and efficiency of correction, such as in a GPU shader of a rendering pipeline, the geometric distortion and the chromatic aberration can be separated. The static distortion grid generated by transformation using camera distortion, viewpoint matching, display lens geometric distortion correction, and chromatic aberration correction can be prepared for rendering the final view 235.

[0037] During the online rendering stage 202, the electronic device 101 uses perspective cameras to capture images in the image capture operation 216. For example, multiple perspective cameras 221 can respectively perform the image capture operation 216 to generate at least a first image and a second image. In an embodiment, the first image and the second image can be the left and right image captures from the left and right perspective cameras 221. In the virtual view rendering operation 217, the electronic device 101 directly renders one or more final views 235 from the captured images and the pre-distorted static distortion grid generated in the initialization stage 201. For example, the virtual view rendering operation 217 can render the final views 235 for display on at least one display panel (such as the display panel(s) 220). In an example, the electronic device 101 can display a first rendered image on a first panel and a second rendered image on a second panel, or the electronic device 101 can display the first rendered image and the second rendered image on different parts of a single panel. Since the static grid has been transformed with camera distortion, viewpoint matching, display lens geometric distortion correction, and chromatic aberration correction in the initialization stage 201, these transformations can reduce the latency in online rendering. Finally, in the virtual view display operation 218, the rendered final views 235 can be displayed on the display panel 220 for the user to view. Depending on the implementation, the virtual view display operation 218 can be pre-rendered or dynamically rendered. When combined distortion is applied to the captured images, the user can view the final views 235 with reduced or minimal distortion.

[0038] It should be noted that Figures 2A to 2F shown or described for Figures 2A to 2F The functions described can be implemented in any suitable manner in the electronic device 101, electronic device 102, electronic device 104, server 106, or other device(s). For example, in an embodiment, one or more software applications or other software instructions executed by the processors 120 of the electronic device 101, electronic device 102, electronic device 104, server 106, or other device(s) can be used to implement or support the functions shown or described for Figures 2A to 2F shown or described for Figures 2A to 2F at least some of the functions described. In other embodiments, dedicated hardware components can be used to implement or support at least some of the functions shown or described for Figures 2A to 2F shown or described for Figures 2A to 2F at least some of the functions described. Generally, any suitable hardware or any suitable combination of hardware and software / firmware instructions can be used to perform Figures 2A to 2F shown or described for Figures 2A to 2F the functions described.

[0039] Although Figures 2A to 2FShows an example of the process 200 for distortion combination and correction for the final view in VST AR, but various changes can be made to Figures 2A to 2F For example, although shown as a series of operations, the various operations in Figures 2A to 2F can overlap, occur in parallel, occur in a different order, or occur any number of times. In addition, Figures 2A to 2F the various components and functions in can be combined, further subdivided, replicated, or rearranged according to specific needs. In addition, if desired or expected, one or more additional components and functions can be included.

[0040] Figure 3 Shows an example camera matrix and lens distortion model 300 according to the present disclosure. The camera matrix and lens distortion model 300 represents here an example of the distortion that may be produced by a specific perspective camera lens 223, which was described above for Figures 2A to 2F and for which pre-distortion can be performed for compensation. As shown in Figure 3 , when light passes through the perspective camera lens 223, the regular grid is distorted. In the example, the camera lens changes the grid from a regular rectangular shape to a barrel distortion shape.

[0041] For a fish-eye lens, a fish-eye lens distortion model can be used to determine geometric distortion. For example, 3D points from the world coordinate system can be transformed into the corresponding points Pc(Xc, Yc, Zc) in the camera coordinate system in a manner such as the following.

[0042] (1) Here, [R t] represents the camera extrinsic characteristics, where R represents the rotation matrix and t represents the translation vector. In an embodiment, the image point can be determined using a pinhole projection in a manner such as the following.

[0043] (2) According to equation (2), the following can be obtained.

[0044] (3) Here, r represents the radius of distortion, and represents the angle of distortion. Based on this, the fish-eye distortion can be expressed as follows.

[0045] (4) Here, k represents the fish-eye distortion. Thus, the distorted point coordinates can be determined as follows.

[0046] (5) In addition, the pixel coordinates can be determined as follows:​ (6) Here, represents the principal point of the camera.

[0047] For a non - fisheye lens, the radial distortion can be determined as follows.

[0048] (7) Here, represents the distorted coordinates, represents the undistorted coordinates, and represents the radial distortion coefficient. In an embodiment, the radius used to identify the radial distortion can be determined as follows.

[0049] (8) In addition, the tangential distortion can be determined as follows.

[0050] (9) Here, represents the tangential distortion coefficient. Based on this, the distortion model can be defined as follows.

[0051] (10) Here, D c represents the distortion model. In addition, the camera matrix can be defined as follows.

[0052] (11) Here, ( ) represents the focal length of the camera, and represents the principal point of the camera.

[0053] When light passes through the display lens 222, the regular grid is also distorted, which can change the grid from a regular rectangular shape to a pincushion - distorted shape. In an embodiment, the display lens distortion can be determined as follows.

[0054] (12) Here, represents the distorted coordinates, represents the undistorted coordinates, and represents the radial distortion coefficient.

[0055] Although Figure 3 shows an example of the camera matrix and the lens distortion model 300, various changes can be made to Figure 3 . For example, the specific mathematical expressions described above are for illustration only and can be changed as needed or desired.

[0056] Figure 4An example distortion combination 400 for displaying lens geometric distortion and chromatic aberration correction according to the present disclosure is shown. The distortion combination 400 can be used, for example, to generate the static grid as described above. As Figure 4 shown, the distortion combination 400 is used to correct the geometric distortion and chromatic aberration from the perspective camera lens 223 and the display lens 222.

[0057] For example, the first color channel can include one of a green color channel, a red color channel, and a blue color channel. For example, the second color channel can include one of a red color channel, a blue color channel, and a green color channel. For example, the third color channel can include one of a blue color channel, a red color channel, and a green color channel.

[0058] In an example, during the mesh transformation process, the green distortion mesh 401 for the green color channel can be pre-distorted to provide geometric distortion correction. The distortion of the red color channel is determined, and the red distortion difference 402 between the red color channel and the green color channel can be determined. Similarly, the distortion of the blue color channel can be determined, and the blue distortion difference 403 between the blue color channel and the green color channel can be determined. Using the red distortion difference 402 and the blue distortion difference 403, the meshes for the red color channel and the blue color channel can be pre-distorted to obtain the red distortion mesh 404 for the red color channel and the blue distortion mesh 405 for the blue color channel.

[0059] To effectively implement the shader, for example, the green distortion mesh 401 of the green color channel is used as a basis, and the red distortion difference 402 and the blue distortion difference 403 can be used in comparison with the green distortion mesh 401. In this way, it is not necessary to transfer all three distortion meshes of the three color channels to the GPU shader. Instead, in an embodiment, only the green distortion mesh 401 of the green color channel and the red distortion difference 402 and the blue distortion difference 403 of the red color channel and the blue color channel are transferred to the GPU shader for rendering. With the pre-distorted meshes 401, 404, and 405 of the green color channel, the red color channel, and the blue color channel rendered and displayed on the display panel 220, the pre-distorted distortion can compensate for the distortion generated by the display lens 222, so that the user can see the regular-shaped meshes of the red color channel 406, the blue color channel 407, and the green color channel, which correct the display lens geometric distortion and chromatic aberration.

[0060] Although Figure 4 an example of the distortion combination 400 for displaying lens geometric distortion and chromatic aberration correction is shown, various changes can be made to Figure 4 it. For example, the above specific distortions and color channels are for illustration only and can be changed as needed or desired.

[0061] Figure 5 Illustrates an example method 500 for distortion combination and correction of a final view in VST AR. For ease of explanation, Figure 5 method 500 is described as being performed using Figure 1 electronic device 101 in network configuration 100. However, method 500 can use any other suitable device(s) and be performed in any other suitable system(s).

[0062] As Figure 5 shown, at step 502, the VST device obtains an image captured by a perspective camera. This can include, for example, the processor 120 of the electronic device 101 receiving an image from the perspective camera 221 of the electronic device 101, where the perspective camera 221 can include a perspective camera lens 223. The perspective camera lens 223 can distort the image of the actual scene based on the intrinsic characteristics of the perspective camera lens 223. At step 504, the VST device identifies a first distortion generated by the perspective camera lens of the perspective camera. This can include, for example, the processor 120 of the electronic device 101 identifying a perspective camera lens distortion 224 generated by the perspective camera lens 223 of the perspective camera 221. When the sensor of the perspective camera 221 views the image through the perspective camera lens 223, a first distortion can be generated based on the intrinsic characteristics of the perspective camera lens 223. At step 506, the VST device identifies a second distortion generated by the display lens. This can include, for example, the processor 120 of the electronic device 101 identifying a display lens distortion 227 generated by the display lens 222. When projecting an image from the display panel 220 through the display lens 222, a second distortion can be generated based on the geometric distortion and chromatic aberration characteristics of the display lens 222.

[0063] At step 508, the VST device determines a combined distortion. This can include, for example, the processor 120 of the electronic device 101 determining a combined distortion based on the combination of the first distortion of the perspective camera lens 223 and the second distortion of the display lens 222. The second distortion of the display lens 222 can be reversed to be combined with the first distortion of the perspective camera lens 223. The distortion and chromatic aberration can be determined based on the combined distortion. At step 510, the VST device pre - distorts the image to cancel the combined distortion. This can include, for example, the processor 120 of the electronic device 101 pre - distorting the image in such a way that the image is undistorted when the user views the pre - distorted image through at least one display lens of the VST AR device to cancel the combined distortion. At step 512, the VST device presents the pre - distorted image. This can include, for example, the processor 120 of the electronic device 101 displaying the pre - distorted image on at least one display panel 220 of the VST AR device to view the image without distortion through the display lens 222.

[0064] In an embodiment, the pre - distortion performed at step 510 includes retrieving a specified distortion grid for a first color channel, where the specified distortion grid is configured to compensate for combined distortion. Access a first specified distortion difference to obtain a difference between the first color channel and the second color channel, and access a second specified distortion difference to obtain a difference between the first color channel and the third color channel. The specified distortion grid can be applied to pre - distort the first color channel of the image, and the first specified distortion difference and the second specified distortion difference can be applied separately with the specified distortion grid to pre - distort the second color channel and the third color channel of the image. The first specified distortion difference and the second specified distortion difference can be applied instead of applying a second distortion grid and a third distortion grid to pre - distort the second color channel and the third color channel of the image. Additionally, in an embodiment, the transformation can also be implemented in a GPU shader, which allows for convenient integration with other dynamic transformations such as depth reprojection and temporal warping, and the GPU shader implementation can be extended to both static and dynamic processing. Further, in an embodiment, the distortion of a display lens distortion model can be determined to compensate only for the distortion of the display lens for geometric distortion and chromatic aberration correction. In an embodiment, a camera undistortion operation can be added to remove camera distortion, and the final display distortion correction may not need to be related to the camera distortion calculation.

[0065] Although Figure 5 shows an example of method 500 for distortion combination and correction for a final view in VST AR, various changes can be made to Figure 5 it. For example, although shown as a series of steps, Figure 5 the individual steps in

[0066] it can overlap, occur in parallel, occur in a different order, or occur any number of times.

[0067] According to an embodiment of the present disclosure, a method performed by an electronic device may include: obtaining an image captured by a perspective camera of a video - see - through (VST) augmented reality (AR) device.

[0068] According to an embodiment of the present disclosure, a method performed by an electronic device may include: identifying a first distortion generated by at least one perspective camera lens.

[0069] According to an embodiment of the present disclosure, a method performed by an electronic device may include: identifying a second distortion generated by at least one display lens of a VST AR device.

[0070] According to an embodiment of the present disclosure, a method performed by an electronic device may include: determining combined distortion based on first distortion and second distortion.

[0071] According to an embodiment of the present disclosure, a method performed by an electronic device may include: pre-distorting an image to cancel out the combined distortion such that when a user views the pre-distorted image through at least one display lens of a VST AR device, the image is not distorted.

[0072] According to an embodiment of the present disclosure, a method performed by an electronic device may include: presenting the pre-distorted image to a user on at least one display of the VST AR device.

[0073] According to an embodiment of the present disclosure, a method performed by an electronic device may include: retrieving a specified distortion grid configured to compensate for the combined distortion, wherein the specified distortion grid is associated with a first color channel.

[0074] According to an embodiment of the present disclosure, a method performed by an electronic device may include: applying the specified distortion grid to pre-distort the first color channel of the image.

[0075] According to an embodiment of the present disclosure, a method performed by an electronic device may include: accessing a first specified distortion difference between a first color channel and a second color channel.

[0076] According to an embodiment of the present disclosure, a method performed by an electronic device may include: applying the specified distortion grid and the first specified distortion difference to pre-distort the second color channel of the image.

[0077] According to an embodiment of the present disclosure, a method performed by an electronic device may include: accessing a second specified distortion difference between a first color channel and a third color channel.

[0078] According to an embodiment of the present disclosure, a method performed by an electronic device may include: applying the specified distortion grid and the second specified distortion difference to pre-distort the third color channel of the image.

[0079] According to an embodiment of the present disclosure, the first specified distortion difference instead of the distortion grid for the second color channel may be passed to a shader.

[0080] According to an embodiment of the present disclosure, the combined distortion may be based on one or more intrinsic characteristics of at least one perspective camera lens.

[0081] According to an embodiment of the present disclosure, the combined distortion is based on one or more geometric distortion and chromatic aberration characteristics of at least one display lens.

[0082] According to an embodiment of the present disclosure, a video see-through (VST) augmented reality (AR) device may include: at least one display including at least one display lens.

[0083] According to an embodiment of the present disclosure, a video see-through (VST) augmented reality (AR) device may include: at least one see-through camera including at least one see-through camera lens and configured to capture an image.

[0084] According to an embodiment of the present disclosure, a video see-through (VST) augmented reality (AR) device may include at least one processor.

[0085] According to an embodiment of the present disclosure, the at least one processor is configured to obtain an image captured by the at least one see-through camera.

[0086] According to an embodiment of the present disclosure, the at least one processor is configured to identify a first distortion generated by the at least one see-through camera lens.

[0087] According to an embodiment of the present disclosure, the at least one processor is configured to identify a second distortion generated by the at least one display lens.

[0088] According to an embodiment of the present disclosure, the at least one processor is configured to determine a combined distortion based on the first distortion and the second distortion.

[0089] According to an embodiment of the present disclosure, the at least one processor is configured to pre-distort the image to cancel the combined distortion such that when a user views the pre-distorted image through the at least one display lens, the image is not distorted.

[0090] According to an embodiment of the present disclosure, the at least one processor is configured to present the pre-distorted image to the user on the at least one display.

[0091] According to an embodiment of the present disclosure, the at least one processor is configured to retrieve a specified distortion grid configured to compensate for the combined distortion, wherein the specified distortion grid is associated with a first color channel.

[0092] According to an embodiment of the present disclosure, the at least one processor is configured to apply the specified distortion grid to pre-distort the first color channel of the image.

[0093] According to an embodiment of the present disclosure, the at least one processor is configured to access a first specified distortion difference between the first color channel and the second color channel.

[0094] According to an embodiment of the present disclosure, the at least one processor is configured to apply the specified distortion grid and the first specified distortion difference to pre-distort the second color channel of the image.

[0095] According to an embodiment of the present disclosure, the at least one processor is configured to access a second specified distortion difference between the first color channel and the third color channel.

[0096] According to an embodiment of the present disclosure, the at least one processor is configured to apply a specified distortion grid and a second specified distortion difference to pre-warp a third color channel of an image.

[0097] According to an embodiment of the present disclosure, the at least one processor is configured to pass a first specified distortion difference instead of the distortion grid used for a second color channel to a shader.

[0098] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to obtain an image captured by a perspective camera of a video see-through (VST) augmented reality (AR) device.

[0099] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to identify a first distortion generated by at least one perspective camera lens.

[0100] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to identify a second distortion generated by at least one display lens of a VST AR device.

[0101] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to determine a combined distortion based on the first distortion and the second distortion.

[0102] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to pre-warp an image to cancel out the combined distortion such that when a user views the pre-warped image through the at least one display lens of the VST AR device, the image is not distorted.

[0103] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to present the pre-warped image to a user on the at least one display of the VST AR device.

[0104] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to retrieve a specified distortion grid configured to compensate for the combined distortion, wherein the specified distortion grid is associated with a first color channel.

[0105] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to apply a specified distortion grid to pre-distort a first color channel of an image.

[0106] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to access a first specified distortion difference between a first color channel and a second color channel.

[0107] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to apply the specified distortion grid and the first specified distortion difference to pre-distort a second color channel of the image.

[0108] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to access a second specified distortion difference between a first color channel and a third color channel.

[0109] According to an embodiment of the present disclosure, a machine-readable medium includes at least one instruction, wherein the at least one instruction, when executed, causes the at least one processor to apply the specified distortion grid and the second specified distortion difference to pre-distort a third color channel of the image.

Claims

1. A method, comprising: Obtaining an image captured by a perspective camera of a video see-through (VST) augmented reality (AR) device; Identifying a first distortion generated by at least one perspective camera lens; Identifying a second distortion generated by at least one display lens of the VST AR device; Determining a combined distortion based on the first distortion and the second distortion; Pre-distorting the image to cancel the combined distortion such that when a user views the pre-distorted image through the at least one display lens of the VST AR device, the image is not distorted; and Presenting the pre-distorted image to the user on at least one display of the VST AR device.

2. The method according to claim 1, wherein, The operation of pre-distorting the image comprises: Retrieving a specified distortion grid configured to compensate the combined distortion, wherein the specified distortion grid is associated with a first color channel; and Applying the specified distortion grid to pre-distort the first color channel of the image.

3. The method according to claim 2, wherein The operation of pre-distorting the image further comprises: Accessing a first specified distortion difference between the first color channel and a second color channel; and Applying the specified distortion grid and the first specified distortion difference to pre-distort the second color channel of the image.

4. The method according to claim 3, wherein The operation of pre-distorting the image further comprises: Accessing a second specified distortion difference between the first color channel and a third color channel; and Applying the specified distortion grid and the second specified distortion difference to pre-distort the third color channel of the image.

5. The method according to any one of claims 3 and 4, wherein The first specified distortion difference rather than a distortion grid for the second color channel is passed to a shader.

6. The method according to any one of claims 1 to 5, wherein The combined distortion is based on one or more intrinsic characteristics of the at least one perspective camera lens.

7. The method according to any one of claims 1 to 6, wherein, The combined distortion is based on one or more geometric distortion and chromatic aberration characteristics of the at least one display lens.

8. A video see-through (VST) augmented reality (AR) device, comprising: At least one display including at least one display lens; At least one perspective camera including at least one perspective camera lens and configured to capture an image; And At least one processing device configured to: Obtain an image captured by the at least one perspective camera; Identify a first distortion generated by the at least one perspective camera lens; Identify a second distortion generated by the at least one display lens; Determine a combined distortion based on the first distortion and the second distortion; Pre-distort the image to cancel the combined distortion such that when a user views the pre-distorted image through the at least one display lens, the image is not distorted; and Present the pre-distorted image to the user on the at least one display.

9. The VST AR device according to claim 8, wherein, To pre-distort the image, the at least one processing device is configured to: Retrieve a specified distortion grid configured to compensate the combined distortion, wherein the specified distortion grid is associated with a first color channel; and Apply the specified distortion grid to pre-distort the first color channel of the image.

10. The VST AR device according to claim 9, wherein, To pre-distort the image, the at least one processing device is further configured to: Access a first specified distortion difference between the first color channel and a second color channel; and Pre - warp the second color channel of the image using the specified distortion grid and the first specified distortion difference.

11. The VST AR device according to claim 10, wherein, To pre - warp the image, the at least one processing device is further configured to: Access a second specified distortion difference between the first color channel and the third color channel; and Pre - warp the third color channel of the image using the specified distortion grid and the second specified distortion difference.

12. The VST AR device according to any one of claims 10 and 11, wherein, The at least one processing device is configured to pass the first specified distortion difference instead of the distortion grid for the second color channel to the shader.

13. The VST AR device according to any one of claims 8 to 12, wherein, The combined distortion is based on one or more intrinsic characteristics of the at least one perspective camera lens.

14. The VST AR device according to any one of claims 8 to 13, wherein, The combined distortion is based on one or more geometric distortion and chromatic aberration characteristics of the at least one display lens.

15. A machine-readable medium comprising instructions, wherein, When executed, the instructions cause at least one processor to implement the method according to any one of claims 1 to 7.

Images