Method and apparatus for multi-lens distortion correction

Abstract

The present disclosure relates to methods and apparatuses for display or graphics processing. Aspects of the present disclosure can calculate first position information based on a first image grid. Additionally, aspects of the present disclosure can determine a first UV image based on the calculated first position information. Further, aspects of the present disclosure can calculate second position information based on a second image grid and the determined first UV image. Aspects of the present disclosure can also determine a second UV image based on the calculated second position information. Aspects of the present disclosure can also generate a third image grid based on the determined second UV image. Additionally, aspects of the present disclosure can calculate third position information based on the third image grid and a camera image. Aspects of the present disclosure can also determine a display image based on the calculated third position information.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims the benefit of the following applications: U.S. Provisional Application No. 62 / 914,379, filed on October 11, 2019, with the title "METHODS AND APPARATUS FOR MULTIPLE LENS DISTORTION CORRECTION"; and U.S. Patent Application No. 16 / 813,534, filed on March 9, 2020, with the title "METHODS AND APPARATUS FOR MULTIPLE LENS DISTORTION CORRECTION", which are hereby incorporated by reference in their entirety. Technical Field

[0003] In general, the present disclosure relates to processing systems, and more particularly, to one or more techniques for display or graphics processing. Background Art

[0004] Computing devices typically utilize a Graphics Processing Unit (GPU) to accelerate the rendering of graphical data for display. Such computing devices can include, for example, computer workstations, mobile phones such as so - called smart phones, embedded systems, personal computers, tablet computers, and video game consoles. The GPU executes a graphics processing pipeline, which includes one or more processing stages that operate together to execute graphics processing commands and output frames. The Central Processing Unit (CPU) can control the operation of the GPU by issuing one or more graphics processing commands to the GPU. Modern CPUs are typically capable of concurrently executing multiple applications, each of which may need to utilize the GPU during its execution. Devices that provide content for visual presentation on a display typically include a GPU.

[0005] Generally, the GPU of a device is configured to execute processes in the graphics processing pipeline. However, with the emergence of wireless communication and smaller handheld devices, the demand for improved graphics processing has been increasing. Summary of the Invention

[0006] A simplified overview of one or more aspects is given below in order to provide a basic understanding of such aspects. This overview is not an exhaustive review of all contemplated aspects, and is neither intended to identify key elements of all aspects nor to delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0007] In one aspect of the present disclosure, a method, a computer-readable medium, and a device are provided. The device may be a headset, a head-mounted display (HMD), a client device, a server, a display processing unit, a display processor, a central processing unit (CPU), a graphics processing unit (GPU), or any device capable of performing display or graphics processing. The device may determine first image information based on a first image. Additionally, the device may generate a first image grid based on the determined first image information. The device may also determine second image information based on a second image. Additionally, the device may generate a second image grid based on the determined second image information. The device may further calculate first position information based on the first image grid. Additionally, the device may determine a first horizontal (U) vertical (V) (UV) image based on the calculated first position information. Further, the device may calculate second position information based on the second image grid and the determined first UV image. The device may also determine a second UV image based on the calculated second position information. The device may further generate a third image grid based on the determined second UV image. The device may also calculate third position information based on the third image grid and a camera image. The device may also determine a display image based on the calculated third position information. The device may further calculate length (X) height (Y) depth (Z) (XYZ) position information based on the first position information. The device may also determine a first XYZ image based on the calculated XYZ position information.

[0008] Details of one or more examples of the present disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the present disclosure will be apparent from the specification, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram illustrating an example content generation system in accordance with one or more techniques of the present disclosure.

[0010] Figure 2 illustrates an example GPU in accordance with one or more techniques of the present disclosure.

[0011] Figure 3 illustrates an example diagram for lens distortion correction in accordance with one or more techniques of the present disclosure.

[0012] Figure 4 illustrates an example diagram for lens distortion correction in accordance with one or more techniques of the present disclosure.

[0013] Figure 5 illustrates an example diagram for lens distortion correction in accordance with one or more techniques of the present disclosure.

[0014] Figure 6 An example diagram for lens distortion correction according to one or more techniques of the present disclosure is shown.

[0015] Figure 7 An example diagram for lens distortion correction according to one or more techniques of the present disclosure is shown.

[0016] Figure 8 An example flowchart of an example method according to one or more techniques of the present disclosure is shown. DETAILED DESCRIPTION

[0017] Lens distortion correction can be applied to multiple lenses, where each lens distorts the input image. Some aspects of lens distortion correction can utilize double pass rendering for distortion correction. In double pass rendering, a camera image can be used as the input image. However, in an augmented reality (AR) context, double pass rendering may result in long rendering times and / or increased latency. This may also result in increased GPU texture memory access, which can be costly. Aspects of the present disclosure can utilize single pass rendering, where a single three-dimensional (3D) rendering pass can simultaneously perform multiple lens distortion corrections. Such single pass rendering can provide multiple advantages, such as faster rendering times and / or reduced latency. This can also result in less GPU data bandwidth usage. Additionally, this may result in reduced GPU computation and / or reduced GPU texture memory access per vertex and pixel. Single pass rendering can also result in improved or sharper image quality.

[0018] Aspects of systems, devices, computer program products, and methods are described more fully hereinafter with reference to the accompanying drawings. However, the present disclosure may be embodied in many different forms and should not be construed as limited to any specific structure or function presented throughout this disclosure. Rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. Based on the teachings herein, those skilled in the art should appreciate that the scope of the present disclosure is intended to cover any aspect of the systems, devices, computer program products, and methods disclosed herein, whether implemented independently of or in combination with other aspects of the present disclosure. For example, a device may be implemented using any number of the aspects described herein, or a method may be implemented. Additionally, the scope of the present disclosure is intended to cover such a device or method implemented using another structure, function, or combination of structures and functions other than or different from the aspects of the present disclosure described herein. Any aspect disclosed herein may be implemented by one or more elements of a claim.

[0019] Although various aspects are described herein, many variations and permutations of these aspects fall within the scope of the present disclosure. Although some potential benefits and advantages of aspects of the present disclosure are mentioned, the scope of the present disclosure is not intended to be limited to particular benefits, uses, or objectives. Rather, aspects of the present disclosure are intended to be widely applicable to different wireless technologies, system configurations, networks, and transmission protocols, some of which are illustrated by way of example in the figures and the following description. The detailed description and the figures are merely illustrative of the present disclosure and not restrictive, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0020] Certain aspects will now be presented with reference to various devices and methods. These devices and methods will be described in the following detailed description and illustrated in the figures by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether an element is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system.

[0021] For example, an element, or any part of an element, or any combination of elements can be implemented as a "processing system" that includes one or more processors (which may also be referred to as processing units). Examples of processors include: microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on a chip (SoCs), baseband processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described throughout this disclosure. One or more processors in the processing system can execute software. Whether referred to as software, firmware, middleware, microcode, hardware description language, or other names, software should be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, processes, functions, etc. The term application can refer to software. As described herein, one or more techniques can refer to an application configured to perform one or more functions, i.e., software. In such an example, the application can be stored in a memory, e.g., on-chip memory of a processor, system memory, or any other memory. The hardware described herein (e.g., a processor) can be configured to execute the application. For example, the application can be described as including code that, when executed by the hardware, causes the hardware to perform one or more techniques described herein. As an example, the hardware can access the code from the memory and execute the code accessed from the memory to perform one or more techniques described herein. In some examples, components are identified in this disclosure. In such examples, a component can be hardware, software, or a combination thereof. A component can be a separate component or a sub-component of a single component.

[0022] Accordingly, in one or more example configurations, the described functionality can be implemented in hardware, software, or any combination thereof. If implemented in software, the functionality can be stored on a computer-readable medium or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media. Storage media can be any available media that can be accessed by a computer. By way of example and not limitation, such computer-readable media can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, combinations of the above types of computer-readable media, or any other medium capable of storing computer-executable code in the form of instructions or data structures that can be accessed by a computer.

[0023] Generally speaking, the present disclosure describes the following technologies: the technologies are used to have a graphics processing pipeline in a single device or multiple devices, improve the rendering of graphics content, and / or reduce the load on a processing unit (i.e., any processing unit configured to execute one or more of the technologies described herein (e.g., GPU)). For example, the present disclosure describes technologies for graphics processing in any device that utilizes graphics processing. Other example benefits are described throughout the present disclosure.

[0024] As used herein, instances of the term "content" may refer to "graphics content", "image", and vice versa. This is true regardless of whether these terms are used as adjectives, nouns, or other parts of speech. In some examples, as used herein, the term "graphics content" may refer to content generated by one or more processes of a graphics processing pipeline. In some examples, as used herein, the term "graphics content" may refer to content generated by a processing unit configured to perform graphics processing. In some examples, as used herein, the term "graphics content" may refer to content generated by a graphics processing unit.

[0025] In some examples, as used herein, the term "display content" may refer to content generated by a processing unit configured to perform display processing. In some examples, as used herein, the term "display content" may refer to content generated by a display processing unit. Graphics content can be processed into display content. For example, a graphics processing unit may output graphics content such as a frame to a buffer (which may be referred to as a frame buffer). A display processing unit may read the graphics content (e.g., one or more frames) from the buffer and perform one or more display processing techniques on it to generate display content. For example, the display processing unit may be configured to perform composition on one or more rendering layers to generate a frame. As another example, the display processing unit may be configured to composite, blend, or otherwise combine two or more layers into a single frame. The display processing unit may be configured to perform scaling on the frame (e.g., upsampling or downsampling). In some examples, a frame may refer to a layer. In other examples, a frame may refer to two or more layers that have been blended together to form a frame, i.e., a frame includes two or more layers, and a frame including two or more layers may subsequently be blended.

[0026] Figure 1FIG. 0 is a block diagram illustrating an example content generation system 100 configured to implement one or more techniques of the present disclosure. The content generation system 100 includes a device 104. The device 104 may include one or more components or circuits for performing the various functions described herein. In some examples, one or more components of the device 104 may be components of a system-on-a-chip (SOC). The device 104 may include one or more components configured to perform one or more techniques of the present disclosure. In the example shown, the device 104 may include a processing unit 120, a content encoder / decoder 122, and a system memory 124. In some aspects, the device 104 may include a plurality of optional components, such as a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131. References to the display 131 may refer to one or more displays 131. For example, the display 131 may include a single display or multiple displays. The display 131 may include a first display and a second display. The first display may be a left-eye display, and the second display may be a right-eye display. In some examples, the first and second displays may receive different frames for presentation thereon. In other examples, the first and second displays may receive the same frames for presentation thereon. In additional examples, the results of graphics processing may not be displayed on the device, e.g., the first and second displays may not receive any frames for presentation thereon. Instead, the frames or the results of graphics processing may be transmitted to another device. In some aspects, this may be referred to as split rendering.

[0027] The processing unit 120 may include an internal memory 121. The processing unit 120 may be configured to perform graphics processing, such as in a graphics processing pipeline 107. The content encoder / decoder 122 may include an internal memory 123. In some examples, the device 104 may include a display processor (e.g., display processor 127) for performing one or more display processing techniques on one or more frames generated by the processing unit 120 before presentation by one or more displays 131. The display processor 127 may be configured to perform display processing. For example, the display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by the processing unit 120. One or more displays 131 may be configured to display or otherwise present the frames processed by the display processor 127. In some examples, one or more displays 131 may include one or more of the following: a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head-mounted display, or any other type of display device.

[0028] Memory external to processing unit 120 and content encoder / decoder 122 (e.g., system memory 124) may be accessible to processing unit 120 and content encoder / decoder 122. For example, processing unit 120 and content encoder / decoder 122 may be configured to read from and / or write to an external memory such as system memory 124. Processing unit 120 and content encoder / decoder 122 may be communicatively coupled to system memory 124 via a bus. In some examples, processing unit 120 and content encoder / decoder 122 may be communicatively coupled to each other via a bus or different connections.

[0029] Content encoder / decoder 122 may be configured to receive graphical content from any source such as system memory 124 and / or communication interface 126. System memory 124 may be configured to store the received encoded or decoded graphical content. Content encoder / decoder 122 may be configured to receive, for example, encoded or decoded graphical content in the form of encoded pixel data from system memory 124 and / or communication interface 126. Content encoder / decoder 122 may be configured to encode or decode any graphical content.

[0030] Internal memory 121 or system memory 124 may include one or more volatile or non-volatile memory or storage devices. In some examples, internal memory 121 or system memory 124 may include RAM, static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media or optical storage media, or any other type of memory.

[0031] According to some examples, internal memory 121 or system memory 124 may be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagated signal. However, the term "non-transitory" should not be construed to mean that internal memory 121 or system memory 124 is non-removable or that its content is static. As an example, system memory 124 may be removed from device 104 and moved to another device. As another example, system memory 124 may be non-removable from device 104.

[0032] The processing unit 120 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), or any other processing unit that can be configured to perform graphics processing. In some examples, the processing unit 120 may be integrated into the motherboard of the device 104. In some examples, the processing unit 120 may be present on a graphics card in a port installed in the motherboard of the device 104, or may otherwise be incorporated within a peripheral device configured to interoperate with the device 104. The processing unit 120 may include one or more processors, e.g., one or more microprocessors, GPUs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the techniques are implemented partially in software, the processing unit 120 may store instructions for the software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 121), and may execute the instructions in hardware using one or more processors to perform the techniques of the present disclosure. Any of the foregoing including hardware, software, a combination of hardware and software, etc. may be considered one or more processors.

[0033] The content encoder / decoder 122 may be any processing unit configured to perform content decoding. In some examples, the content encoder / decoder 122 may be integrated into the motherboard of the device 104. The content encoder / decoder 122 may include one or more processors, e.g., one or more microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the techniques are implemented partially in software, the content encoder / decoder 122 may store instructions for the software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 123), and may execute the instructions in hardware using one or more processors to perform the techniques of the present disclosure. Any of the foregoing including hardware, software, a combination of hardware and software, etc. may be considered one or more processors.

[0034] In some aspects, the content generation system 100 may include an optional communication interface 126. The communication interface 126 may include a receiver 128 and a transmitter 130. The receiver 128 may be configured to perform any of the receiving functions described herein with respect to device 104. Additionally, the receiver 128 may be configured to receive information from another device, such as, for example, eye or head position information, rendering commands, or location information. The transmitter 130 may be configured to perform any of the sending functions described herein with respect to device 104. For example, the transmitter 130 may be configured to send information that may include a request for content to another device. The receiver 128 and the transmitter 130 may be combined into a transceiver 132. In such an example, the transceiver 132 may be configured to perform any of the receiving functions and / or sending functions described herein with respect to device 104.

[0035] Referring again to Figure 1 , in certain aspects, the graphics processing pipeline 107 may include a determination component 198 configured to determine first image information based on a first image. The determination component 198 may also be configured to generate a first image grid based on the determined first image information. The determination component 198 may also be configured to determine second image information based on a second image. The determination component 198 may also be configured to generate a second image grid based on the determined second image information. The determination component 198 may also be configured to calculate first position information based on the first image grid. The determination component 198 may also be configured to determine a first UV image based on the calculated first position information. The determination component 198 may also be configured to calculate second position information based on the second image grid and the determined first UV image. The determination component 198 may also be configured to determine a second UV image based on the calculated second position information. The determination component 198 may also be configured to generate a third image grid based on the determined second UV image. The determination component 198 may also be configured to calculate third position information based on the third image grid and a camera image. The determination component 198 may also be configured to determine a display image based on the calculated third position information. The determination component 198 may also be configured to calculate length (X) height (Y) depth (Z) (XYZ) position information based on the first position information. The determination component 198 may also be configured to determine a first XYZ image based on the calculated XYZ position information.

[0036] As described herein, a device such as device 104 can refer to any device, apparatus, or system configured to perform one or more of the techniques described herein. For example, a device can be a server, a base station, a user equipment, a client device, a station, an access point, a computer (e.g., a personal computer, a desktop computer, a laptop computer, a tablet computer, a computer workstation, or a mainframe computer), an end product, a device, a telephone, a smartphone, a server, a video game platform or console, a handheld device (e.g., a portable video game device, or a personal digital assistant (PDA)), a wearable computing device (e.g., a smartwatch, an augmented reality device or a virtual reality device), a non-wearable device, a display or display device, a television, a television set-top box, an intermediate network device, a digital media player, a video streaming device, a content streaming device, an in-vehicle computer, any mobile device, any device configured to generate graphical content, or any device configured to perform one or more of the techniques described herein. The processes herein can be described as being performed by specific components (e.g., a GPU), but in other embodiments, other components (e.g., a CPU) consistent with the disclosed embodiments can be used to perform them.

[0037] A GPU can process multiple types of data or data packets in a GPU pipeline. For example, in some aspects, a GPU can process two types of data or data packets, e.g., context register packets and draw call data. A context register packet can be a collection of global state information (e.g., information about global registers, shader programs, or constant data) that can condition how a graphics context will be processed. For example, a context register packet can include information about a color format. In some aspects of a context register packet, there can be bits indicating which workload belongs to the context register. Additionally, there can be multiple functions or programs running simultaneously and / or in parallel. For example, a function or program can describe some operation, such as a color mode or a color format. Thus, a context register can define multiple states of a GPU.

[0038] The context state can be used to determine how a separate processing unit (e.g., a vertex fetcher (VFD), a vertex shader (VS), a shader processor, or a geometry processor) operates and / or in what mode the processing unit operates. To this end, a GPU can use context registers and programming data. In some aspects, a GPU can generate workloads (e.g., vertex or pixel workloads) in a pipeline based on the context register definitions of modes or states. Certain processing units (e.g., a VFD) can use these states to determine certain functions, e.g., how to assemble vertices. Since these modes or states can change, a GPU may need to change the corresponding context. Additionally, the workloads corresponding to the modes or states can follow the changing modes or states.

[0039] Figure 2 illustrates an example GPU 200 according to one or more techniques of the present disclosure. As Figure 2 shown, the GPU 200 includes a command processor (CP) 210, a draw call packet 212, a VFD 220, a VS 222, a vertex cache (VPC) 224, a triangle setup engine (TSE) 226, a rasterizer (RAS) 228, a Z process engine (ZPE) 230, a pixel interpolator (PI) 232, a fragment shader (FS) 234, a render backend (RB) 236, an L2 cache (UCHE) 238, and a system memory 240. Although Figure 2 shown that the GPU 200 includes processing units 220 - 238, the GPU 200 may include multiple additional processing units. Additionally, the processing units 220 - 238 are merely examples, and according to the present disclosure, the GPU may use any combination or order of processing units. The GPU 200 also includes a command buffer 250, a context register packet 260, and a context state 261.

[0040] As Figure 2 shown, the GPU may utilize the CP (e.g., CP 210) or a hardware accelerator to parse the command buffer into a context register packet (e.g., context register packet 260) and / or a draw call data packet (e.g., draw call packet 212). Then, the CP 210 may send the context register packet 260 or the draw call data packet 212 to a processing unit or block in the GPU via separate paths. Additionally, the command buffer 250 may change the different states of the context registers and draw calls. For example, the command buffer may be constructed in the following manner: context registers of context N, draw calls of context N, context registers of context N + 1, and draw calls of context N + 1.

[0041] The GPU may render images in a variety of different ways. In some cases, the GPU may use immediate mode rendering or tiled rendering to render images. In a tiled rendering GPU, an image may be divided or segmented into different parts or tiles. After dividing the image, each part or tile may be rendered individually. A tiled rendering GPU may divide a computer graphics image into a grid format such that each part of the grid (i.e., a tile) is rendered individually. In some aspects, during the binned path, the image may be divided into different bins or tiles. Additionally, in the binned path, different primitives may be shaded in certain bins, e.g., using draw calls. In some aspects, during the binned path, a visibility stream may be constructed, where visible primitives or drawing calls may be identified.

[0042] In some aspects of rendering, there can be multiple processing stages or passes. For example, rendering can be performed in two passes, such as a visibility pass and a rendering pass. During the visibility pass, the GPU can ingest a rendering workload, record the positions of primitives or triangles, and then determine which primitives or triangles fall within which portions of the frame. In some aspects of the visibility pass, the GPU can also identify or tag the visibility of each primitive or triangle in the visibility stream. During the rendering pass, the GPU can ingest the visibility stream and process a portion of the frame at a time. In some aspects, the visibility stream can be analyzed to determine which primitives are visible or invisible. Thus, the visible primitives can be processed. By doing so, the GPU can reduce the workload of processing or rendering invisible primitives that are unnecessary.

[0043] Aspects of the present disclosure can be applied to a variety of different types of content, such as virtual reality (VR) content, augmented reality (AR) content, and / or extended reality (XR) content. In VR content, the content displayed at the user device can correspond to augmented content, such as content presented on a server or the user device. In AR or XR content, a portion of the content displayed at the user device can correspond to real-world content, such as an object in the real world, and a portion of the content can be augmented content. Additionally, the augmented content and the real-world content can be displayed in an optical see-through or video see-through device such that the user can view the real-world object and the augmented content simultaneously.

[0044] In some aspects, in VR camera perspective mode, a VR headset can be used as an AR headset. For example, two external cameras pointing outwards can capture real-world objects, such as in the left and right camera images. The left and right camera images can also be displayed on an internal LCD or OLED display. Additionally, a pair of display lenses can be mounted in front of the display. Then, the user can view the external world via the display lenses, the display, and / or the external cameras. In some aspects, the external cameras can be referred to as tracking cameras because they can track images in the real world. Thus, the images displayed on the display or the client device can be the images tracked by the tracking cameras. Thus, the real-world images can be tracked by the camera lenses and then transmitted to the display and / or the display lenses. Additionally, the lenses may bend the light, which can cause distortion.

[0045] In some cases, because there are multiple lenses, each lens may distort the input image. Lens distortion correction can be applied to multiple lenses so that straight lines can remain straight as seen by the user. Thus, lens distortion correction can attempt to ensure that straight lines are presented to the user as straight lines rather than as curved or distorted lines due to lens distortion. Additionally, each lens may have a different distortion. Further, the order of lens distortion correction may be important because reversing the order of lens distortion may result in different corrected images. Lens distortion can also include a variety of different specifications. For example, lens distortion correction can be mobile platform friendly, lens distortion correction can be fast, such as 60 frames per second or more, and lens distortion correction can utilize the GPU for performance.

[0046] Some aspects of the present disclosure can utilize some aspects of dual-pass rendering. In dual-pass rendering, the camera image can be used as the input image. The first rendering pass can create and use a 3D mesh and / or an application programming interface (API) or OpenGL rendering to render the camera image with camera lens distortion correction, i.e., straight lines can be straight. Aspects of the present disclosure can refer to this image as the corrected camera image. Additionally, in the rendering pass, the present disclosure can obtain the input image and then bend the image to create a new image.

[0047] The second rendering pass can create and use a 3D mesh and / or an API or OpenGL rendering to render the display image with display lens distortion correction, i.e., straight lines may be curved. Aspects of the present disclosure can refer to this image as the pre-corrected display image. The pre-corrected display image can be displayed on the display, and then the image can be distorted by the display lens. Additionally, the user can observe the corrected display image, where the straight lines can be straight again. Thus, aspects of the present disclosure can start with straight lines and then bend the lines in the direction opposite to the distortion of the display lens. Thus, if the display lens bends the image to the left, the present disclosure can bend the image to the right to counteract the display lens, and the image can appear straight.

[0048] Figure 3 FIG. 300 illustrates one or more techniques in accordance with the present disclosure. As Figure 3 shown, FIG. 300 includes camera calibration data 310, display lens calibration data 312, three-dimensional (3D) mesh 320, 3D mesh 322, camera image 330, first rendering 340, corrected camera image 350, second rendering 342, pre-corrected display image 352, display lens 360, and display image 370. Figure 3 FIG. 300 illustrates one aspect of lens distortion correction in accordance with the present disclosure.

[0049] In some aspects, the camera image 330 can be from a tracking camera that appears to include a fisheye lens distortion. The image can be calibrated, which can include information about the amount by which the image is warped or blurred. This information can also be used to create a 3D mesh 320, which can include transformation data or a way to perform an image transformation. The 3D mesh 320 can perform a pinch effect on the camera image 330 such that the fisheye lens distortion on the image 330 can be eliminated.

[0050] As Figure 3 shown, the camera calibration data 310 can be used to create a 3D mesh 320, and the 3D mesh 320 can be combined with the camera image 330 for a first rendering 340. The first rendering 340 can be used to obtain a corrected camera image 350. After obtaining the corrected camera image 350, the present disclosure can bulge the image for a pinch at a pre-corrected display end (e.g., a display lens 360). Additionally, the display lens calibration data 312 can be used to create a 3D mesh 322, and the 3D mesh 322 can be combined with the corrected camera image 350 to produce a second rendering 342. The second rendering 342 can then result in a pre-corrected display image 352 that can bulge the image. The pre-corrected display image 352 can result in the display lens 360, and the display lens 360 can ultimately result in a display image 370 that a user views.

[0051] As described above, in dual-pass rendering, the camera image can be used as an input image. However, in an AR context, dual-pass rendering can result in increased rendering time as well as increased latency. This can also result in increased GPU texture memory access, which can be costly. Thus, there is a current need for a rendering process that can allow for faster rendering time and / or reduced latency.

[0052] Some aspects of the present disclosure can utilize single-pass rendering in order to address the above disadvantages of dual-pass rendering. In single-pass rendering, aspects of the present disclosure can use a single 3D rendering to simultaneously perform the above two distortion corrections. The single-pass rendering can provide multiple advantages such as faster rendering time and / or reduced latency in an AR context. Single-pass rendering can also result in less GPU data bandwidth usage for 3D mesh vertices and image pixel data. Additionally, single-pass rendering can result in reduced GPU computation per vertex and pixel. Furthermore, single-pass rendering can result in reduced GPU texture memory access, which can be costly.

[0053] Single-pass rendering can also result in improved or sharper image quality. For example, each texture sample fetch can perform bilinear interpolation between adjacent pixels. Additionally, a weighted average of pixels can result in a blurring effect on the fetched color samples. By reducing the number of texture fetches, aspects of the present disclosure can reduce or eliminate blurring.

[0054] Some aspects of single-pass rendering can be difficult to solve because image distortion can be both non-linear and non-commutative. Non-linear image distortion can mean that some pixels in the image are transformed in different ways. Non-commutative image distortion can mean that swapping the order of the distortions or performing the distortions simultaneously may not provide the same result as performing the distortions in one order.

[0055] In addition, image transformations that produce different effects (e.g., bulging and / or pinching the image) can be located in different parts of the image. This can make the image transformation difficult to predict. Also, the order of the image transformations is important. In some aspects, non-commutative transformations can include a variety of different distortions, e.g., rotational distortion or barrel distortion. Thus, depending on whether the rotational distortion is performed before the barrel distortion (or vice versa), the resulting image transformation may be different.

[0056] Figure 4 FIG. 400 illustrates one or more techniques in accordance with the present disclosure. As Figure 4 shown, FIG. 400 includes camera calibration data 410, display lens calibration data 412, a 3D mesh 420, a camera image 430, a first rendering 440, a pre-corrected display image 452, a display lens 460, and a display image 470. Figure 4 FIG. 400 illustrates one aspect of lens distortion correction in accordance with the present disclosure.

[0057] As Figure 2 shown, single-pass rendering in accordance with the present disclosure can combine camera calibration data 410 with display lens calibration data 412. This combination can produce a 3D mesh 420. The mesh 420 can be combined with the camera image 430 to produce a first rendering 440. The first rendering 440 can be used to obtain a pre-corrected display image 452. The pre-corrected display image 452 can result in a display lens 460, which can ultimately result in a display image 470 that a user views.

[0058] In some aspects, single-pass rendering in accordance with the present disclosure can generate a separate 3D mesh from camera calibration data and display lens calibration data. The mesh from the camera calibration data can be used to calculate first position information. Additionally, a first horizontal (U) vertical (V) (UV) image or map can be determined based on the calculated first position information. In some aspects, calculating the first position information can correspond to rendering UV information as red (R), green (G), blue (B) (RGB) color information.

[0059] In some aspects, creating a UV image or map can be a 3D modeling process of projecting a two-dimensional (2D) image onto the surface of a 3D model for texture mapping. For example, a UV image or map can be a set of UV texture coordinates of the vertices of a mesh, which correspond to the 3D or XYZ position coordinates of the vertices of the mesh. For example, a UV image or map can be a set of two-dimensional points that match each 3D or XYZ vertex.

[0060] Additionally, a UV image can be a color image, where the red (R) channel of a pixel corresponds to the value of the horizontal (U) texture coordinate, the green (G) channel of the pixel corresponds to the value of the vertical (V) texture coordinate, and the blue (B) channel of the pixel can be set to zero. In some aspects, a UV image can map 3D triangle points to the 2D pixels of an image that can be used in texture mapping, i.e., wrap the image around the 3D model. Additionally, a UV image can be referred to as a UV map, a UV rendered image, or a UV lookup texture.

[0061] A first UV image can be combined with a display lens calibration data grid to calculate second position information. In some aspects, calculating the second position information can correspond to rendering input texture coordinate information. Then a second UV image can be determined based on the second position information. Additionally, a third image grid can be determined based on the determined second UV image. The third image grid can be combined with a color camera image to generate additional position information, and then the final image that the user observes at the display can be generated.

[0062] In some aspects, when generating the third image grid, certain geometric positions can be converted into coordinate information. For example, three-dimensional (XYZ) positions can be converted into UV texture coordinates. Additionally, in some aspects, a checkerboard texture can be used to render the aforementioned UV image.

[0063] Figure 5 FIG. 500 is shown in accordance with one or more techniques of the present disclosure. As Figure 5 shown, FIG. 500 includes camera calibration data or first image information 510, display lens calibration data or second image information 512, first grid 520, second grid 522, first position information 530, second position information 532, first UV image 540, second UV image 542, third grid 550, camera image 560, third position information 562, and display image 570. Figure 5 An aspect of single-pass rendering for lens distortion correction in accordance with the present disclosure is shown. As Figure 5 shown, the image from the camera is located to the right of the dashed line. For example, an image taken by a user's head-mounted headset or head-mounted display (HMD). Additionally, the portion to the left of the dashed line can be a one-time preprocessing, and the portion to the right of the dashed line can be a per-frame runtime processing. InFigure 5 In this case, each image information (e.g., the first image information 510 or the second image information 512) can be associated with a different lens. For example, in Figure 5 this case, there can be two lenses (e.g., the first lens and the second lens), which respectively correspond to the first image information 510 and the second image information 512. However, aspects of the present disclosure are not limited to a specific number of lenses, e.g., two lenses. As will be described below, the present disclosure can utilize four lenses and any other appropriate number of lenses, e.g., eight or sixteen lenses. In addition, these multiple lenses may not need to be perfectly aligned with each other.

[0064] Figure 5 It is shown that the first grid 520 can be based on the first image information 510. In addition, the first position information 530 can be based on the first grid 520, and the first UV image 540 can be based on the first position information 530. The second grid 522 can be based on the second image information 512. In addition, the second position information 532 can be based on the second grid 522 and the first UV image 540. In addition, the second UV image 542 can be based on the second position information 532. The third grid 550 can be based on the second UV image 542. In addition, the third position information 562 can be based on the camera image 560 and the third grid 550. Finally, the display image 570 can be based on the third position information 562.

[0065] As Figure 5 shown, when rendering a 3D grid (e.g., the first grid 520 or the second grid 522) instead of generating an image, the present disclosure can calculate the position information of each pixel based on the 3D grid. Thus, aspects of the present disclosure can calculate the position information of each pixel from one or more grids. Additionally, aspects of the present disclosure can determine a horizontal (U) vertical (V) (UV) image or a UV lookup texture based on this position information.

[0066] As described above, the present disclosure can determine a 3D grid based on an image. When the image is mapped, the present disclosure can assign position coordinates to each pixel in the image. For example, a horizontal (U) value (e.g., from 0 to 1) and a vertical (V) value (e.g., from 0 to 1) can be assigned to each pixel. Aspects of the present disclosure can also defer searching or looking up an image generated by a camera. For example, each time an image is searched or looked up, the image quality may be degraded. Thus, aspects of the present disclosure can avoid including intermediate images in the distortion solution.

[0067] In some cases, the aforementioned rendering can produce an image where the color represents the position of a pixel from the camera image, e.g., UV texture coordinates. For example, these UV coordinate values can correspond to color coordinate values. Thus, these coordinates can be addresses that can be from the image and allow determination of information for each pixel. Additionally, the dimensions and aspect ratio can correspond to the area of a display used to show the camera image.

[0068] Additionally, multiple meshes can be calculated, which can include vertex positions and / or texture coordinates. As described above, the mesh can modify or change the input image. Additionally, application programming interface (API) rendering or OpenGL rendering can be a texture map lookup operation. In some aspects, the input texture image may be distorted and output. The present disclosure can use high-precision (e.g., 32-bit) floating-point texture samples and / or an output render buffer. Additionally, a UV image or map can be used as a texture to be used with another UV image or Figure 1 texture to be used.

[0069] In some aspects, in a second render pre-pass, the UV texture coordinates from a first lens correction can be, for example, the final texture coordinates for each pixel, which can allow vertices to determine their texture coordinates. Thus, the image distortion process can be performed multiple times on the UV image or map. In some aspects, this process can be performed via rendering of optical distortion through multiple lenses. For example, the aforementioned image distortion process can mitigate distortion from one lens to another.

[0070] Additionally, aspects of the present disclosure can construct a third 3D mesh and its corresponding texture coordinates, e.g., as a final runtime render mesh. In some aspects, the third mesh can be determined based on coordinates of previous pixel information or UV information. Thus, the third mesh can be calculated based on the position or color information of previous data. Additionally, the position information can be XYZ coordinates or UV coordinates. The geometric or XYZ positions can also be evenly spaced.

[0071] Furthermore, the UV texture coordinates can be determined by looking up values in a previously generated UV lookup texture or image. The present disclosure can also consider sampling between pixels, e.g., by using bilinear interpolation. In some aspects, the present disclosure can generate curved UV texture coordinates. Additionally, the present disclosure can generate curved XYZ coordinates.

[0072] Additionally, aspects of the present disclosure can utilize a variety of different processes, such as rendering the position of pixels and / or creating pixel lookup coordinates as the image pixel output value. In these aspects, the color of the pixels can be rendered instead of their position. Aspects of the present disclosure can also use the position image as an input texture for another distortion process, for example, to further bend and / or recalculate the final position of the desired output pixels. Additionally, aspects of the present disclosure can use the final position to retrieve, for example, the color of the camera input image all at once.

[0073] In some cases, aspects of the present disclosure can be used with multiple different lenses. For example, aspects of the present disclosure can be utilized by two, four, eight, sixteen, or any suitable number of lenses. Additionally, the lenses may not need to be perfectly aligned with each other.

[0074] Figure 6 FIG. 600 is shown in accordance with one or more techniques of the present disclosure. As Figure 6 shown, FIG. 600 includes first image information 610, second image information 612, third image information 614, fourth image information 616, first grid 620, second grid 622, third grid 624, fourth grid 626, first position information 630, second position information 632, third position information 634, fourth position information 636, first UV image 640, second UV image 642, third UV image 644, fourth UV image 646, fifth grid 650, camera image 660, fifth position information 662, and display image 670. Figure 6 An aspect of single-pass rendering for lens distortion correction using multiple lenses (e.g., four lenses) is shown. As Figure 6 shown, the image from the camera is to the right of the dashed line, for example, an image captured by a user's head-mounted headset or head-mounted display (HMD).

[0075] Figure 6 It can be shown that each image information can be associated with a different lens. As Figure 6 shown, there can be four lenses, for example, a first lens, a second lens, a third lens, and a fourth lens, which respectively correspond to the first image information 610, the second image information 612, the third image information 614, and the fourth image information 616. Thus, aspects of the present disclosure are not limited to a specific number of lenses, for example, two lenses. As Figure 6 shown, the present disclosure can utilize four lenses as well as any other suitable number of lenses, for example, eight or sixteen lenses. Additionally, these multiple lenses may not need to be perfectly aligned with each other.

[0076] Figure 6The display first grid 620 may be based on the first image information 610. Additionally, the first position information 630 may be based on the first grid 620, and the first UV image 640 may be based on the first position information 630. The second grid 622 may be based on the second image information 612. Additionally, the second position information 632 may be based on the second grid 622 and the first UV image 640. Additionally, the second UV image 642 may be based on the second position information 632. The third grid 624 may be based on the third image information 614. Additionally, the third position information 634 may be based on the third grid 624 and the second UV image 642. The third UV image 644 may be based on the third position information 634. The fourth grid 626 may be based on the fourth image information 616. The fourth position information 636 may be based on the fourth grid 626 and the third UV image 644. The fourth UV image 646 may be based on the fourth position information 636. The fifth grid 650 may be based on the fourth UV image 646. Additionally, the fifth position information 662 may be based on the camera image 660 and the fifth grid 650. Finally, the display image 670 may be based on the fifth position information 662.

[0077] In some aspects, the present disclosure may convert an image space to a camera space, e.g., from pixels to mm. The following formula may be used: cameraX = (imageX – center_X) / focus_X; and cameraY = (imageY – center_Y) / focus_Y. In some cases, the image may be a specific number of pixels, e.g., 640x400 pixels. Additionally, the image may include the following values: Center_X = 324.5 pixels, Center_Y = 198.8 pixels, field of view (FOV) = 0.925 radians, Focal X = 269.4 pixels, Focal Y = 269.4 pixels. Additionally, in 3D camera space: -1.188 may correspond to +1.188 mm horizontally, and -0.742 may correspond to +0.742 mm vertically.

[0078] In some aspects, an application programming interface (API) may be used to render the grid to correct for fisheye image distortion. For example, aspects of the present disclosure may utilize the following formula As shown in the above formula, r d is the fisheye (distorted) radius of each vertex, i.e., the distance from the vertex to the center of the image grid, w is the field of view (in radians), and r u is the corrected (undistorted) radius of each vertex.

[0079] Aspects of the present disclosure may also include display lens grid generation. In some aspects, this can be the same method as described above for calculating the pixel radius. However, the calculation can utilize a mathematical formula using a radius parameter. In some aspects, the lens distortion polynomial can include a distortion scale, e.g., distortion scale = K0 + K1*r + K2*r 2 + K3*r 3 + K4*r 4 + K5*r 5 + K6*r 6 . K0, K1, K2, K3, K4, K5, K6 can be calibration numbers given by the lens manufacturer, and r can be the radius described above. The foregoing formula can be used to calculate UV texture coordinates.

[0080] Additionally, aspects of the present disclosure may include other methods of multi-lens distortion correction. As previously described, the final result of distortion correction can be a grid in which the UV texture coordinates are repositioned from a neutral position (i.e., no distortion) to a desired position, which may cause the desired image to bend. In another method, the XYZ vertex positions of the 3D grid can be modified to achieve the same bend. Thus, the UV texture coordinates can remain unchanged. This can be beneficial because asynchronous spatial warping (ASW), which is another technique used in AR or VR, may bend the image. In ASW, modifying the vertex positions can result in improved end-to-end error compared to modifying the UV texture positions, e.g., a 13.7% improvement. Therefore, when combining lens correction and ASW, it may be beneficial to perform all steps together in the XYZ space.

[0081] The XYZ distortion method can be similar to the method given above, but the present disclosure can render the vertex XYZ as a color for each lens correction. For example, red can be the X coordinate, green can be the Y coordinate, and blue can be the Z coordinate. In a first rendering, the present disclosure can also render the texture UV as a color and save the image. In a second rendering, the present disclosure can use the XYZ color image from the first rendering pass. In the final grid construction, the present disclosure can use the first UV color image rendering to determine the UV values. Additionally, the present disclosure can use the final XYZ image rendering to determine the final vertex positions.

[0082] Figure 7 FIG. 700 illustrates one or more techniques in accordance with the present disclosure. As Figure 7As shown, Figure 700 includes camera calibration data or first image information 710, display lens calibration data or second image information 712, first grid 720, second grid 722, first position information 730, second position information 732, XYZ position information 734, first UV image 740, first XYZ image 742, second XYZ image 744, third grid 750, camera image 760, third position information 762, and display image 770. Figure 7 An aspect of single-pass rendering of the XYZ method based on the above lens distortion correction is shown. As Figure 7 shown, the image from the camera is located to the right of the dashed line, for example, an image captured by a user's headset or HMD.

[0083] Figure 7 The display first grid 720 can be based on the first image information 710. In addition, the first position information 730 can be based on the first grid 720, and the first UV image 740 can be based on the first position information 730. The XYZ position information 734 can be based on the first position information 730, and the first XYZ image 742 can be based on the XYZ position information 734. The second grid 722 can be based on the second image information 712. In addition, the second position information 732 can be based on the second grid 722 and the first XYZ image 742. In addition, the second XYZ image 744 can be based on the second position information 732. The third grid 750 can be based on the second XYZ image 744 and the first UV image 740. In addition, the third position information 762 can be based on the camera image 760 and the third grid 750. Finally, the display image 770 can be based on the third position information 762.

[0084] Figures 5 - 7 An example of the foregoing process for lens distortion correction is shown. As Figures 5 - 7 shown, aspects of the present disclosure (e.g., the servers and client devices herein) can perform multiple different steps or processes to correct lens distortion. For example, the servers and client devices herein can determine first image information based on a first image or lens, e.g., first image information 510. In addition, the servers and client devices herein can generate a first image grid based on the determined first image information (e.g., first image information 510), e.g., first grid 520. The servers and client devices herein can also determine second image information based on a second image or lens, e.g., second image information 512. In addition, the servers and client devices herein can generate a second image grid based on the determined second image information (e.g., second image information 512), e.g., second grid 522.

[0085] The server and client devices of the present disclosure can also calculate first position information (e.g., first position information 530) based on a first image grid (e.g., first grid 520). In some aspects, the first position information can be based on at least one of first pixel position information or first UV texture coordinates. Additionally, the server and client devices of the present disclosure can determine a first UV image (e.g., first UV image 540) based on the calculated first position information (e.g., first position information 530).

[0086] Furthermore, the server and client devices of the present disclosure can calculate second position information (e.g., second position information 532) based on a second image grid (e.g., second grid 522) and the determined first UV image (e.g., first UV image 540). In some aspects, the second position information can be based on at least one of second pixel position information or second UV texture coordinates. The server and client devices of the present disclosure can also determine a second UV image (e.g., second UV image 542) based on the calculated second position information (e.g., second position information 532). Additionally, the server and client devices of the present disclosure can also generate a third image grid (e.g., third grid 550) based on the determined second UV image (e.g., second UV image 542).

[0087] The server and client devices of the present disclosure can also calculate third position information (e.g., third position information 562) based on a third image grid (e.g., third grid 550) and a camera image (e.g., camera image 560). Moreover, the server and client devices of the present disclosure can determine a display image (e.g., display image 570) based on the calculated third position information (e.g., third position information 562). In some aspects, the generated third image grid (e.g., third grid 550) can be associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application.

[0088] The server and client devices of the present disclosure can also calculate length (X), height (Y), depth (Z) (XYZ) position information (e.g., XYZ position information 734) based on the first position information (e.g., first position information 730). The server and client devices of the present disclosure can also determine a first XYZ image (e.g., first XYZ image 742) based on the calculated XYZ position information (e.g., XYZ position information 734). In some aspects, the second position information (e.g., second position information 732) can be calculated based on a second image grid (e.g., second grid 722) and the determined first XYZ image (e.g., first XYZ image 742).

[0089] In addition, the second UV image can be a second XYZ image, e.g., the second XYZ image 744. Additionally, a third image grid, e.g., the third grid 750, can be generated based on the second XYZ image (e.g., the second XYZ image 744) and the first UV image (e.g., the first UV image 740). Further, the XYZ position information (e.g., the XYZ position information 734) can be based on at least one of XYZ pixel position information or XYZ texture coordinates. In some aspects, the head-mounted display can apply the image grid to the camera image input. Additionally, a device can present an image including a camera and a renderer, where the renderer can be arranged to process an image from the camera using a 3D grid. In some cases, the device can be a head-mounted display. Further, the first image grid can be based on the camera calibration data of the device. Additionally, the second image grid can be derived according to, e.g., the display lens calibration data of the display lens used to present an image from a display within the device to a user.

[0090] Figure 8 FIG. 800 is an example flowchart showing an example method according to one or more techniques of the present disclosure. The method can be performed by a device such as a head-mounted display, an HMD, a client device, a server, a display processing unit, a display processor, a CPU, a GPU, or a device for display or graphics processing. At 802, the device can determine first image information based on a first image, as described in the example associated with Figures 3 - 7 the example.

[0091] At 804, the device can generate a first image grid based on the determined first image information, as described in the example associated with Figures 3 - 7 the example. At 806, the device can also determine second image information based on a second image, as described in the example associated with Figures 3 - 7 the example. At 808, the device can generate a second image grid based on the determined second image information, as described in the example associated with Figures 3 - 7 the example.

[0092] At 810, the device can also calculate first position information based on the first image grid, as described in the example associated with Figures 3 - 7 the example. In some aspects, the first position information can be based on at least one of first pixel position information or first UV texture coordinates, as described in the example associated with Figures 3 - 7 the example. At 812, the device can determine a first UV image based on the calculated first position information, as described in the example associated with Figures 3 - 7 the example.

[0093] At 814, the device can calculate length (X) height (Y) depth (Z) (XYZ) position information based on the first position information, as combined withFigures 3 - 7 as described in the example of. At 816, the device may determine a first XYZ image based on the calculated XYZ position information, as described in conjunction with Figures 3 - 7 the example of. In some aspects, the second position information may be calculated based on the second image grid and the determined first XYZ image, as described in conjunction with Figures 3 - 7 the example of. Additionally, the second UV image may be a second XYZ image, as described in conjunction with Figures 3 - 7 the example of. Additionally, a third image grid may be generated based on the second XYZ image and the first UV image, as described in conjunction with Figures 3 - 7 the example of. Additionally, the XYZ position information may be based on at least one of XYZ pixel position information or XYZ texture coordinates, as described in conjunction with Figures 3 - 7 the example of.

[0094] At 818, the device may calculate second position information based on the second image grid and the determined first UV image, as described in conjunction with Figures 3 - 7 the example of. In some aspects, the second position information may be based on at least one of second pixel position information or second UV texture coordinates, as described in conjunction with Figures 3 - 7 the example of. At 820, the device may also determine a second UV image based on the calculated second position information, as described in conjunction with Figures 3 - 7 the example of. At 822, the device may generate a third image grid based on the determined second UV image, as described in conjunction with Figures 3 - 7 the example of.

[0095] At 824, the device may calculate third position information based on the third image grid and the camera image, as described in conjunction with Figures 3 - 7 the example of. At 826, the device may determine a display image based on the calculated third position information, as described in conjunction with Figures 3 - 7 the example of. In some aspects, the generated third image grid may be associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application, as described in conjunction with Figures 3 - 7 the example of.

[0096] In one configuration, a method or apparatus for graphics processing is provided. The apparatus can be a headset, HMD, client device, server, display processing unit, display processor, CPU, GPU, or some other processor that can perform display or graphics processing. In one aspect, the apparatus can be the processing unit 120 within the device 104, or can be some other hardware within the device 104 or another device. The apparatus can include a unit for determining first image information based on a first image. The apparatus can include means for generating a first image grid based on the determined first image information. The apparatus can include a unit for determining second image information based on a second image. The apparatus can include a unit for generating a second image grid based on the determined second image information. The apparatus can include a unit for calculating first position information based on the first image grid. The apparatus can include a unit for determining a first UV image based on the calculated first position information. The apparatus can include a unit for calculating second position information based on the second image grid and the determined first UV image. The apparatus can include a unit for determining a second UV image based on the calculated second position information. The apparatus can include a unit for generating a third image grid based on the determined second UV image. The apparatus can include a unit for calculating third position information based on the third image grid and a camera image. The apparatus can include a unit for determining a display image based on the calculated third position information. The apparatus can include a unit for calculating length (X) height (Y) depth (Z) (XYZ) position information based on the first position information. The apparatus can include a unit for determining a first XYZ image based on the calculated XYZ position information.

[0097] The subject matter described herein can be implemented to realize one or more benefits or advantages. For example, the described graphics processing techniques can be used by a headset, HMD, client device, server, display processing unit, display processor, GPU, CPU, or some other processor that can perform display or graphics processing to implement the lens distortion correction techniques described herein. This can also be implemented at low cost compared to other display or graphics processing techniques. In addition, the display or graphics processing techniques herein can improve or accelerate data processing or execution. In addition, the display or graphics processing techniques herein can increase resource or data utilization and / or resource efficiency. Additionally, aspects of the present disclosure can utilize a lens distortion correction process that can result in faster rendering times and / or reduced latency. The lens distortion correction process can also result in reduced GPU data bandwidth usage. Additionally, the lens distortion correction process can result in reduced GPU calculations per vertex and pixel and / or reduced GPU texture memory access. The lens distortion correction process can also result in improved or sharper image quality.

[0098] According to the present disclosure, unless the context dictates otherwise, the term "or" can be interpreted as "and / or". Additionally, although phrases such as "one or more" or "at least one" may have been used for some features disclosed herein and not for others, features for which such language is not used can be interpreted to imply such meaning unless the context dictates otherwise.

[0099] In one or more examples, the functions described herein can be implemented in hardware, software, firmware, or any combination thereof. For example, although the term "processing unit" has been used throughout the present disclosure, such a processing unit can be implemented in hardware, software, firmware, or any combination thereof. If any function, processing unit, technique, or other module described herein is implemented in software, any function, processing unit, technique, or other module can be stored on or transmitted through a computer-readable medium as one or more instructions or code. A computer-readable medium can include a computer data storage medium or a communication medium, where the communication medium includes any medium that facilitates the transfer of a computer program from one place to another. In this way, a computer-readable medium generally can correspond to: (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. The data storage medium can be any available medium that can be accessed by one or more computers or one or more processors to obtain instructions, code, and / or data structures for implementing the techniques described in the present disclosure. By way of example and not limitation, such computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices. As used herein, disk and optical disks can include compact disk (CD), laser disk, optical disk, digital versatile disk (DVD), floppy disk, and Blu-ray disk, where disks typically reproduce data magnetically, while optical disks utilize lasers to optically reproduce data. Combinations of the above should also be included within the scope of computer-readable media. Any computer program product can include a computer-readable medium.

[0100] The code can be executed by one or more processors, such as one or more digital signal processors (DSPs), general microprocessors, application specific integrated circuits (ASICs), arithmetic logic units (ALUs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, as used herein, the term "processor" can refer to any one of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, the techniques can be implemented entirely in one or more circuits or logic elements.

[0101] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs) or a set of ICs (e.g., a chipset). Various components, modules, or units are described in the present disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but need not necessarily be implemented by distinct hardware units. Rather, as described above, the various units may be combined in any hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) in conjunction with appropriate software and / or firmware.

[0102] Individual examples have been described. These and other examples are within the scope of the appended claims.

Claims

1. A method for display processing, comprising: Calculating first position information based on a first image grid, wherein the first image grid is based on first image information for a first image, and wherein the first image information corresponds to camera calibration data; Determining a first horizontal and vertical image based on the calculated first position information, wherein the first horizontal and vertical image is associated with first horizontal and vertical texture coordinates of vertices of a grid corresponding to the vertices of the 3D or XYZ position coordinates of the first image grid; Calculating second position information based on a second image grid and the determined first horizontal and vertical image, wherein the second image grid is based on second image information for a second image, and wherein the second image information corresponds to display lens calibration data; Determining a second horizontal and vertical image based on the calculated second position information, wherein the second horizontal and vertical image is associated with second horizontal and vertical texture coordinates of vertices of a grid corresponding to the vertices of the 3D or XYZ position coordinates of the second image grid; Generating a third image grid based on the determined second horizontal and vertical image; Calculating third position information based on the third image grid and a camera image; and Determining a display image based on the calculated third position information.

2. The method according to claim 1, further comprising: Determining first image information based on the first image.

3. The method according to claim 2, further comprising: Generating the first image grid based on the determined first image information.

4. The method according to claim 1, further comprising: Determining second image information based on the second image.

5. The method according to claim 4, further comprising: Generating the second image grid based on the determined second image information.

6. The method according to claim 1, further comprising: Calculating length, height, depth XYZ position information based on the first position information; and And Determining a first XYZ image based on the calculated XYZ position information.

7. The method according to claim 6, wherein The second position information is calculated based on the second image grid and the determined first XYZ image.

8. The method according to claim 7, wherein, The second horizontal and vertical image is a second XYZ image.

9. The method according to claim 8, wherein, The third image grid is generated based on the second XYZ image and the first horizontal and vertical image.

10. The method according to claim 1, wherein, The first position information is based on at least one of first pixel position information or first horizontal and vertical texture coordinates.

11. The method according to claim 1, wherein, The second position information is based on at least one of second pixel position information or second horizontal and vertical texture coordinates.

12. The method according to claim 6, wherein The XYZ position information is based on at least one of XYZ pixel position information or XYZ texture coordinates.

13. The method according to claim 1, wherein, The generated third image grid is associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application.

14. An apparatus for display processing, comprising: A memory; And At least one processor coupled to the memory and configured to: Calculate first position information based on a first image grid, wherein the first image grid is based on first image information for a first image, and wherein the first image information corresponds to camera calibration data; Determine a first horizontal-vertical image based on the calculated first position information, wherein the first horizontal-vertical image is associated with first horizontal-vertical texture coordinates of vertices of a grid corresponding to the vertices of the 3D or XYZ position coordinates of the first image grid; Calculate second position information based on a second image grid and the determined first horizontal-vertical image, wherein the second image grid is based on second image information for a second image, and wherein the second image information corresponds to display lens calibration data; Determine a second horizontal-vertical image based on the calculated second position information, wherein the second horizontal-vertical image is associated with second horizontal-vertical texture coordinates of vertices of a grid corresponding to the vertices of the 3D or XYZ position coordinates of the second image grid; Generate a third image grid based on the determined second horizontal-vertical image; Calculate third position information based on the third image grid and a camera image; and Determine a display image based on the calculated third position information.

15. The apparatus according to claim 14, wherein, The at least one processor is further configured to: Determine first image information based on a first image.

16. The apparatus according to claim 15, wherein, The at least one processor is further configured to: Generate the first image grid based on the determined first image information.

17. The device according to claim 14, wherein, The at least one processor is further configured to: Determine second image information based on a second image.

18. The apparatus according to claim 17, wherein, The at least one processor is further configured to: Generate the second image grid based on the determined second image information.

19. The apparatus according to claim 14, wherein, The at least one processor is further configured to: Calculate length-height-depth XYZ position information based on the first position information; and Determine a first XYZ image based on the calculated XYZ position information.

20. The device according to claim 19, wherein, The second position information is calculated based on the second image grid and the determined first XYZ image.

21. The device according to claim 20, wherein, The second horizontal-vertical image is a second XYZ image.

22. The device according to claim 21, wherein, The third image grid is generated based on the second XYZ image and the first horizontal-vertical image.

23. The apparatus according to claim 14, wherein The first position information is based on at least one of first pixel position information or first horizontal-vertical texture coordinates.

24. The apparatus according to claim 14, wherein The second position information is based on at least one of second pixel position information or second horizontal-vertical texture coordinates.

25. The apparatus according to claim 19, wherein The XYZ position information is based on at least one of XYZ pixel position information or XYZ texture coordinates.

26. The device according to claim 14, wherein The generated third image grid is associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application.

27. A device for display processing, comprising: A unit for calculating first position information based on a first image grid, wherein the first image grid is based on first image information for a first image, and wherein the first image information corresponds to camera calibration data; A unit for determining a first horizontal-vertical image based on the calculated first position information, wherein the first horizontal-vertical image is associated with first horizontal-vertical texture coordinates of vertices of a grid corresponding to the vertices of the 3D or XYZ position coordinates of the first image grid; A unit for calculating second position information based on a second image grid and a determined first horizontal and vertical image, wherein the second image grid is based on second image information for a second image, and the second image information corresponds to display lens calibration data; A unit for determining a second horizontal and vertical image based on the calculated second position information, wherein the second horizontal and vertical image is associated with second horizontal and vertical texture coordinates of vertices of a grid corresponding to 3D or XYZ position coordinates of vertices of the second image grid; A unit for generating a third image grid based on the determined second horizontal and vertical image; A unit for calculating third position information based on the third image grid and a camera image; and A unit for determining a display image based on the calculated third position information.

28. The apparatus according to claim 27, further comprising: A unit for determining first image information based on a first image.

29. The apparatus according to claim 28, further comprising: A unit for generating the first image grid based on the determined first image information.

30. The apparatus according to claim 27, further comprising: A unit for determining second image information based on a second image.

31. The apparatus according to claim 30, further comprising: A unit for generating the second image grid based on the determined second image information.

32. The apparatus according to claim 27, further comprising: A unit for calculating length, height, depth XYZ position information based on the first position information; and A unit for determining a first XYZ image based on the calculated XYZ position information.

33. The apparatus according to claim 32, wherein, The second position information is calculated based on the second image grid and the determined first XYZ image.

34. The apparatus according to claim 33, wherein, The second horizontal and vertical image is a second XYZ image.

35. The apparatus according to claim 34, wherein The third image grid is generated based on the second XYZ image and the first horizontal and vertical image.

36. The device according to claim 27, wherein, The first position information is based on at least one of first pixel position information or first horizontal and vertical texture coordinates.

37. The apparatus according to claim 27, wherein, The second position information is based on at least one of second pixel position information or second horizontal and vertical texture coordinates.

38. The apparatus according to claim 32, wherein, The XYZ position information is based on at least one of XYZ pixel position information or XYZ texture coordinates.

39. The apparatus according to claim 27, wherein The generated third image grid is associated with at least one of an augmented reality (AR) application, an extended reality (XR) application, or a virtual reality (VR) application.

40. A computer-readable medium storing computer-executable code for display processing, which, when executed by a processor, causes the processor to perform the following operations: Calculate first position information based on a first image grid, wherein, The first image grid is based on first image information for a first image, and the first image information corresponds to camera calibration data; Determining a first horizontal and vertical image based on the calculated first position information, wherein the first horizontal and vertical image is associated with first horizontal and vertical texture coordinates of vertices of a grid corresponding to 3D or XYZ position coordinates of vertices of the first image grid; Calculate second position information based on a second image grid and the determined first horizontal-vertical image, wherein the second image grid is based on second image information for a second image, and wherein the second image information corresponds to display lens calibration data; Determine a second horizontal-vertical image based on the calculated second position information, wherein the second horizontal-vertical image is associated with second horizontal-vertical texture coordinates of vertices of a grid corresponding to 3D or XYZ position coordinates of vertices of the second image grid; Generate a third image grid based on the determined second horizontal-vertical image; Calculate third position information based on the third image grid and a camera image; and Determine a display image based on the calculated third position information.

Images