Distributed real-time automatic white balance correction for artificial reality applications

Abstract

The invention discloses distributed real-time automatic white balance correction for artificial reality applications. Aspects of the present disclosure relate to a process for applying an automatic white balance (AWB) correction technique in a distributed manner to an image captured at an imaging device. The AWB is used to adjust the color balance of the image so that the image looks more natural and more attractive to viewers. The AWB may neutralize color cast that may be present in the image, such as color cast caused by lighting conditions or sensors of the camera. No AWB correction may lead to incorrect image color (e.g., green hue), while AWB insufficiency may often be considered incorrect image hue. In examples, at least a portion of the computational complexity involved in performing the AWB correction technique may be transferred to a companion computing device, such as a co-processing device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 513,006, filed on July 11, 2023, entitled “Distributed Real-Time Auto White Balance Correction for Artificial Reality Applications,” and U.S. Non-Provisional Patent Application No. 18 / 615,124, filed on March 25, 2024, both of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure is directed to applying Auto White Balance (AWB) processing to still images and streaming video while offloading some of the AWB processing. Background Art

[0004] Streaming video in a head-mounted display (HMD) involves delivering video content directly to the user's eyes via a wearable display device. This technology is often used in virtual reality (VR), mixed reality (MR), and augmented reality (AR) (collectively referred to as extra reality (XR)) applications, where users want to be fully immersed in a simulated environment. In VR applications, users are surrounded by a simulated environment, while in AR and MR applications, virtual objects are overlaid on the real world. In both cases, streaming video to the HMD is critical to creating a seamless and immersive experience. Summary of the invention

[0005] According to some embodiments, a portion of a distributed AWB correction process may be applied to a live video stream or real-time video stream using a lower resolution image or frame. In contrast, another portion of the AWB correction process may be applied to a high resolution image or frame. AWB correction algorithms are typically too computationally expensive to run on an HMD; however, generating AWB statistics is typically less expensive and uses less memory than operating on the original high resolution frame or image. For example, image data analysis for determining one or more configuration settings of the AWB correction algorithm may be performed at a device such as an HMD device. The AWB correction algorithm may be performed at a co-processing device (e.g., a local computing device, a mobile phone, a remote server, etc.) external to the HMD. That is, the AWB statistics generation process and the AWB correction algorithm may be decoupled from each other, whereby the AWB statistics generation process may be performed on a lower resolution frame or image at the HMD; and then the statistics generated by the AWB statistics generation process may be transmitted from the HMD to the co-processing device, so that the AWB correction algorithm may be performed on a higher resolution frame or image. Therefore, AWB correction may be performed on images and / or videos in a distributed manner without sacrificing image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram showing an overview of a device on which some embodiments of the present technology may operate.

[0007] Figure 2A is a line diagram illustrating a virtual reality headset that may be used in some embodiments of the present technology.

[0008] Figure 2B is a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.

[0009] Figure 2C is a line diagram illustrating controllers that, in some implementations, a user may hold with one or both hands to interact with an artificial reality environment.

[0010] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0011] Figure 4 is a block diagram illustrating components that may be used in a system employing the disclosed technology in some implementations.

[0012] Figure 5 is a flow chart illustrating a process for performing distributed AWB correction on streaming or real-time video in some implementations.

[0013] Figure 6 is a flow chart illustrating a process for performing distributed AWB correction on a single high resolution image acquired by an imaging device in some embodiments.

[0014] Figure 7 is a conceptual diagram showing details of a system for performing distributed AWB correction on streaming or real-time video used in some embodiments.

[0015] Figure 8 is a conceptual diagram showing details of a system for performing distributed AWB correction on a single high resolution image acquired by an imaging device used in some embodiments.

[0016] The technology described herein may be better understood by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numbers refer to identical or functionally similar elements. DETAILED DESCRIPTION

[0017] Various aspects of the present disclosure relate to processes for applying automatic white balance (AWB) correction techniques to images acquired at an imaging device in a distributed manner. AWB is used to adjust the color balance of an image to make the image appear more natural and more appealing to a viewer. AWB can neutralize color casts that may exist in an image, such as those caused by lighting conditions or the camera's sensor. Failure to perform AWB correction may result in incorrect image color (e.g., a green tint), while insufficient AWB is often perceived as incorrect image tint.

[0018] AWB correction is typically applied as part of an imaging pipeline embedded within an Image Signal Processor (ISP). The ISP can be physically attached to the same processing unit (System on Chip or SoC) as the imaging device. This allows processes associated with the AWB algorithm to access image data via a wired physical connection and correct the image data in a timely manner. In the case of artificial reality (XR) applications, more stringent requirements on power and latency make it challenging to run computationally expensive algorithms that require a lot of power. Therefore, it may be challenging to perform AWB correction using the same processing unit attached to the image sensor. Therefore, at least a portion of the calculations involved in performing AWB correction can be transferred to a companion computing device (e.g., a co-processor) that is part of the XR system, where the companion computing device can be part of a head-mounted device (HMD) or communicate in real time through a wired connection or a wireless connection between the HMD and another processing device.

[0019] Since there are no latency issues, AWB correction can be performed as part of a distributed wireless XR system. Performing AWB in this distributed manner would enable XR products to perform AWB correction remotely while meeting the power and latency requirements of the HMD. However, since at least part of the AWB correction process is not performed on the same processing unit attached to the imaging device, this latency is introduced in part due to wireless communication and large wireless bandwidth requirements, which may make it infeasible for XR systems to perform AWB correction completely remotely in a distributed manner.

[0020] According to some embodiments, a portion of a distributed AWB correction process may be applied to a live video stream or real-time video stream using a lower resolution image or frame. In contrast, another portion of the AWB correction process may be applied to a high resolution image or frame. AWB correction algorithms are typically too computationally expensive to run on an HMD; however, generating AWB statistics is typically less expensive and uses less memory than operating on the original high resolution frame or image. For example, image data analysis for determining one or more configuration settings of the AWB correction algorithm may be performed at a device such as an HMD device. The AWB correction algorithm may be performed at a co-processing device (e.g., a local computing device, a mobile phone, a remote server, etc.) external to the HMD. That is, the AWB statistics generation process and the AWB correction algorithm may be decoupled from each other, whereby the AWB statistics generation process may be performed on a lower resolution frame or image at the HMD; and then the statistics generated by the AWB statistics generation process may be transmitted from the HMD to the co-processing device, so that the AWB correction algorithm may be performed on a higher resolution frame or image. Therefore, AWB correction may be performed on images and / or videos in a distributed manner without sacrificing image quality.

[0021] Embodiments of the disclosed technology may include an artificial reality system, or may be implemented in conjunction with an artificial reality system. Artificial reality or hyperreality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and artificial reality or hyperreality may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with collected content (e.g., real-world photos). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality, and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave" environment or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0022] As used herein, "virtual reality" or "VR" refers to an immersive experience in which the user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views images of the real world after they have passed through a computing system. For example, a tablet with a camera on the back can capture multiple images of the real world and then display them on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images as they pass through the system, for example by adding virtual objects. "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is generated in part by a computing system and in part from light reflected from objects in the real world. For example, an MR head-mounted viewer can be shaped as a pair of glasses with a transparent display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, thereby allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of VR, AR, MR, or any combination or hybrid thereof.

[0023] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1 is a block diagram illustrating an overview of devices on which some embodiments of the disclosed technology may run. These devices may include multiple hardware components of a computing system 100 that apply automatic white balance (AWB) correction techniques to images acquired at an imaging device in a distributed manner. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted viewer that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted viewer and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head-mounted viewer and other processing operations are transferred to the core processing component. The following is combined with Figure 2A and Figure 2B An example head mounted viewer is described. In some implementations, location data and environmental data may be collected solely by sensors incorporated into the head mounted viewer, while in other implementations, one or more non-head mounted computing devices may include sensor components that may track environmental data or location data.

[0024] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among the computing devices 101 to 103).

[0025] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be transmitted by a hardware controller that interprets the signals received from the input devices and transmits the information to the processor 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0026] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus, such as a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus, or a wireless connection. The processor 110 can communicate with a hardware controller of a device (e.g., a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices include: a liquid crystal display (LCD) display screen, a light emitting diode (LED) display screen, a projection display, a holographic display, or an augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or network card, a video chip or video card, an audio chip or audio card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0027] In some embodiments, input from I / O devices 140 (e.g., cameras, depth sensors, inertial motion unit (IMU) sensors, global positioning system (GPS) units, LiDAR or other time-of-flight sensors, etc.) can be used by computing system 100 to identify and map the user's physical environment while tracking the user's position in that environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., topology, grid, etc.) of an area (which can be a room, building, outdoor space, etc.) and / or obtain a map previously generated by computing system 100 or another computing system that has mapped the area. The SLAM system can track users within the area based on factors such as GPS data, match identified objects and structures with mapped objects and structures, monitor acceleration and other position changes, etc.

[0028] The computing system 100 may include a communication device that can communicate wirelessly or wired with other local computing devices or network nodes. The communication device can communicate with another device or server over a network, for example, using a transmission control protocol / internet protocol (TCP / IP). The computing system 100 can use the communication device to distribute operations across multiple network devices.

[0029] Processor 110 may access memory 150, which may be included in one of the computing devices of computing system 100, or may be distributed across multiple computing devices of computing system 100, or other external devices. Memory includes one or more hardware devices for volatile storage or non-volatile storage, and may include both read-only memory and writable memory. For example, memory may include one or more of the following: random access memory (RAM); various cache memories; CPU registers; read-only memory (ROM); and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, and tape drive. Memory is not a propagating signal that is separated from the underlying hardware; therefore, memory is non-transitory. Memory 150 may include program memory 160 storing programs and software, such as operating system 162, AWB correction system 164, and other application programs 166. Memory 150 may also include data storage 170 which may include high and low resolution images and video, AWB correction statistics, configuration data, settings, user options or preferences, etc. which may be provided to program storage 160 or any element of computing system 100 .

[0030] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with the present technology include, but are not limited to, XR head-mounted viewers, personal computers, server computers, handheld devices or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices, etc.

[0031] Figure 2A2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. HMD 200 includes a front rigid body 205 and a strap 210. Front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a plurality of locators 225, and one or more computing units 230. Position sensors 220, IMU 215, and computing units 230 may be located inside HMD 200 and may not be visible to the user. In various embodiments, IMU 215, position sensors 220, and locators 225 may track the movement and position of HMD 200 in the real world and in an artificial reality environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, locators 225 may emit infrared beams that produce light spots on real objects around HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position, force, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the HMD 200 may detect light points. The computing unit 230 in the HMD 200 may use the detected light points to infer the position and movement of the HMD 200, and to recognize the shape and position of real objects around the HMD 200.

[0032] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of the electronic display 245 include a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., micro-LED, laser (LASER)), some other display, or some combination thereof.

[0033] In some embodiments, the HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the HMD 200 (e.g., through light emitted from the HMD 200), which the PC can use in conjunction with outputs from the IMU 215 and the position sensor 220 to determine the position and movement of the HMD 200.

[0034] Figure 2B 2 is a line diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 GHz link) as indicated by link 256. In other embodiments, the mixed reality HMD system 250 includes only a head-mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a transparent display 258 and a frame 260. The frame 260 can accommodate various electronic components (not shown), such as light projectors (e.g., lasers, LEDs, etc.), cameras, eye tracking sensors, micro-electromechanical systems (MEMS) components, network components, etc.

[0035] The projector can be coupled to the transparent display 258, for example, via an optical element to display media to the user. The optical element can include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. for guiding the light from the projector to the user's eyes. Image data can be transmitted from the core processing component 254 to the HMD 252 via the link 256. The controller in the HMD 252 can convert the image data into a plurality of light pulses from the projector, which can be transmitted to the user's eyes via the optical element as output light. The output light can be mixed with the light passing through the transparent display 258, thereby allowing the output light to present the following virtual objects: these virtual objects appear to exist in the real world.

[0036] Similar to HMD 200, HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow HMD system 250 to track itself with 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects so that they appear to be stationary when HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0037] Figure 2C Controllers 270 (including controllers 276A and 276B) are shown, and in some embodiments, a user can hold the controllers with one or both hands to interact with the artificial reality environment presented by the HMD 200 and / or the HMD system 250. The controllers 270 can communicate with the HMD directly or via an external device (e.g., the core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit light points farther away. The HMD 200 or HMD system 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., to track the controllers with 3DoF or 6DoF). The computing unit 230 or the core processing component 254 in the HMD 200 can use this tracking to monitor the position and movement of the user's hands in combination with the IMU and position outputs. The controller may also include various buttons (e.g., buttons 272A-272F) and / or joysticks (e.g., joysticks 274A-274B) that a user may actuate to provide input and interact with objects.

[0038] In various embodiments, the HMD 200 or HMD system 250 may also include additional subsystems (e.g., an eye tracking unit, an audio system, various network components, etc.) to monitor the user's interactions and indications of intent. For example, in some embodiments, instead of or in addition to the controller, one or more cameras included in the HMD 200 or HMD system 250 or from an external camera can monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources can illuminate either or both eyes of the user, and the HMD 200 or HMD system 250 can use eye-facing cameras to capture reflections of the light to determine eye position (e.g., based on a set of reflections around the user's cornea), thereby modeling the user's eyes and determining gaze direction.

[0039] Figure 3 305B may be a HMD 200 or a HMD system 250. The client computing device 305 may be a logical connection to one or more remote computers (e.g., a server computing device) through a network 330 to operate in a network environment.

[0040] In some implementations, server computing device 310 may be an edge server that receives client requests and coordinates fulfillment of those requests through other servers (e.g., server computing devices 320A-320C). Server computing devices 310 and 320 may include a computing system, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, each server computing device may be a distributed computing environment that includes multiple computing devices located at the same physical location or at geographically different physical locations.

[0041] Client computing device 305, and server computing devices 310 and 320 can all act as servers or clients of one or more other server / client devices. Server computing device 310 can be connected to database 315. Server computing devices 320A to 320C can each be connected to corresponding databases 325A to 325C. As described above, each server computing device 310 or 320 can correspond to a group of servers, and each of these servers can share a database, or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can all be distributed computing environments containing multiple computing devices, can be located in their corresponding servers, or can be located at the same physical location or at geographically different physical locations.

[0042] Network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. Network 330 may be the Internet or some other public or private network. Client computing device 305 may be connected to network 330 via a network interface, such as via wired or wireless communications. Although the connections between server computing device 310 and each server computing device 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including network 330 or a separate public or private network.

[0043] Figure 44 is a block diagram showing various components 400 that can be used in a system that employs the disclosed technology in some embodiments. Each component 400 can be included in one device of the computing system 100, or can be distributed across multiple devices of the computing system 100. Each component 400 includes hardware 410, middleware 420, and various dedicated components 430. As described above, the system implementing the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices (also known as I / O) 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.) and a storage memory 418. In various embodiments, the storage memory 418 can be one or more of the following: a local device; an interface to a remote storage device; or a combination thereof. For example, the storage memory 418 can be one or more hard drives or flash drives accessible via a system bus, or can be a cloud storage provider (e.g., located in a database 315 or 325) or other network storage accessible via one or more communication networks. In various embodiments, each component is implemented on a server computing device 310 or 320).

[0044] Middleware 420 may include components that transfer resources between hardware 410 and dedicated components 430. For example, middleware 420 may include an operating system, services, drivers, a basic input output system (BIOS), controller circuits, or other hardware systems or software systems.

[0045] The dedicated components 430 may include software or hardware configured to perform operations for applying an automatic white balance (AWB) correction technique to an image acquired at an imaging device in a distributed manner. The dedicated components 430 may include an image acquisition module 434; an AWB statistics generation module 436; an automated exposure correction (AEC) statistics generation module 438; an AWB correction module 440; an AEC module 442; and components and application programming interfaces (APIs) (e.g., interface 432) that may be used to provide a user interface, transfer data, and control each dedicated component. In some embodiments, each component 400 may be located in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more of the dedicated components 430. For example, the AWB statistics generation module 436 may be located at the HMD 252, while the AWB correction module 440 may be located at a co-processing device, such as the core processing component 254, other client computing devices 305, or server computing devices 310 or 320. Although the specialized components 430 are depicted as separate components, these specialized components 430 may be logical functions or other non-physically distinct functions, and / or may be sub-modules or code blocks of one or more applications.

[0046] Image acquisition module 434 can acquire one or more images from an imaging device (e.g., a camera). In some examples, image acquisition module 434 can acquire a high-resolution image and / or a low-resolution image. In some examples, image acquisition module 434 can acquire a series of images, such as a video. Alternatively or additionally, image acquisition module 434 can acquire a single high-resolution image. Figure 5 Frame 502, Figure 6 Frame 602 and Figure 7 and Figure 8 Additional details related to acquiring images from imaging devices are provided.

[0047] The AWB statistics generation module 436 can generate statistics upon which AWB correction can be based. For example, the AWB statistics can include, but are not limited to, the ratio of red pixels to green pixels, and in some cases, the ratio of blue pixels to green pixels. In some examples, the AWB statistics can also include the distribution of red pixels, green pixels, and blue pixels throughout the image. In some embodiments, the AWB statistics can be generated for the entire image; alternatively or additionally, the AWB statistics can be generated for one or more regions of the image. However, in general, the AWB statistics can be associated with an estimate of the ambient light temperature of the image. The following description of the AWB statistics can be used to describe the ambient light temperature of the image. Figure 5 Frame 504, Figure 6 Box 604 and Figure 7 and Figure 8 Additional details concerning the generation of AWB statistics are provided.

[0048] AWB correction module 440 may perform AWB correction processing that includes determining changes to one or more pixel values ​​in an image. Thus, the co-processing device may perform calculations associated with determining changes to one or more pixel values ​​in the absence of an image based on the AWB statistics received from AWB statistics generation module 436. AWB correction module 440 may then apply these changes to the received high resolution image to produce an AWB corrected high resolution image. Figure 5 Frames 506 to 510, Figure 6 Boxes 606 to 610 and Figure 7 and Figure 8 Additional details related to performing the AWB correction process are provided.

[0049] The AEC statistics generation module 438 can generate AEC statistics, which may include, but are not limited to, the average brightness of the image, the standard deviation of one or more pixel values ​​in the image, the histogram or distribution of each pixel value in the image, the image contrast, the image saturation, and the image exposure time. The AEC module 442 can apply processing to the image to determine one or more adjustment parameters, such as exposure time and / or gain, which can be returned to the imaging device. In various examples, the one or more adjustment parameters may be the actual exposure time and / or the actual gain setting, or may be a change to the exposure time or a change to the gain setting. Thus, the imaging device can combine these adjustments to obtain a subsequent image, which may be a high resolution image. The following is about Figure 5 Frames 504 to 508, Figure 6 Boxes 604 to 608 and Figure 7 and Figure 8 Additional details are provided regarding generating AEC statistics and executing the AEC process.

[0050] It will be understood by those skilled in the art that the above Figures 1 to 4 The components shown in the flowcharts and in each of the flowcharts discussed below may be changed in various ways. For example, the order of the logic may be rearranged, sub-steps may be performed in parallel, the logic shown may be omitted, other logic may be included, etc. In some embodiments, one or more of the above components may perform one or more of the processes described below.

[0051] Figure 55 is a flow chart illustrating a process 500 for performing distributed AWB correction on streaming or real-time video in some embodiments. In some embodiments, the process 500 is initiated at the request of a user. For example, the user may indicate that he or she wants the HMD to record a video or otherwise view a video acquired from an imaging device associated with the HMD. Thus, at block 502, the imaging device may provide a streaming video to the HMD, the streaming video comprising a plurality of images or frames.

[0052] At block 504, process 500 may generate AEC statistics and AWB statistics based on an image or frame of a streaming video. The image or frame may have a resolution that allows the image or frame to be transmitted wired or wirelessly from the HMD to the co-processing device and / or from the co-processing device to the HMD, depending on one or more latency constraints of the connection. In some examples, the resolution of the image or frame may need to be reduced to comply with one or more constraints of the connection. In various examples, the AEC statistics may include, but are not limited to, the average brightness of the image, the standard deviation of one or more pixel values ​​in the image, the histogram or distribution of each pixel value in the image, the image contrast, the image saturation, and the image exposure time. The AWB statistics may include, but are not limited to, the ratio of red pixels to green pixels, and in some cases, the ratio of blue pixels to green pixels. In some examples, the AWB statistics may also include the distribution of red pixels, green pixels, and blue pixels throughout the image. In some embodiments, the AWB statistics may be generated for the entire image; alternatively or additionally, the AWB statistics may be generated for one or more regions of the image. However, in general, the AWB statistics may be associated with an estimate of the ambient light temperature of the image.

[0053] At box 506, the AEC statistics may be provided to the AEC module. In addition, the AWB statistics may be transmitted to a co-processing device located outside the HMD (e.g., a local dedicated processing unit such as client computing device 305B, a local multi-purpose device such as client computing device 305A, or a remote device such as server computing device 310 or 320) via a wired communication interface or a wireless communication interface. At box 508, the AEC module at the HMD may apply processing to the image to determine one or more adjustment parameters, such as exposure time and / or gain, that may be returned to the imaging device. In various examples, the one or more adjustment parameters may be an actual exposure time and / or an actual gain setting, or may be a change to the exposure time or a change to the gain setting. Thus, the imaging device may combine these adjustments to acquire subsequent images or frames of video.

[0054] At block 510, the co-processing device may receive an image from the AEC module and match the image with the previously received AWB statistics provided by the AWB statistics generation module. Although process 500 is shown as a single process, it should be understood that blocks 510 and 512 of process 500 may be performed on separate co-processing devices separate from blocks 502 to 508, and therefore may be separate processes. The AWB correction module may then perform an AWB correction process on the image. In various examples, the AWB correction process may include determining a change to one or more pixel values ​​in the image. Therefore, based on the AWB statistics received from the AWB statistics generation module, calculations associated with changes to one or more pixel values ​​may be performed in the absence of the image. The AWB correction module may then apply these changes to the received image, thereby generating an AWB-corrected image or frame of the video. Of course, it should be understood that calculations associated with changes to one or more pixel values ​​may be performed in the presence of the image. At block 512, the AWB-corrected image may then be output for display. In various examples, the co-processing device may transmit the AWB-corrected image back to the HMD, either wired or wirelessly, for display to the user.

[0055] Figure 6600 is a flowchart showing a process 600 for performing distributed AWB correction on a single high-resolution image collected by an imaging device in some embodiments. In some embodiments, process 600 is initiated at the request of a user. For example, a user may indicate that he or she wants the HMD to acquire an image (e.g., a high-resolution image) from an imaging device associated with the HMD. Therefore, at box 602, the imaging device may acquire an image. Similar to process 500, a series of time data may be used to ensure that the AWB correction process operates correctly. However, due to the high resolution of the image, it may not be feasible to acquire multiple consecutive high-resolution images due to memory and power limitations of the imaging device and / or HMD. In addition, it may not be feasible to transmit these high-resolution frames to a co-processing device with an acceptable delay. Therefore, at box 602, a series of low-resolution images are acquired or streamed, and the series of low-resolution images are used for AEC statistics generation and AWB statistics generation. However, the AEC module can acquire high-resolution images to determine one or more automatic exposure corrections using AEC statistics generated from low-resolution images. That is, one or more low-resolution images may be used to generate AEC statistics and AWB statistics; the AEC statistics and AWB statistics of the low-resolution images may then be used to determine the AEC processing and AWB processing that may be performed on the high-resolution images. In other words, regions of the low-resolution images may be used to understand regions of the high-resolution image that may be processed using the AWB correction process by using the AEC statistics and AWB statistics generated from the low-resolution images.

[0056] At box 604, process 600 can generate AEC statistics and AWB statistics based on the series of low-resolution images. In various examples, the AEC statistics may include, but are not limited to: the average brightness of the image, the standard deviation of one or more pixel values ​​in the image, the histogram or distribution of each pixel value in the image, the image contrast, the image saturation, and the image exposure time. AWB statistics may include, but are not limited to: the ratio of red pixels to green pixels, and in some cases, the ratio of blue pixels to green pixels. In some examples, the AWB statistics may also include the distribution of red pixels, green pixels, and blue pixels throughout the image. In some embodiments, AWB statistics may be generated for the entire image; alternatively or additionally, AWB statistics may be generated for one or more regions of the image. However, in general, AWB statistics may be associated with an estimate of the ambient light temperature of the image.

[0057] At block 606, the AEC statistics may be provided to the AEC module. In addition, the AWB statistics may be transmitted to a co-processing device external to the HMD (e.g., a local dedicated processing unit such as client computing device 305B, a local multi-purpose device such as client computing device 305A, or a remote device such as server computing device 310 or 320) via a wired communication interface or a wireless communication interface. That is, a series of AWB statistics may be transmitted to the co-processing device several frames before the high-resolution image is acquired. This allows the AWB correction process to determine the correct decision or application of AWB using minimal bandwidth. Once the high-resolution image is transmitted to the co-processing device, the AWB correction may be performed.

[0058] At block 608, the AEC module may apply processing to the high-resolution image acquired from the imaging device to determine one or more adjustment parameters, such as exposure time and / or gain, that may be returned to the imaging device. In various examples, the one or more adjustment parameters may be an actual exposure time and / or an actual gain setting, or may be a change to the exposure time or a change to the gain setting. Thus, the imaging device may incorporate these adjustments to acquire a subsequent image, which may be a high-resolution image.

[0059] At block 610, the co-processing device may receive the high-resolution image from the AEC module and perform an AWB correction process on the high-resolution image based on the AWB statistics previously transmitted for the low-resolution image. Although process 600 is shown as a single process, it should be understood that blocks 610 and 612 of process 600 may be performed on separate co-processing devices separate from blocks 602 to 608, and thus may be separate processes. In various examples, the AWB correction process may include determining changes to one or more pixel values ​​in the high-resolution image. Therefore, based on the AWB statistics received from the AWB statistics generating module, calculations associated with changes to one or more pixel values ​​may be performed in the absence of a high-resolution image. The AWB correction module may then apply these changes to the received high-resolution image, thereby generating an AWB-corrected image. Of course, it should be understood that calculations associated with changes to one or more pixel values ​​may be performed in the presence of a high-resolution image. At block 612, the AWB-corrected image may then be output for display. In various examples, the co-processing device may transmit the AWB-corrected image back to the HMD for display to the user.

[0060] Figure 7is a conceptual diagram depicting a system 700 for performing distributed AWB correction used in some embodiments. The system 700 may include an HMD 702 and a co-processing device 704. One or more images acquired from an imaging device 706 may be provided to an AEC statistics generator 708 and an AWB statistics generator 710. The image or frame may have a resolution that allows the image or frame to be transmitted wired or wirelessly from the HMD 702 to the co-processing device 704 and / or from the co-processing device 704 to the HMD 702, depending on one or more limitations of the connection. In some examples, the resolution of the image or frame may be reduced to comply with one or more limitations of the connection.

[0061] The AEC statistics generator 708 can generate AEC statistics, which may include, but are not limited to, the average brightness of the image, the standard deviation of one or more pixel values ​​in the image, the histogram or distribution of each pixel value in the image, the image contrast, the image saturation, and the image exposure time. The AEC statistics can be provided to the AEC module 712. The AEC module 712 can apply processing to the image to determine one or more adjustment parameters, such as exposure time and / or gain, which can be returned to the imaging device 706 via the communication path 714. In each example, the one or more adjustment parameters can be the actual exposure time and / or the actual gain setting, or can be a change in the exposure time or a change in the gain setting. Therefore, the imaging device 706 can combine these adjustments to obtain subsequent images or frames of the video.

[0062] The AWB statistics generated by the AWB statistics generator 710 may be transmitted to the AWB correction module 720 located at the co-processing device 704 via the wireless communication link 718. The AWB statistics may include, but are not limited to, the ratio of red pixels to green pixels, and in some cases, the ratio of blue pixels to green pixels. In some examples, the AWB statistics may also include the distribution of red pixels, green pixels, and blue pixels throughout the image. In some embodiments, the AWB statistics may be generated for the entire image; alternatively or additionally, the AWB statistics may be generated for one or more regions of the image. However, in general, the AWB statistics may be associated with an estimate of the ambient light temperature of the image.

[0063] The AWB correction module 720 may then perform AWB correction processing on the image received from the AEC module 712 via a wired or wireless communication link 716 by matching the received image with the previously received AWB statistics provided by the AWB statistics generator 710. The communication link 716 may be the same as or different from the communication link 718. In various examples, the AWB correction module 720 may perform processing that includes determining changes to one or more pixel values ​​in the image. Thus, based on the AWB statistics received from the AWB statistics generator 710, the co-processing device 704 may perform calculations associated with determining changes to one or more pixel values ​​in the absence of an image. The AWB correction module 720 may then apply these changes to the received image, thereby generating an AWB-corrected image or frame of the video. Of course, it should be understood that calculations associated with changes to one or more pixel values ​​may be performed in the presence of an image. The AWB correction module 720 may then provide the AWB-corrected image to one or more other ISP blocks 722 for additional processing. The resulting images provided by one or more other ISP blocks 722 may be transmitted wired or wirelessly back to the HMD for display to the user.

[0064] Figure 8 8 is a conceptual diagram depicting a system 800 for performing distributed AWB correction on high-resolution static images used in some embodiments. The system 800 may include an HMD 802 and a co-processing device 804. One or more images acquired from an imaging device 806 may be provided to an AEC statistics generator 808 and an AWB statistics generator 810. In various examples, the one or more images may be a series of low-resolution images (e.g., streaming images). Therefore, the statistics generated by the AEC statistics generator 808 and the AWB statistics generator 810 are based on the series of low-resolution images.

[0065] The AEC statistics generator 808 can generate AEC statistics, which may include, but are not limited to, the average brightness of the image, the standard deviation of one or more pixel values ​​in the image, the histogram or distribution of each pixel value in the image, the image contrast, the image saturation, and the image exposure time. The AEC statistics can be provided to the AEC module 812. The AEC module 812 can obtain a high-resolution image 814 from the imaging device 806 and apply processing to the image to determine one or more adjustment parameters, such as exposure time and / or gain, which can be returned to the imaging device 806 via the communication path 816. In each example, the one or more adjustment parameters can be an actual exposure time and / or an actual gain setting, or can be a change in the exposure time or a change in the gain setting. Therefore, the imaging device 806 can combine these adjustments to obtain a subsequent image, which can be a high-resolution image.

[0066] The AWB statistics generated by the AWB statistics generator 810 may be transmitted to an AWB correction module 820 located at the co-processing device 804 via a wired or wireless communication link 818. The AWB statistics may include, but are not limited to, a ratio of red pixels to green pixels, and in some cases, a ratio of blue pixels to green pixels. In some examples, the AWB statistics may also include a distribution of red pixels, green pixels, and blue pixels throughout the image. In some embodiments, the AWB statistics may be generated for the entire image; alternatively or additionally, the AWB statistics may be generated for one or more regions of the image. However, in general, the AWB statistics may be associated with an estimate of the ambient light temperature of the image.

[0067] The AWB correction module 820 may then perform an AWB correction process on the high-resolution image received from the AEC module 812 via the communication link 822. The communication link 818 may be the same as or different from the communication link 822. In various examples, the AWB correction module 820 may perform a process that includes determining a change to one or more pixel values ​​in the image. Thus, based on the AWB statistics received from the AWB statistics generator 810, the co-processing device 804 may perform calculations associated with determining a change to one or more pixel values ​​without the image being present. The AWB correction module 820 may then apply these changes to the received high-resolution image, thereby generating an AWB-corrected high-resolution image. Of course, it should be understood that calculations associated with changes to one or more pixel values ​​may be performed in the presence of the image. The AWB correction module 820 may then provide the AWB-corrected image to one or more other ISP blocks 824 for additional processing. The resulting image provided by the one or more other ISP blocks 824 may be transmitted back to the HMD for display to the user and / or storage in a storage location.

[0068] The "embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mentioned in this specification means that the specific features, structures, or characteristics described in conjunction with the embodiment are included in at least one embodiment of the present disclosure. The appearance of these phrases in different places in the specification does not necessarily refer to the same embodiment, nor is it necessarily a separate embodiment or alternative embodiment that is mutually exclusive of other embodiments. In addition, various features are described that may be present in some embodiments but not in other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not in other embodiments.

[0069] As used herein, being higher than a threshold value means that the value of the item being compared is higher than other values ​​specified, the item being compared is located in an item with a maximum value, a certain specified number, or the item being compared has a value within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of the item being compared is lower than other values ​​specified, the item being compared is located in an item with a minimum value, a certain specified number, or the item being compared has a value within a specified minimum percentage value. As used herein, being within a threshold value means that the value of the comparison item is between two other values ​​specified, the comparison item is located in an item of a middle specified number, or the comparison item has a value within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting fast connection" may be understood to mean selecting a connection with a value corresponding to the connection speed with which it is higher than a threshold value that is assigned.

[0070] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items, such as A and A; B, B and C; A, A, B, C and C; etc.

[0071] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as example forms of implementing the appended claims. Therefore, the embodiments and implementations are not limited except in the appended claims.

[0072] Any patent, patent application and other references mentioned above are incorporated herein by reference. If necessary, various aspects may be modified to provide further embodiments using the systems, functions and concepts of the various references mentioned above. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.

[0073] In some embodiments, the present technology may include: a method for performing the following steps; a computer-readable storage medium storing instructions that, when executed by a computing system, cause the computing system to perform a process including the following steps; or a computing system including one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the computing system to perform a process including the following steps: receiving an image from an imaging device of a head-mounted display device; obtaining resource requirements associated with an automatic white balance correction process for the image; using the resource requirements to determine that an amount of resources for performing the automatic white balance correction process is unavailable; based on determining that an amount of resources for performing the automatic white balance correction process is unavailable, performing distributed automatic white balance correction in the following manner: A) obtaining a low-resolution image of the image obtained from the imaging device; B) generating automatic white balance correction statistics using the low-resolution image; and C) providing the image and the automatic white balance correction statistics of the low-resolution image to a co-processing device via a network connection, wherein the co-processing device is located outside the head-mounted display device; and receiving the automatically white balance corrected image by the head-mounted display device.

Claims

1. A method for performing distributed automatic white balance correction on an image acquired from an imaging device of a head mounted display, the method comprising: receiving, at the co-processing device, an image from an imaging device of the head mounted display via a network connection, the head mounted display being located external to the co-processing device, wherein the head mounted display has applied automatic exposure correction to the received image; and wherein the head mounted display locally generates automatic white balance correction statistics for a version of the image without applying the automatic exposure correction to the version of the image, and provides the automatic white balance correction statistics to the co-processing device via the network connection; receiving, at the co-processing device and separately from receiving the image, the automatic white balance correction statistics generated by the head mounted display; Applying, by the co-processing device, at least one automatic white balance correction technique to the image based on the received automatic white balance correction statistics; and The automatically white balance corrected image is provided back to the head mounted display for output.

2. The method according to claim 1, in, The automatic white balance correction statistics are generated by the head mounted display for a version of the image that is a first resolution image; The resolution of the image received from the imaging device is greater than the resolution of the first resolution image.

3. The method according to claim 2, wherein: The automatic white balance correction statistics generated based on a version of the image at the first resolution are generated to adjust a higher resolution set of pixels in the image received from the imaging device.

4. The method according to claim 1, wherein: The automatic white balance correction statistics generated include one or more of the following: the ratio of red pixels to green pixels; the ratio of blue pixels to green pixels; the distribution of red pixels, green pixels and blue pixels throughout the image; an estimate of the ambient light temperature of the image; or any combination thereof.

5. The method according to claim 1, wherein: The version of the image for which the automatic white balance correction statistics are generated is a region of the image that is smaller than the entire image.

6. The method according to claim 1, wherein: The automatic exposure correction applied includes generating one or more automatic exposure correction statistics, and the one or more automatic exposure correction statistics include one or more of the following items: the average brightness of the image; the standard deviation of one or more pixel values ​​in the image; the histogram or distribution of each pixel value in the image; image contrast; image saturation; image exposure time; or any combination thereof.

7. The method according to claim 1, wherein: The applied automatic exposure correction determines an exposure time and / or gain, which is provided to the imaging device and used by the imaging device to acquire one or more subsequent images.

8. A computer-readable storage medium storing instructions which, when executed by a computing system, cause the computing system to perform a process for performing distributed automatic white balance correction on an image acquired from an imaging device of a head mounted display, the process comprising: receiving, at a co-processing device, an image from the imaging device of the head mounted display via a network connection, the head mounted display being external to the co-processing device, wherein the head mounted display locally generates automatic white balance correction statistics for a version of the image and provides the automatic white balance correction statistics to the co-processing device via the network connection; receiving, at the co-processing device and separately from receiving the image, the automatic white balance correction statistics generated by the head mounted display; and At least one automatic white balance correction technique is applied to the image by the co-processing device based on the received automatic white balance correction statistics.

9. The computer-readable storage medium according to claim 8, in, The head mounted display has applied automatic exposure correction to the received image; and wherein the version of the image is a version to which the automatic exposure correction is not applied.

10. The computer-readable storage medium of claim 8, wherein: The process also includes providing the automatically white balance corrected image back to the head mounted display for output.

11. The computer-readable storage medium according to claim 8, wherein: The applied automatic exposure correction determines an exposure time and / or gain, which is provided to the imaging device and used by the imaging device to acquire one or more subsequent images.

12. The computer-readable storage medium according to claim 8, in, The automatic white balance correction statistics are generated by the head mounted display for a version of the image that is a first resolution image; and The resolution of the image received from the imaging device is greater than the resolution of the first resolution image.

13. The computer-readable storage medium of claim 12, wherein: The automatic white balance correction statistics generated based on a version of the image at the first resolution are generated to adjust a higher resolution set of pixels in the image received from the imaging device.

14. The computer-readable storage medium of claim 8, wherein: The automatic white balance correction statistics generated include one or more of the following: the ratio of red pixels to green pixels; the ratio of blue pixels to green pixels; the distribution of red pixels, green pixels and blue pixels throughout the image; an estimate of the ambient light temperature of the image; or any combination thereof.

15. The computer-readable storage medium of claim 8, wherein: The version of the image for which the automatic white balance correction statistics are generated is a region of the image that is smaller than the entire image.

16. The computer-readable storage medium of claim 8, wherein: The automatic exposure correction applied includes generating one or more automatic exposure correction statistics, and the one or more automatic exposure correction statistics include one or more of the following items: the average brightness of the image; the standard deviation of one or more pixel values ​​in the image; the histogram or distribution of each pixel value in the image; image contrast; image saturation; image exposure time; or any combination thereof.

17. A computer-readable storage medium storing instructions which, when executed by a computing system, cause the computing system to perform a process for performing distributed automatic white balance correction on an image acquired from an imaging device of a head mounted display, the process comprising: receiving, at a co-processing device, an image from the imaging device of the head mounted display via a network connection, the head mounted display being external to the co-processing device, wherein the head mounted display locally generates automatic white balance correction statistics for a version of the image and provides the automatic white balance correction statistics to the co-processing device via the network connection; receiving, at the co-processing device and separately from receiving the image, the automatic white balance correction statistics generated by the head mounted display; and At least one automatic white balance correction technique is applied to the image by the co-processing device based on the received automatic white balance correction statistics.

18. The computing system of claim 17, in, The head mounted display has applied automatic exposure correction to the received image; and wherein the version of the image is a version to which the automatic exposure correction is not applied.

19. The computing system of claim 17, in, The automatic white balance correction statistics are generated by the head mounted display for a version of the image that is a first resolution image; wherein the resolution of the image received from the imaging device is greater than the resolution of the first resolution image; and wherein the automatic white balance correction statistics generated based on a version of the image at the first resolution are generated to adjust a higher resolution set of pixels in the image received from the imaging device.

20. The computing system of claim 17, wherein: The version of the image for which the automatic white balance correction statistics are generated is a region of the image that is smaller than the entire image.

Images