Computer vision processing circuit

Abstract

The invention relates to a computer vision processing circuit. There is provided an electronic device, which may include: a sensor for capturing an image; a computer vision processing (CVP) circuit that receives the captured image and has a subsystem that operates in a first power domain; a back-end image signal processing pipeline operating in a second power domain different from the first power domain; and optionally a display configured to receive content for display from the back-end image signal processing pipeline. A multi-client scheduler in the CVP circuit may receive and reorder image requests received from a plurality of client processors in the electronic device. The multi-client scheduler may interrogate an image server on the electronic device to determine whether an image request may be satisfied by an existing image currently stored on the image server.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 939,051, filed on November 6, 2024, and U.S. Provisional Patent Application No. 63 / 621,665, filed on January 17, 2024, the entireties of which are hereby incorporated by reference. Technical Field

[0002] The present disclosure generally relates to electronic devices, and more particularly to electronic devices having transparent displays. Background Art

[0003] Some electronic devices include transparent displays that present images near a user's eyes. The transparent displays allow a user's physical environment to be viewed through the transparent displays. For example, an extended reality headset may include a transparent display. Such electronic devices having transparent displays may include cameras for capturing images of the surrounding environment. It is in this context that the embodiments herein are generated. Summary of the Invention

[0004] One aspect of the present invention provides an electronic device including: one or more sensors configured to capture images; computer vision processing circuitry configured to receive the captured images and having a plurality of subsystems configured to operate in a first power domain; and a backend image signal processing pipeline configured to operate in a second power domain different from the first power domain. The electronic device may optionally include one or more displays configured to receive content for display from the backend image signal processing pipeline. The sensors may include one or more outward-facing cameras configured to capture images of the environment and / or one or more inward-facing cameras configured to capture images of the eyes. The computer vision processing circuitry may be configured to output a processed image according to a first image processing requirement, while the backend image signal processing pipeline may be configured to output a processed image according to a second image processing requirement different from the first image processing requirement. The backend image signal processing pipeline may be selectively deactivated, such as when the one or more displays are not outputting content.

[0005] One aspect of the present disclosure provides a method of operating an electronic device, the method comprising: outputting, by a plurality of clients running on the electronic device, an image request; receiving, by a multi-client scheduler, the image request from the client and feeding the image request into a queue; reordering, by the multi-client scheduler, at least some of the image requests in the queue; and fulfilling at least some of the image requests in the queue by triggering one or more image sensors in the electronic device to capture an image. A first client among the clients may be configured to execute a first set of algorithms, while a second client among the clients may be configured to execute a second set of algorithms different from the first set of algorithms. A first scheduling agent executed on the first client may be configured to manage a first request output from the first set of algorithms, while a second scheduling agent executed on the second client may be configured to manage a second request output from the second set of algorithms. The image requests may be reordered in the queue based on a deadline or timing requirement specified in the image request, a priority level associated with the image request, and / or whether the image request is associated with a user-facing algorithm or a non-user-facing algorithm.

[0006] One aspect of the present disclosure provides a method of operating an electronic device having one or more sensors. The method may include: running a plurality of clients on the electronic device; receiving, by a multi-client scheduler operable to communicate with the clients, an image request from one of the clients; and determining, before triggering the one or more sensors to capture a new image, whether the image request can be satisfied by an existing image currently stored on an image server within the electronic device. The method may further include: in response to determining that the image request can be satisfied by an existing image currently stored on the image server, returning a pointer to the existing image to the client; and in response to determining that the image request cannot be satisfied by an existing image currently stored on the image server, triggering the one or more sensors to capture a new image. The image request may include a requirement specifying one or more of the following: a timing or deadline requirement, an image resolution, an exposure level, a camera type, and a number of consecutive frames to be captured. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 is a diagram of an exemplary system having a transparent display in accordance with some embodiments.

[0008] Figure 2 is a diagram showing exemplary hardware components that may be included in Figure 1 a system of the type shown.

[0009] Figure 3Is a diagram showing an exemplary computer vision processing (CVP) circuit coupled to multiple processors (clients) according to some embodiments.

[0010] Figure 4 Is a diagram of an exemplary scheduling agent that can be executed on a processor according to some embodiments.

[0011] Figure 5 Is a diagram of an exemplary relationship computation graph according to some embodiments.

[0012] Figure 6 Is for operating according to some embodiments Figure 3 A flowchart of exemplary steps of an operating subsystem of the type shown in

[0013] Figure 7 Is a diagram showing an exemplary computer vision processing (CVP) circuit and multiple processors coupled to a shared image server according to some embodiments.

[0014] Figure 8 Is for operating according to some embodiments Figure 7 A flowchart of exemplary steps of an operating subsystem of the type shown in Detailed Description

[0015] The physical environment can refer to the physical world that people can sense and / or interact with without the help of electronic devices. The physical environment can include physical features, such as physical surfaces or physical objects. For example, the physical environment corresponds to a physical park that includes physical trees, physical buildings, and physical people. People can directly sense and / or interact with the physical environment, such as through vision, touch, hearing, taste, and smell.

[0016] In contrast, an extended reality (XR) environment refers to a fully or partially simulated environment that people sense and / or interact with via an electronic device. For example, an XR environment can include augmented reality (AR) content, mixed reality (MR) content, virtual reality (VR) content, etc. In the case of an XR system, a subset of a person's physical movements or their representations is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner that complies with at least one physical law.

[0017] As an example, an XR system can detect head movement and, in response, adjust the graphical content and sound field presented to a person in a manner similar to how such views and sounds would change in a physical environment. As another example, an XR system can detect movement of an electronic device (e.g., a mobile phone, a tablet computer, a laptop computer, etc.) presenting the XR environment and, in response, adjust the graphical content and sound field presented to a person in a manner similar to how such views and sounds would change in a physical environment. In some cases (e.g., for accessibility reasons), an XR system can adjust the characteristics of graphical content in the XR environment in response to a representation of physical movement (e.g., a voice command).

[0018] There are many different types of electronic systems that enable a person to sense and / or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed to be placed on a person's eyes (e.g., similar to contact lenses), headsets / earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablet computers, and desktop / laptop computers. A head-mounted system can have one or more speakers and an integrated opaque display. Alternatively, a head-mounted system can be configured to receive an external opaque display (e.g., a smartphone). A head-mounted system can incorporate one or more imaging sensors for capturing images or video of the physical environment and / or one or more microphones for capturing audio of the physical environment.

[0019] A head-mounted system can have a transparent or translucent display instead of an opaque display. The transparent or translucent display can have a medium through which light representing an image is directed to a person's eyes. The display can utilize digital light projection, organic light-emitting diodes (OLEDs), LEDs, micro-light-emitting diodes (uLEDs), liquid crystal on silicon, laser-scanned light sources, or any combination of these technologies. The medium can be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In some specific instances, the transparent or translucent display can be configured to selectively become opaque. A projection-based system can employ retinal projection techniques that project graphical images onto a person's retina. The projection system can also be configured to project virtual objects into the physical environment, such as as a hologram or on a physical surface.

[0020] Figure 1The system 10 (sometimes referred to as an electronic device 10, a head-mounted device 10, etc.) can be a head-mounted device having one or more displays. The display in the system 10 can include a display 20 (sometimes referred to as a near-eye display) mounted within a support structure (housing) 8. The support structure 8 can be shaped like a pair of glasses or goggles (e.g., a support frame), can form a housing having a helmet shape, or can have other configurations for assisting in mounting and securing components of the near-eye display 20 on or near the user's head. The near-eye display 20 can include one or more display modules such as display module 20A, and one or more optical systems such as optical system 20B. The display module 20A can be mounted in a support structure such as support structure 8. Each display module 20A can emit light 38 (image light) and redirect the light toward the user's eyes at the eye-zone 24 using an associated one of the optical systems 20B. The display 20 is optional and can be omitted from the device 10.

[0021] The operation of the system 10 can be controlled using the control circuit 16. The control circuit 16 can be configured to perform operations in the system 10 using hardware (e.g., dedicated hardware or circuitry), firmware, and / or software. Software code and other data for performing operations in the system 10 are stored on a non-transitory computer-readable storage medium (e.g., a tangible computer-readable storage medium) in the control circuit 16. The software code is sometimes referred to as software, data, program instructions, instructions, or code. The non-transitory computer-readable storage medium (sometimes generally referred to as memory) can include non-volatile memory such as non-volatile random access memory (NVRAM), one or more hard disk drives (e.g., disk drives or solid-state drives), one or more removable flash drives, or other removable media, etc. The software stored on the non-transitory computer-readable storage medium can be executed on the processing circuitry of the control circuit 16. The processing circuitry can include an application-specific integrated circuit having processing circuitry, one or more microprocessors, digital signal processors, graphics processing units, a central processing unit (CPU), or other processing circuitry.

[0022] System 10 may include input-output circuitry such as input-output device 12. Input-output device 12 may be used to permit system 10 to receive data from external devices (e.g., a tethered computer, a portable device such as a handheld device or a laptop computer, or other electrical devices), and to permit a user to provide user input to the head-mounted device 10. Input-output device 12 may also be used to collect information about the environment in which system 10 (e.g., the head-mounted device 10) operates. The output component in device 12 may permit system 10 to provide output to the user, and may be used to communicate with external electronic devices. Input-output device 12 may include one or more cameras 14 (sometimes referred to as image sensors 14). Camera 14 may be used to collect images of physical objects, which may optionally be digitally merged with virtual objects on a display in system 10. Input-output device 12 may include sensors and other components 18 (e.g., accelerometers, gyroscopes, depth sensors, light sensors, haptic output devices, speakers, batteries, wireless communication circuitry for communicating between system 10 and external electronic devices, etc.).

[0023] A camera 14 mounted on the front of system 10 and facing outward (toward the front of system 10 and away from the user) may sometimes be referred to herein as an outward-facing, externally-facing, forward, or front camera. Camera 14 may capture visual ranging information, image information that is processed to locate objects in the user's field of view (e.g., such that virtual content may be appropriately registered relative to real-world objects), image content that is displayed in real-time to a user of system 10, and / or other suitable image data. For example, the outward-facing camera may permit system 10 to monitor movement of system 10 relative to the environment surrounding system 10 (e.g., the camera may be used as part of a visual ranging system or a visual-inertial ranging system). The outward-facing camera may also be used to capture images of the environment that are displayed to a user of system 10. If desired, images from multiple outward-facing cameras may be merged with each other and / or outward-facing camera content may be merged with computer-generated content for the user.

[0024] The display module 20A can be a liquid crystal display, an organic light emitting diode display, a laser-based display, or other types of displays. The optical system 20B can form a lens that allows an observer (see, for example, the eyes of the observer at the viewing zone 24) to view an image on the display 20. There can be two optical systems 20B associated with the user's respective left and right eyes (e.g., for forming a left lens and a right lens). A single display 20 can generate images for both eyes, or a pair of displays 20 can be used to display images. In a configuration with multiple displays (e.g., a left-eye display and a right-eye display), the focal length and positioning of the lenses formed by the system 20B can be selected such that any gap present between the displays will be invisible to the user (e.g., such that the images of the left and right displays overlap or merge seamlessly).

[0025] If desired, the optical system 20B can include a transparent structure (e.g., an optical combiner, etc.) that allows image light from a physical object 28 to be optically combined with virtual (computer-generated) images such as the virtual image in the image light 38. Light from a physical object 28 in the physical environment or scene can sometimes be referred to and defined herein as world light, scene light, ambient light, external light, or environmental light. In this type of system, a user of the system 10 can view both the physical environment surrounding the user and computer-generated content overlaid on the physical environment. The camera 14 can also be used in the device 10 (e.g., an arrangement where the camera captures an image of the physical object 28 and modifies the content and presents it as virtual content at the optical system 20B).

[0026] If desired, the system 10 can include wireless circuitry and / or other circuitry to support communication with a computer or other external equipment (e.g., a computer that supplies image content to the display 20). During operation, the control circuit 16 can provide image content to the display 20. The content can be received remotely (e.g., from a computer or other content source coupled to the system 10) and / or can be generated by the control circuit 16 (e.g., text, other computer-generated content, etc.). The content provided by the control circuit 16 to the display 20 can be viewed by an observer at the viewing zone 24.

[0027] Figure 2 is a diagram showing illustrative hardware components that can be included within a system (e.g., device 10) of the type described in connection with Figure 1 as shown Figure 2As shown, device 10 may include one or more hardware and / or software subsystems, including one or more outward-facing image sensing subsystems (such as outward-facing camera 50), one or more tracking subsystems (such as tracking sensor 54), computer vision processing (CVP) circuitry (such as CVP circuitry 60), a separate image signal processing pipeline (such as a high-quality (backend) pipeline 72), and one or more displays 20.

[0028] One or more cameras 50 may be used to collect information about the external real-world environment or scene surrounding device 10. Camera 50 may include Figure 1 one or more front cameras of the front cameras 14. At least some of the cameras 50 may be configured to capture a series of images of a scene, and the series of images may be presented to the user using display 20 as a live video pass-through feed. The live video pass-through feed is sometimes referred to as pass-through content. Such front cameras used to obtain pass-through content are sometimes referred to as scene or pass-through cameras. Camera 50 may include a color image sensor and / or an optional monochrome (black and white) image sensor.

[0029] The cameras 50 may have different fields of view. Some of the cameras 50 may have a wide or ultra-wide field of view, while some of the cameras 50 may have a relatively narrow field of view. Not all of the cameras 50 need to be used to capture pass-through content. Some of the cameras 50 may be forward-facing (e.g., oriented towards the scene in front of the user); some of the cameras 50 may be downward-facing (e.g., oriented towards the user's torso, hands, or other parts of the user); some of the cameras 50 may be side / landscape-facing (e.g., oriented towards the left and right sides of the user); and some of the cameras 50 may be oriented in other directions relative to the front of device 10. All of these cameras 50 configured to collect information about the external physical environment surrounding device 10 are sometimes referred to as and defined as "outward-facing" or "outward" cameras.

[0030] The tracking sensor 54 may include a gaze tracking subsystem, sometimes referred to as a gaze tracker, configured to acquire gaze information or point-of-gaze information. The gaze tracker may employ one or more inward-facing cameras and / or other gaze tracking components (e.g., eye-facing components and / or other light sources that emit light beams such that reflections of the light beams from the user's eyes can be detected) to monitor the user's eyes. One or more gaze tracking sensors 54 may be oriented towards the user's eyes and may track the user's gaze. The cameras in the gaze tracking subsystem may determine the position of the user's eyes (e.g., the center of the user's pupil), may determine the direction in which the user's eyes are oriented (the direction of the user's gaze), may determine the pupil size of the user (e.g., such that light modulation and / or other optical parameters are adjusted based on the pupil size, and / or the sequential amount for spatially adjusting one or more of these parameters, and / or the region of one or more of these optical parameters), may be used to monitor the current focal length of the lens in the user's eye (e.g., whether the user is focused on the near field or the far field, which may be used to evaluate whether the user is daydreaming or thinking strategically or tactically), and / or other gaze information. The gaze tracking cameras may sometimes be referred to as inward-facing cameras, gaze detection cameras, eye tracking cameras, gaze tracking cameras, or eye monitoring cameras. If desired, other types of optical sensors (e.g., infrared and / or visible light emitting diodes and light detectors, etc.) may also be used to monitor the user's gaze.

[0031] The tracking sensor 54 may also include a face and body tracking subsystem configured to perform face tracking (e.g., to capture images of the user's jaw, mouth, etc. when the device is worn on the user's head) and body tracking (e.g., by capturing images of the user's torso, arms, hands, legs, etc. when the device is worn on the user's head). If desired, the face and body tracking subsystem may also track the user's head pose by directly determining any movement, yaw, pitch, roll, etc. of the head-mounted device 10. The yaw, roll, and pitch of the user's head may collectively define the "head pose" of the user. For example, the tracking sensor 54 may include an inertial measurement unit (IMU). The inertial measurement unit may include one or more gyroscopes, gyrocompasses, accelerometers, magnetometers, other inertial sensors, and other position and motion sensors. These position and motion sensors may assume that the head-mounted device 10 is mounted on the user's head. Thus, in this document, references to head pose, head movement, yaw of the user's head (e.g., rotation about the vertical axis), pitch of the user's head (e.g., rotation about the left-right axis), roll of the user's head (e.g., rotation about the front-back axis), etc. may be considered interchangeable with references to device pose, device movement, yaw of the device, pitch of the device, roll of the device, etc. In certain embodiments, the tracking sensor 54 may also include a six degrees of freedom (DoF) tracking subsystem. A six DoF tracking subsystem or sensor may be used to monitor rotational motions such as roll, pitch, and yaw as well as position / translation motions in a 3D environment.

[0032] The tracking sensor 54 may optionally also include a hand tracking subsystem sometimes referred to as a hand tracker, which is configured to monitor the user's hand movements / gestures to obtain hand gesture data. For example, the hand tracker may include a camera and / or other gesture tracking components (e.g., outward-facing components and / or light sources that emit light beams such that reflections of the light beams from the user's hand can be detected) to monitor the user's hand. One or more hand tracking sensors may be directed at the user's hand and may track the movements associated with the user's hand, determine whether the user is performing a tap or swipe movement with his / her fingertips or hand, determine whether the user is performing a non-contact button press or object selection operation with his / her hand, determine whether the user is performing a grasping or holding movement with his / her hand, determine whether the user is using his / her hand or finger to point at or pinch a given object presented on the display 20, determine whether the user is performing a waving or hitting movement with his / her hand, or may generally measure / monitor three-dimensional non-contact gestures ( "air gestures") associated with the user's hand. The tracking sensor 54, which is operable to obtain gaze, pose, hand gesture, and other information related to the movement of the user of the device 10, is sometimes collectively referred to as a "user tracking" sensor.

[0033] Figure 2 An example in which the outward-facing camera 50 and the tracking sensor 54 (e.g., an optical sensor for obtaining gaze, pose, and / or other user-related data) are shown as separate and distinct subsystems is illustrative. In some embodiments, one or more of the outward-facing cameras 50 may also be employed to obtain pose information, location information, and / or other movement / location information associated with the device 10. To help protect the privacy of the user, best practices may be used to handle any personal user information collected by the sensors. These best practices include meeting or exceeding any applicable privacy rules. Opt-in and opt-out options and / or other options may be provided that allow the user to control the use of their personal data.

[0034] The electronic device 10 can be configured to collect context information about the surrounding real-world (physical) environment or scene. Collecting context information can, for example, include identifying one or more objects of interest in the environment, detecting when a user enters a particular room or environment, detecting when a user engages in a particular activity, detecting the current location of the device 10, detecting the current user context or usage scenario (e.g., detecting whether the user is currently watching a movie, playing a video game, or talking to another person or avatar), and / or determining other context information related to the operation of the device 10. Collecting context information can involve using the outward-facing camera 50 to capture one or more images and / or obtaining data from the tracking sensor 54. Such images captured for context purposes need not be output by the display 20 for human consumption. Thus, the processing requirements and complexity for handling such images can be less than the conventional image signal processing steps required to process images output by the display for human consumption (viewing).

[0035] According to one embodiment, the image signal processing circuitry on the device 10 can be segmented into a first part including computer vision processing (CVP) circuitry 60 and a separate second part including a high-quality (HQ) pipeline 72. Images and / or data outputs from the sensors 50 and 54 that only need to be analyzed for context purposes (not processed by the high-quality pipeline 72) can be processed using only the CVP circuitry 60, while images and / or data outputs from the sensors 50 and 54 that are to be output on the display 20 for human viewing can be processed by the CVP circuitry 60 and the high-quality pipeline 72. Components within the CVP circuitry 60 can operate in a first power domain, while components within the HQ pipeline 72 can operate in a second power domain different from the first power domain (e.g., the CVP circuitry 60 and the HQ pipeline 72 can be configured to operate in different power domains).

[0036] Components in the CVP circuit 60 can generally operate in a lower power domain relative to components in the HQ pipeline 72. The high-quality pipeline 72 can be power-gated. When processing an image to be output on the display 20 for human consumption, the high-quality pipeline 72 can be selectively activated (e.g., powered on) to perform some or all of the image processing functions provided by the HQ pipeline 72. When processing an image only for context purposes (e.g., to support one or more computer vision algorithms running on the device 10) without having to display such an image, the HQ pipeline 72 can be selectively deactivated (e.g., powered off or idled) to save power. In other words, the CVP circuit 60 consumes a first amount of power when activated, while the HQ pipeline 72 consumes a second amount of power greater than the first amount of power when activated. Operating the image signal processing circuit on the device 10 in this manner can be technically beneficial for minimizing power consumption on the device 10. This is beneficial for small and lightweight devices 10 that can be battery-powered for all-day use.

[0037] As Figure 2 shown, the CVP circuit 60 can include one or more hardware and / or software subsystems, such as a sensor interface 62, a front-end (FE) processor 64, a statistical front-end (FE) processor 66, a statistical back-end (BE) processor 68, a central processing unit (CPU) (such as a computer vision processing (CVP) CPU 70) and / or other image signal processing components. The sensor interface 62 can be configured to receive images (e.g., raw pixel data) from the camera 50, the tracking sensor 54, and / or other image sensors within the device 10. The front-end processor 64 can be configured to perform bad / defective pixel correction, image scaling or merging operations, image cropping or resizing, and / or other front-end or image preprocessing operations. The statistical FE processor 66 can be configured to collect pixel statistics, such as minimum pixel value, maximum pixel value, average pixel value, color plane information (e.g., red, green, and blue color planes), color, and / or luminance histograms, and other front-end image statistics. The statistical BE processor 68 can be configured to convert an image from the raw Bayer domain to a color image and can generate additional statistical information.

[0038] The color image output from the statistical BE processor 68 can be provided to one or more downstream computer vision processing algorithms or tasks running on the device 10 (e.g., the processor 68 can output the image to one or more client processors). The statistical FE processor 66 and the BE processor 68 can be collectively referred to as the CVP statistical pipeline. Although the CVP circuit 60 is shown as including a single instance of the interface 62, the processor 64, the processor 66, and the processor 68, the CVP circuit 60 can include multiple sensor interface blocks 62 for interfacing with multiple sensors, multiple front-end processors 64 for performing image preprocessing operations in parallel, multiple processors 66 for performing front-end statistical calculations in parallel, and multiple processors 68 for performing back-end statistical calculations in parallel. The computer vision processing CPU 70 can be configured to manage and coordinate the operations of the blocks 62, 64, 66, and 68 to process each incoming image frame.

[0039] The computer vision processing circuit 60 mainly includes components for performing front-end image signal processing operations. Therefore, the computer vision processing circuit 60 is sometimes referred to as a "front-end" image signal processing (ISP) circuit. In contrast, the HQ pipeline 72 mainly includes components configured to perform back-end image signal processing operations. Therefore, the high-quality pipeline 72 is sometimes referred to as a "back-end" image signal processing (ISP) circuit. The high-quality (back-end) pipeline 72 can be a more complex and higher power consumption version of the statistical back-end processor 68 of the CVP circuit 60. For example, the HQ pipeline 72 can include components configured to perform bad / defective pixel correction, noise reduction, white balance, demosaicing, color space conversion, tone mapping (e.g., including global and local tone mapping), color correction, gamma correction, shadow correction, image sharpening, high dynamic range (HDR) correction, edge-aware local image adjustment, image fusion (e.g., fusing multiple image frames together for noise reduction and high dynamic range), image signal processing operations not present in the CVP circuit 60 at all, and / or other image signal processing functions for outputting the corresponding image for display.

[0040] The images output by the back-end processor 68 of the CVP circuit 60 may be processed according to a first set of image processing requirements that may optionally produce a lower fidelity (quality) image for computer vision consumption, while the images output by the HQ pipeline 72 may be processed according to a second set of image processing requirements that are different from the first set of image processing requirements that may optionally produce a relatively higher fidelity (quality) image to be displayed for human consumption. In some embodiments, the CVP circuit 60 may be configured to output a processed image having a first quality and / or using a first amount of power, while the HQ pipeline 72 may be configured to output a processed image having a second quality greater than the first quality and / or using a second amount of power greater than the first amount of power. In some embodiments, the CVP circuit 60 may be configured to output a processed image by performing a first set of image processing operations, while the HQ pipeline 72 may be configured to output a processed image by performing additional image processing operations that are different from the first set of image processing operations. The image output by the processor 68 may be provided as a result to one or more client processors (see, e.g., Figure 3 ), wherein the HQ pipeline 72 can output the content for human consumption via the display 20 Figure 2 The examples are illustrative. Display 20 is optional and may be omitted from device 10. If desired, the content output from HQ pipeline 72 may be stored in memory for later processing.

[0041] Figure 3 1 shows how computer vision processing circuitry 60 may be further coupled to one or more processors 80 within device 10. Figure 3 As shown, the CVP circuit 60 can be coupled to at least three different processors 80 within the device 10. This is illustrative. In general, the CVP circuit 60 can be coupled to two or more processors 80, three or more processors 80, four or more processors 80, or any suitable number of processors 80 within the device 10. The various processors 80 coupled to the CVP circuit 60 can have different computing capabilities and different power requirements. At least one of the processors 80 can act as a main processor (sometimes referred to as an application processor), which typically has a more complex software stack and consumes more power than the other processors 80. The application processor can be responsible for driving the display 20. The other processors 80 can be provided with a simpler software stack and can consume less power than the main (application) processor. Because the application processor consumes more power, the CVP circuit 60 can be configured to wake up the application processor only when necessary.

[0042] Each processor 80 can be configured to run or execute one or more algorithms 82. At least some of the algorithms 82 can be computer vision processing algorithms. As an example, one of the low-power processors 80 can be configured to perform gaze detection by running a gaze detection algorithm 82. As another example, one of the low-power processors 80 can be configured to run a context vision processing algorithm 82 to detect certain objects of interest within the environment. As another example, one of the low-power processors 80 can be configured to run a context vision processing algorithm 82 to determine whether a user is in a particular room or a particular type of environment. As another example, one of the low-power processors 80 can be configured to run a context vision processing algorithm 82 to detect whether a user is engaged in a certain type of activity (e.g., detecting when a user is exercising, walking, sleeping, eating, etc.). As yet another example, one of the low-power processors 80 can be configured to run a networking algorithm 82.

[0043] The various algorithms 82 can generate requests to complete certain tasks or jobs, such as requests to capture one or more images by a sensor. According to some embodiments, each processor 80 can be provided with a local intelligent scheduler 84 that is configured to determine a more insightful schedule for the execution of time-sensitive tasks and jobs requested by the algorithms 82 on that processor 80. The scheduler 84 (sometimes referred to herein as a scheduling agent or algorithm scheduler) can receive time-sensitive tasks from one or more algorithms 82 on the same processor 80 and output an optimized schedule for the requested tasks. The scheduler 84 can be considered to reside in the application (or user) layer. The application layer can refer to a class of programs that execute in a mode with restricted privileges. In such a mode, a program can be prohibited from executing instructions defined by a particular instruction set architecture.

[0044] Figure 4 Illustrates how the scheduler 84 can include a system health monitor 110, a graph analyzer 112, and an executor 114. The system health monitor 110 can be configured to monitor ongoing changes in the power, performance, and thermal statistics of the device 10 to proactively determine the current health level of the system, which can then be used by the scheduler 84 to determine which resources are currently available for executing time-sensitive tasks and schedule the time-sensitive tasks accordingly. The scheduler 84 can proactively modify its schedule as the system health changes over time. For example, if the scheduler 84 determines based on the health information that it will no longer be able to meet a particular timing constraint in the near future, the scheduler 84 can contact the algorithms that requested the time-sensitive tasks and allow them to decide how to handle the deteriorating health of the device before it reaches a certain problematic threshold.

[0045] The graph analyzer 112 can obtain an overall view of the time-sensitive task being requested by analyzing the current health levels, computational graphs, and other metadata output from the system health monitor 110 in order to generate an appropriate schedule for the time-sensitive task. Figure 5 An illustrative relational computational graph 116 showing the interrelationships of tasks to be performed by the algorithm 82 is shown. For a given requested time-sensitive task, such a computational graph 116 can include graph nodes that specify time-sensitive tasks that provide inputs to be used when executing the given time-sensitive task and time-sensitive tasks that should receive the output of the given task once the given task is completed. The computational graph 116 can also include additional information such as the resources required to execute the time-sensitive task, the timing constraints associated with the time-sensitive task, the interrelationships of different tasks, etc. Based on this information, the scheduler 84 can determine a corresponding schedule 118 indicating how the time-sensitive task should be implemented to improve performance and optimize resource usage. The scheduler 84 is thus sometimes also referred to as a resource manager. In some embodiments, the scheduler 84 can focus on identifying the critical path in the execution of a set of time-sensitive tasks and attempt to schedule the tasks along that critical path in a manner that meets the timing constraints of the time-sensitive tasks. The executor 114 can consume or execute the schedule 118 determined by the graph analyzer 112.

[0046] Configured in this way, the scheduler 84 can act as a processor-level or client-level gating agent that determines whether requests received from the algorithm 82 should be forwarded to the CVP circuit 60. The scheduler 84 can decide not to pass the request to the CVP circuit 60 if it determines that the received request can be locally processed (e.g., by retrieving an image that has been or recently captured). In some embodiments, the scheduler 84 can allocate a set of credits among the various algorithms 82 running on a given processor 80. The amount of credit assigned by the scheduler 84 to each algorithm 82 determines the number of requests that each algorithm 82 can make. For example, a first amount of credit can be assigned to a first algorithm 82; a second amount of credit different from the first amount can be assigned to a second algorithm 82; a third amount of credit different from the first and second amounts can be assigned to a third algorithm 82; and so on. Such credits can be used to manage shared access to a bounded memory pool, ensuring that no single client exceeds the allotted amount of memory. The scheduler 84 can thus provide processor-level arbitration among requests issued from the various algorithms 82.

[0047] Each processor 80 can independently output one or more requests for performing time-sensitive tasks to the CVP circuit 60. For example, the scheduler 84 in each processor 80 can output one or more requests to the client interface 90 of the CVP circuit 60. The client interface 90 can be configured to communicate with or receive requests from multiple scheduling agents 84 in multiple processors 80. Thus, the various processors 80 that independently send requests to the client interface 90 of the CVP circuit 60 are sometimes referred to as "clients", client processors, or client subsystems. Thus, requests received from various clients are sometimes referred to as client requests. An example of each client being implemented as a separate processing unit Figure 3 is illustrative. In some embodiments, more than one client can be implemented on a single processor. If desired, multiple clients and the CVP circuit 60 can optionally be implemented as part of a single processor or system-on-chip within the device 10. In other embodiments, at least some of the multiple clients can optionally be implemented as one or more processors or part of a system-on-chip within another electronic device separate from the device 10.

[0048] The computer vision processing circuit 60 can include a multi-client scheduling subsystem, such as a multi-client scheduler 92, which is configured to arbitrate requests received from various client handlers 80. In other words, a "multi-client" scheduler can refer to and is hereby defined as a scheduling agent configured to receive requests from multiple clients running on the device 10 or on multiple electronic devices 10. The multi-client scheduler 92 is thus sometimes referred to as a multi-client arbitration block. The multi-client scheduler 92 can be Figure 2 part of the computer vision processing CPU 70 shown. The multi-client scheduler 92 can push the received client requests into a queue 94, which is sometimes referred to as the multi-client scheduler queue. The multi-client scheduler 92 can be responsible for establishing tasks or jobs corresponding to client requests received from different processors 80. The multi-client scheduler 92 can optionally consolidate hardware usage between two separate requests. For example, two independent client image capture requests for an outward-facing camera 50 with different image resolution requirements for a wide field of view (FoV) can be fulfilled by a single image capture at the higher resolution requirement and then scaled down to meet the other request with the lower resolution requirement.

[0049] The multi-client scheduler 92 can also be configured to optimize power savings. For example, if there are no upcoming requests for the sensor in queue 94, the scheduler 92 can power off the sensor. As another example, if there are no client requests within the upcoming few milliseconds in queue 94 (e.g., if queue 94 is empty, or if there are no requests with deadlines to be met within the next 10 milliseconds, within the next 1 millisecond - 10 milliseconds, within the next 10 milliseconds - 100 milliseconds, etc.), the scheduler 92 can temporarily instruct the CVP circuit 60 to enter a sleep state. The scheduler 92 can be configured to wake up the CVP circuit 60 in response to receiving a new client request. As another example, the scheduler 92 can optionally treat a streaming request as a one-shot image capture for periodic scheduling. As another example, the scheduler 92 can re-use the sensor for different captures for low frame rate streaming requests (e.g., for 1Hz streaming requests, for 2Hz streaming requests, for 1Hz - 5Hz streaming requests, etc.).

[0050] Images captured based on client requests received at the CVP circuit 60 can be stored on the memory 100. The memory 100 can be a volatile memory (e.g., dynamic or static random access memory), a non-volatile memory (e.g., flash memory or other electrically programmable read-only memory configured to form a solid state drive), or other types of storage devices. The captured images stored on the memory 100 can be organized as part of an image server, such as the image server 102, which represents a database of recently captured images or historical images. Older images (e.g., images captured more than a minute ago, more than ten minutes ago, more than an hour ago, more than a day ago, or images captured when the user was in a different location or when the user was engaged in a different activity) may no longer be relevant and can optionally be deleted from the image server 102. Each client processor 80 can also retrieve images directly from the image server 102 (e.g., via the data path 104). The memory 100, which can be accessed by the CVP circuit 60 and each client processor 80, can thus sometimes be referred to as shared memory.

[0051] Figure 6 is a flowchart of illustrative steps for operating an operating subsystem of the type described in connection with Figures 3 to 5 During the operation of block 200, the scheduling agent 84 within each client processor 80 can locally manage requests received from the various algorithms 82. The scheduler 84 can analyze one or more computational graphs associated with the algorithms 82 and can determine an optimized schedule for the requested tasks.

[0052] During operation of block 202, the CVP circuit 60 may receive client requests from one or more processors 80. The various processors 80 may independently send client requests to the client interface 90. For example, the client interface 90 may receive requests from multiple scheduling agents 84 in parallel. The client requests received at the interface 90 may be pushed into the scheduler queue 94.

[0053] During operation of block 204, the multi-client scheduler 92 may sort the requests in queue 94 based on deadlines, priority levels, algorithm types, and / or other parameters. As an example, requests with closer deadlines may be pushed to the front of the queue, while requests with later deadlines may be pushed to the back of the queue. As another example, requests marked with a higher priority level may be pushed to the front of the queue, while requests marked with a lower priority level or requests with no priority marking may be pushed to the back of the queue. As yet another example, requests associated with user-facing algorithms may be considered higher priority and pushed to the front of the queue, while non-user-facing background algorithms may be considered lower priority and pushed to the back of the queue. User-facing algorithms may include algorithms for displaying an enlarged version of one or more objects within the user's field of view (e.g., to increase the readability of small text). User-facing algorithms may also include algorithms related to tracking the user's gaze (e.g., tracking when the user's gaze aligns with one or more user interface elements being displayed by the device 10).

[0054] During operation of block 206, the multi-client scheduler 92 may optionally coalesce, combine, or merge two or more related requests. For example, consider a scenario where the CVP circuit 60 receives three different image capture requests where the first request requires an image at full (highest) resolution, where the second request requires a lower resolution image with 2x2 pixel binning, and where the third request requires an even lower resolution image with 4x4 pixel binning. Here, the multi-client scheduler 92 may determine that a single image capture at the highest (full) resolution is sufficient to satisfy all three requests. For example, the multi-client scheduler 92 may employ a pyramid scaler that is part of the front-end image signal processing circuitry to downscale the full-resolution image (e.g., via 2x2 binning) to satisfy the second request, and to downscale the full-resolution image (e.g., via 4x4 binning) to satisfy the third request.

[0055] During operation of block 208, the CVP circuit 60 may fulfill requests in queue 94 according to the order determined from block 204. The multi-client scheduler 92 may fulfill requests in the queue based on a best-effort service approach. If the multi-client scheduler 92 cannot complete all required tasks before the specified deadline, the scheduler 92 may send a negative acknowledgement back to the corresponding client processor 80 (via the client interface 90) so that the processor is aware that one or more requests have not been fulfilled before the specified deadline. The multi-client scheduler 92 may also optionally discard low-priority requests if it does not have sufficient bandwidth to fulfill all requests in a timely manner.

[0056] During operation of block 210, the multi-client scheduler 92 may optionally obtain results from one algorithm or client to satisfy a request from another algorithm or client. For example, consider a scenario where the CVP circuit 60 is currently fulfilling a streaming request for a user-facing algorithm. Such a streaming request may acquire a series of images. During this streaming, the CVP circuit 60 may receive a one-time capture request from another algorithm that only needs a single image for context computer vision processing. In such a case, the multi-client scheduler 92 may extract a selected image frame from the series of streaming images and pass the selected image to the CVP algorithm to satisfy the one-time capture request. If needed, the multi-client scheduler 92 may optionally employ a pyramid scaler to reduce the resolution of one or more images passed to a downstream algorithm for computer vision consumption while the images presented to the display for human consumption remain at full (highest quality) resolution.

[0057] Figure 6 The operations are illustrative. In some embodiments, one or more of the described operations may be modified, replaced, or omitted. In some embodiments, one or more of the described operations may be performed in parallel. In some embodiments, additional processes may be added or inserted between the described operations. If needed, the order of certain operations may be reversed or changed, and / or the timing of the described operations may be adjusted so that they occur at slightly different times. In some embodiments, the described operations may be distributed across a larger system.

[0058] The computer vision processing circuit 60 may keep track of the most recent image captures and may optionally use one or more of the most recent image captures to respond to client requests from the processor 80 when certain requirements trigger a new capture. Figure 7 is an illustration showing how the CVP circuit 60 and the client processor 80 are coupled to a shared image 100. As Figure 7As shown, an image server 102 with a repository of recently captured images and / or historically captured images can be stored on a memory 100. The image server 102 (sometimes referred to as an image database or image repository) can be accessed by a CVP circuit 60 and various client processors 80. The CVP circuit 60 can have read and write privileges to the image server 102 (e.g., in addition to retrieving data from the image server 102, the computer vision processing CPU 70 can also update or add entries in the image server 102). In contrast, the client processors 80 may have read-only privileges to the image server 102 (e.g., each processor 80 can only retrieve data from the image server 80). Each processor 80 can issue an image request to the CVP circuit 60. In response to receiving the image request, the CVP circuit 60 can return a pointer (e.g., a memory pointer) to the requesting processor 80, and the processor 80 can use this pointer to retrieve the corresponding image from the server 102.

[0059] Figure 8 is for operating in conjunction with Figure 3 and Figure 7 is a flowchart of exemplary steps for operating an operating subsystem of the type described. During the operation of block 300, the CVP circuit 60 can receive an image request from the client processor 80. The image request is a request for an image that meets certain requirements and is sometimes referred to as a client request. During block 300, the CVP circuit 60 can receive client requests from multiple processors 80 simultaneously.

[0060] During the operation of block 302, the CVP circuit 60 queries the image server 102 to check if the requested image currently resides on the image server 102. The CVP circuit 60 can check if the existing images on the server 102 meet one or more of the requirements in the image request. As an example, the image request can request a well-exposed image captured within the last 5 seconds using an outward-facing camera 50 with a wide field of view at full resolution ( Figure 2 ). The CVP circuit 60 can determine if this image already exists on the server 102 (e.g., by checking if the given image is bright enough, if the given image was captured using the wide-angle camera 50, if the given image was captured within the last 5 seconds, and if the given image was captured at the highest resolution), and if so, return a pointer to that image to the requesting client 80 without having to trigger a new image capture. Processing client requests in this way can be technically advantageous and beneficial in avoiding having to capture new image frames and can minimize the power consumption of the device 10.

[0061] The above examples where the image request lists requirements regarding exposure level, resolution, timing, and camera type are illustrative. For example, the image request can specify a timing deadline (e.g., the time point by which the requested image must be returned to the client). For example, the image request can specify a past time point at which the image should be captured. For example, the image request can specify a future time point at which the image will be captured. For example, the image request can specify the minimum signal-to-noise ratio (SNR) of the image. For example, the image request can specify the maximum amount of motion blur that can be tolerated. For example, the image request can specify the number of consecutive frames that need to be captured. For example, the image request can specify the type of frames that need to be captured. For example, the image request can specify a number of bracketed frames with EV (exposure value) apertures that do not need to be captured.

[0062] These example requirements or parameters are illustrative. In some cases, all the specified requirements in the request must be met. In some cases, only a subset of the specified requirements in the request must be. If desired, at least some of the specified requirements can be optionally relaxed. For example, a slightly underexposed image that otherwise meets all other requirements can be considered a sufficient match. For example, an image captured 6 seconds ago that violates the requirement of capturing an image within the past 5 seconds but otherwise meets all other requirements can be considered a sufficient match.

[0063] During the operation of block 304, if the CVP circuit 60 determines that the requested image already exists on the image server 102 (e.g., if the circuit 60 identifies an existing image on the server 102 that sufficiently meets the specified requirements), then the CVP circuit 60 can return a pointer or image identifier to that image and optionally increment the reference count of the image. The reference count can be used to save the image for the client. If the client no longer needs the image, the CVP circuit 60 can decrement or reset the reference count to release the image. The released image can be deleted from the image server 102 within a specified time period to clear space for new images.

[0064] During the operation of block 306, if the CVP circuit 60 determines that the requested image does not exist on the image server 102 (e.g., if the circuit 60 does not find a match), then the CVP circuit 60 can instruct one or more corresponding sensors on the device 10 to capture one or more new images to satisfy the client request. Although Figure 8 Block 306 is shown as occurring after block 304, only one of the operations of block 304 and block 306 is performed for any given client image request. Thus, if the operation of block 304 is being performed, the operation of block 306 can be skipped. Conversely, if the operation of block 306 is being performed, the operation of block 304 can be skipped.

[0065] Figure 8The operations are merely illustrative. In some embodiments, one or more of the described operations may be modified, replaced, or omitted. In some embodiments, one or more of the described operations may be performed in parallel. In some embodiments, additional processes may be added or inserted between the described operations. If desired, the order of certain operations may be reversed or changed, and / or the timing of the described operations may be adjusted such that they occur at slightly different times. In some embodiments, the described operations may be distributed across a larger system.

[0066] According to one embodiment, an electronic device is provided that includes: one or more sensors configured to capture images; computer vision processing circuitry configured to receive the captured images and having a plurality of subsystems configured to operate in a first power domain; and a backend image signal processing pipeline coupled to the computer vision processing circuitry and configured to operate in a second power domain different from the first power domain.

[0067] According to another embodiment, the electronic device optionally includes one or more displays configured to receive content for display from the backend image signal processing pipeline.

[0068] According to another embodiment, the one or more sensors optionally include one or more outward-facing cameras configured to capture images of the environment.

[0069] According to another embodiment, the one or more sensors optionally include one or more inward-facing cameras configured to capture images of the eyes.

[0070] According to another embodiment, the computer vision processing circuitry is optionally further configured to output a processed image according to a first image processing requirement, and the backend image signal processing pipeline is optionally further configured to output a processed image according to a second image processing requirement different from the first image processing requirement.

[0071] According to another embodiment, the computer vision processing circuitry is optionally configured to output a processed image having a first quality or using a first amount of power, and the backend image signal processing pipeline is optionally configured to output a processed image having a second quality greater than the first quality or using a second amount of power greater than the first amount of power.

[0072] According to another embodiment, the computer vision processing circuit is optionally configured to output a processed image by performing a first set of image processing operations, and the backend image signal processing pipeline is optionally configured to output a processed image by performing additional image processing operations different from the first set of image processing operations.

[0073] According to another embodiment, the backend image signal processing pipeline is optionally selectively deactivated.

[0074] According to another embodiment, the computer vision processing circuit optionally includes: a sensor interface coupled to the one or more sensors; a front-end processing subsystem configured to receive images from the sensor interface; a statistical pipeline configured to receive images from the front-end processing subsystem; and a processing unit configured to coordinate the operations of the sensor interface, the front-end processing subsystem, and the statistical pipeline.

[0075] According to another embodiment, the electronic device optionally includes: a first client processor coupled to the computer vision processing circuit and configured to execute a first set of algorithms; and a second client processor optionally coupled to the computer vision processing circuit and optionally configured to execute a second set of algorithms different from the first set of algorithms.

[0076] According to one embodiment, a method of operating an electronic device is provided, the method including: outputting, by at least one client running on the electronic device, an image request; receiving, by a multi-client scheduler, the image request from the at least one client and feeding the image request into a queue; reordering, by the multi-client scheduler, at least some of the image requests in the queue; and fulfilling at least some of the image requests in the queue by capturing images by activating one or more image sensors in the electronic device.

[0077] According to another embodiment, the method optionally includes: running, by the at least one client, a first set of algorithms including a gaze tracking algorithm; and running, by an additional client running on the electronic device or on an additional electronic device separate from the electronic device, a second set of algorithms different from the first set of algorithms.

[0078] According to another embodiment, the method optionally includes: managing, by a first scheduling agent running on the at least one client, a first request output from the first set of algorithms; and managing, by a second scheduling agent running on the additional client, a second request output from the second set of algorithms.

[0079] According to another embodiment, reordering at least some of the image requests in the queue optionally includes reordering at least some of the image requests in the queue based on a deadline or timing requirement specified in the image request.

[0080] According to another embodiment, reordering at least some of the image requests in the queue optionally includes reordering at least some of the image requests in the queue based on a priority level associated with the image request.

[0081] According to another embodiment, reordering at least some of the image requests in the queue optionally includes reordering at least some of the image requests in the queue based on whether the image request is associated with a user-facing algorithm or a non-user-facing algorithm.

[0082] According to another embodiment, the method optionally includes: using the multi-client scheduler to determine whether at least two of the received image requests can be satisfied by a single image capture; and in response to determining that the at least two of the received image requests can optionally be satisfied by a single image capture, aggregating the at least two of the received image requests into a single image request.

[0083] According to another embodiment, the method optionally includes: using the multi-client scheduler to trigger an image capture and return a corresponding result to the at least one client; and using the multi-client scheduler to return the result to an additional client different from the at least one client without triggering another image capture.

[0084] According to one embodiment, a method of operating an electronic device having one or more sensors is provided, the method including: executing one or more clients on the electronic device; using a multi-client scheduler coupled to the one or more clients to receive an image request from a client among the one or more clients; and determining whether the image request can be satisfied by an existing image currently stored on an image server within the electronic device before triggering the one or more sensors to capture a new image.

[0085] According to another embodiment, the method optionally includes: in response to determining that the image request can be satisfied by an existing image currently stored on the image server, returning a pointer to the existing image to the client; and in response to determining that the image request cannot be satisfied by an existing image currently stored on the image server, triggering the one or more sensors to capture a new image.

[0086] According to another embodiment, the method optionally includes: increasing a reference count of the existing image in response to determining that the image request can be satisfied by an existing image currently stored on the image server.

[0087] According to another embodiment, the received image request optionally includes requirements specifying one or more of the following: a timing or deadline requirement, an image resolution, an exposure level, a camera type, and a number of consecutive frames to be captured.

[0088] According to another embodiment, determining whether the image request can be satisfied by an existing image currently stored on the image server optionally includes determining whether the existing image satisfies at least some of the requirements specified in the received image request.

[0089] The foregoing is illustrative only and various modifications may be made to the described embodiments. The foregoing embodiments may be implemented singly or in any combination.

Claims

1. An electronic device, comprising: One or more sensors configured to capture images; Computer vision processing circuitry configured to receive the captured images and having a plurality of subsystems configured to operate in a first power domain; And A backend image signal processing pipeline coupled to the computer vision processing circuitry and configured to operate in a second power domain different from the first power domain.

2. The electronic device according to claim 1, further comprising: One or more displays configured to receive content for display from the backend image signal processing pipeline.

3. The electronic device according to claim 1, wherein the one or more sensors include: One or more outward-facing cameras configured to capture images of the environment.

4. The electronic device according to claim 3, wherein the one or more sensors include: One or more inward-facing cameras configured to capture images of the eyes.

5. The electronic device according to claim 1, wherein: The computer vision processing circuitry is further configured to output a processed image according to a first image processing requirement; and The backend image signal processing pipeline is further configured to output a processed image according to a second image processing requirement different from the first image processing requirement.

6. The electronic device according to claim 5, wherein: The computer vision processing circuitry is configured to output a processed image having a first quality or using a first amount of power; and The backend image signal processing pipeline is configured to output a processed image having a second quality greater than the first quality or using a second amount of power greater than the first amount of power.

7. The electronic device according to claim 5, wherein: The computer vision processing circuitry is configured to output a processed image by performing a first set of image processing operations; and The backend image signal processing pipeline is configured to output a processed image by performing additional image processing operations different from the first set of image processing operations.

8. The electronic device according to claim 1, wherein the backend image signal processing pipeline is selectively deactivated.

9. The electronic device according to claim 1, wherein the computer vision processing circuitry includes: A sensor interface coupled to the one or more sensors; A front-end processing subsystem configured to receive images from the sensor interface; A statistical pipeline configured to receive images from the front-end processing subsystem; And A processing unit configured to coordinate the operations of the sensor interface, the front-end processing subsystem, and the statistical pipeline.

10. The electronic device according to claim 1, further comprising: A first client processor coupled to the computer vision processing circuitry and configured to execute a first set of algorithms; And A second client processor, the second client processor being coupled to the computer vision processing circuitry and configured to execute a second set of algorithms different from the first set of algorithms.

11. A method of operating an electronic device, comprising: Outputting an image request using at least one client running on the electronic device; Receiving the image request from the at least one client using a multi-client scheduler and feeding the image request into a queue; Reordering at least some of the image requests in the queue using the multi-client scheduler; And Fulfilling at least some of the image requests in the queue by directing one or more image sensors in the electronic device to capture an image.

12. The method of claim 11, further comprising: Running a first set of algorithms including a gaze tracking algorithm using the at least one client; Running a second set of algorithms different from the first set of algorithms using an additional client running on the electronic device or on an additional electronic device separate from the electronic device; Managing a first request output from the first set of algorithms using a first scheduling agent running on the at least one client; And Managing a second request output from the second set of algorithms using a second scheduling agent running on the additional client.

13. The method of claim 11, wherein reordering at least some of the image requests in the queue comprises reordering at least some of the image requests in the queue based on a deadline or timing requirement specified in the image request.

14. The method of claim 11, wherein reordering at least some of the image requests in the queue comprises reordering at least some of the image requests in the queue based on a priority level associated with the image request.

15. The method of claim 11, wherein reordering at least some of the image requests in the queue comprises reordering at least some of the image requests in the queue based on whether the image request is associated with a user-facing algorithm or a non-user-facing algorithm.

16. The method of claim 11, further comprising: Determining, using the multi-client scheduler, whether at least two of the received image requests can be satisfied by a single image capture; And In response to determining that the at least two of the received image requests can be satisfied by a single image capture, coalescing the at least two of the received image requests into a single image request.

17. The method of claim 11, further comprising: Triggering an image capture using the multi-client scheduler and returning a corresponding result to the at least one client; And Returning the result to an additional client different from the at least one client using the multi-client scheduler without triggering another image capture.

18. A method of operating an electronic device having one or more sensors, the method comprising: Executing one or more clients on the electronic device; Receiving an image request from a client among the one or more clients by using a multi-client scheduler coupled to the one or more clients; And Before triggering the one or more sensors to capture a new image, determining whether the image request can be satisfied by an existing image currently stored on an image server within the electronic device.

19. The method according to claim 18, further comprising: In response to determining that the image request can be satisfied by an existing image currently stored on the image server, returning a pointer to the existing image to the client; In response to determining that the image request cannot be satisfied by an existing image currently stored on the image server, triggering the one or more sensors to capture a new image; And In response to determining that the image request can be satisfied by an existing image currently stored on the image server, increasing a reference count of the existing image.

20. The method according to claim 18, wherein: The received image request includes a requirement specifying one or more of the following: a timing or deadline requirement, an image resolution, an exposure level, a camera type, and a number of consecutive frames to be captured; and Determining whether the image request can be satisfied by an existing image currently stored on the image server includes determining whether the existing image satisfies at least some of the requirements specified in the received image request.

Images