Camera dynamic voting to optimize fast sensor mode power

Abstract

Systems, methods, and computer readable media for camera dynamic voting to optimize fast sensor mode power are provided. In some examples, a computing device may obtain a plurality of votes associated with a plurality of components of a shared power supply based on performing dynamic votes. The computing device may determine a voting result based on the plurality of votes. The computing device may increase or decrease a clock rate and a voltage of the power supply based on the voting result to produce an updated clock rate and an updated voltage. The computing device may then apply the updated clock rate and the updated voltage to an image processor.

Description

Technical Field

[0001] The present application relates to image processing. In some examples, aspects of the present application relate to systems and techniques for dynamic camera voting to optimize fast sensor mode power. Background Art

[0002] The increasing versatility of digital camera products has allowed them to be integrated into a wide variety of devices and has expanded their use across diverse applications. For example, phones, drones, cars, computers, televisions, and many other devices today are often equipped with camera devices. Camera devices allow users to capture images and / or video (e.g., including frames of images) from any system equipped with a camera device. Images and / or video can be captured for recreational use, professional photography, surveillance, automation, and other applications. Furthermore, camera devices are increasingly equipped with specialized features for modifying images or creating artistic effects on images. For example, many camera devices are equipped with image processing capabilities for generating various effects on captured images.

[0003] For image processing, fast sensor readout (FSR) is a common sensor operating mode used by many original equipment manufacturers (OEMs) to improve image quality (IQ) because employing FSR can result in a reduction in rolling shutter-related artifacts in the image. Compared to normal readout mode, FSR mode allows for faster readout of image sensor data for an image frame while maintaining the same exposure time as normal readout mode. The faster the readout of image sensor data is performed, the lower the amount of rolling shutter-related artifacts that will be present in the rendered image. FSR mode can also allow for a reduction in required sensor power (for example, in some cases, FSR mode can allow for a sensor power saving of 160 to 260 milliwatts compared to normal readout mode). Therefore, employing FSR for image processing can be advantageous from both an IQ perspective and a sensor power perspective.

[0004] However, FSR mode can impact the power consumption of the chipset (e.g., which may include an image signal processor). For example, FSR mode may require the image signal processor to operate at a very high clock rate and voltage to complete image frame processing during the compressed readout time of the FSR. Furthermore, FSR mode may require memory (e.g., double data rate (DDR) memory) to run at a high clock rate to enable rapid output of data from the image sensor processor. Therefore, improved techniques for optimizing the power consumption of FSR mode may be beneficial. Summary of the Invention

[0005] The following presents a simplified summary of one or more aspects disclosed herein. Therefore, the following summary should neither be considered an exhaustive overview of all contemplated aspects nor be considered to identify key or critical elements related to all contemplated aspects or to delineate the scope associated with any particular aspect. Therefore, the sole purpose of the following summary is to present certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form prior to the detailed description presented below.

[0006] Systems and techniques for camera dynamic voting to optimize fast sensor mode power are described. According to at least one example, a method for processing image data is provided. The method includes: obtaining a plurality of votes associated with a plurality of components sharing a power supply based on performing dynamic voting; determining voting results based on the plurality of votes; increasing or decreasing a clock rate and a voltage of the power supply based on the voting results to generate an updated clock rate and an updated voltage; and applying the updated clock rate and the updated voltage to an image processor.

[0007] In another illustrative example, an apparatus for processing image data is provided. The apparatus includes at least one memory and at least one processor, the at least one processor being coupled to the at least one memory and configured to: obtain a plurality of votes associated with a plurality of components sharing a power supply based on performing dynamic voting; determine voting results based on the plurality of votes; increase or decrease a clock rate and a voltage of the power supply based on the voting results to generate an updated clock rate and an updated voltage; and apply the updated clock rate and the updated voltage to an image processor.

[0008] In another illustrative example, a non-transitory computer-readable medium is provided. The computer-readable medium includes instructions stored thereon that, when executed by at least one processor, cause the at least one processor to: obtain a plurality of votes associated with a plurality of components sharing a power supply based on performing dynamic voting; determine voting results based on the plurality of votes; increase or decrease a clock rate and a voltage of the power supply based on the voting results to generate an updated clock rate and an updated voltage; and apply the updated clock rate and the updated voltage to an image processor.

[0009] In another illustrative example, an apparatus for processing image data is provided. The apparatus includes: means for obtaining a plurality of votes associated with a plurality of components sharing a power supply based on performing dynamic voting; means for determining voting results based on the plurality of votes; means for increasing or decreasing a clock rate and a voltage of the power supply based on the voting results to generate an updated clock rate and an updated voltage; and means for applying the updated clock rate and the updated voltage to an image processor.

[0010] Aspects generally include methods, apparatus, systems, computer program products, non-transitory computer-readable media, user devices, user equipment, wireless communication devices, and / or processing systems as substantially described with reference to and as illustrated in the accompanying drawings and description.

[0011] In some aspects, each of the devices described above is, or may be part of, or include, a mobile device, a smart or connected device, a camera system, and / or an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device). In some examples, the device may include, or be part of, a vehicle, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotic device or system, an aviation system, or other equipment. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the devices described above may include one or more sensors. In some cases, the one or more sensors may be used to determine the device's location, the device's status (e.g., tracking status, operational status, temperature, humidity level, and / or other status), and / or for other purposes.

[0012] Some aspects include a device having a processor configured to perform one or more operations of any of the methods outlined above. Further aspects include a processing device for use in a device, the processing device configured with processor-executable instructions to perform the operations of any of the methods outlined above. Further aspects include a non-transitory processor-readable storage medium having stored thereon processor-executable instructions configured to cause the processor of the device to perform the operations of any of the methods outlined above. Further aspects include a device having components for performing the functions of any of the methods outlined above.

[0013] The features and technical advantages of the examples according to the present disclosure have been outlined quite broadly above so that the detailed description that follows may be better understood. Additional features and advantages will be described below. The concepts and specific examples disclosed may be readily utilized as a basis for modifying or designing other structures for achieving the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein (both their organization and method of operation) and the associated advantages will be better understood from the following description when considered in conjunction with the accompanying drawings. Each of the figures in the drawings is provided for the purpose of illustration and description and not as a definition of limitations of the claims. The foregoing and other features and aspects will become more apparent upon reference to the following description, claims, and accompanying drawings.

[0014] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. This subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all of the drawings, and each claim.

[0015] The foregoing and other features and embodiments will become more fully apparent upon reference to the following description, claims, and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Examples of various specific implementations are described in detail below with reference to the following drawings:

[0017] Figure 1 is a block diagram illustrating an example architecture of an image capture and processing system according to some examples.

[0018] Figure 2 is a block diagram illustrating examples of interactions between components of an image capture and processing system according to some examples.

[0019] Figure 3 is a block diagram of an example device that may be used for dynamic voting in a camera to optimize fast sensor mode power, according to some examples.

[0020] Figure 4 is a block diagram illustrating the operation of an image signal processor pipeline according to some examples.

[0021] Figure 5 is a diagram illustrating examples of camera sensor timing according to some examples.

[0022] Figure 6 is a graph illustrating example power consumption of an entire camera module according to some examples.

[0023] Figure 7are graphs each illustrating an example power consumption waveform captured at a Mobile Industry Processor Interface (MIPI) of a camera, according to some examples.

[0024] Figure 8 is a diagram illustrating an example of a system for cameras showing data streams according to some examples.

[0025] Figure 9 are graphs each illustrating an example of an activity timeline of an image frame according to some examples.

[0026] Figure 10 is a graph illustrating chipset power of a camera and example power consumption of an entire camera module according to some examples.

[0027] Figure 11 is a diagram illustrating an example of timing for a camera employing a dynamic voting mechanism to optimize fast sensor mode power according to some examples.

[0028] Figure 12 is a diagram illustrating an example of a system employing a dynamic voting mechanism to optimize fast sensor mode power according to some examples.

[0029] Figure 13 is a table illustrating examples of power savings when dynamic voting is employed to optimize fast sensor mode power, according to some examples.

[0030] Figure 14 is a flow chart illustrating another example of a process for camera dynamic voting to optimize fast sensor mode power according to some examples.

[0031] Figure 15 is a diagram illustrating an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0032] Provide certain aspects and embodiments of the present disclosure below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of each embodiment of the application. However, it will be apparent that each embodiment can be put into practice without these specific details. Each drawing and description are not intended to be restrictive.

[0033] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the subsequent description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0034] A camera is a device that uses an image sensor to receive light and capture image frames (such as still images or video frames). The terms "image," "image frame," and "frame" are used interchangeably herein. A camera may include a processor, such as an image signal processor (ISP), that receives one or more image frames and processes them. For example, raw image frames captured by a camera sensor may be processed by the ISP to generate a final image. The processing performed by the ISP may be performed using a plurality of filters or processing blocks applied to the captured image frames, such as denoising or noise filtering, edge enhancement, color balancing, contrast, intensity adjustment (such as darkening or brightening), tonal adjustment, and the like. Image processing blocks or modules may include lens / sensor noise correction, Bayer filters, demosaicing, color conversion, correction or enhancement / suppression of image properties, noise reduction filters, sharpening filters, and the like.

[0035] A camera can be configured with a variety of image capture and image processing operations and settings. Different settings produce images with different appearances. Some camera operations, such as automatic exposure control (AEC) and automatic white balance (AWB), are determined and applied before or during image capture. Additional camera operations, applied before, during, or after image capture, include operations involving zoom (e.g., zooming in or out), ISO, aperture size, f-stop, shutter speed, and gain. Other camera operations may configure post-processing of the image, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color.

[0036] As previously mentioned, FSR is a common sensor operating mode used by many OEMs to improve IQ for image processing, as employing FSR can result in a reduction in rolling shutter-related artifacts in the image. Compared to normal readout mode, FSR mode allows for faster readout of image sensor data for an image frame while maintaining the same exposure time as normal readout mode. The faster the readout of image sensor data is performed, the lower the amount of rolling shutter-related artifacts that will be present in the rendered image. FSR mode can also reduce required sensor power (for example, in some cases, FSR mode can save 160 to 260 milliwatts of sensor power compared to normal readout mode). Therefore, utilizing FSR mode for image processing can be beneficial from both an IQ perspective and a sensor power perspective.

[0037] However, FSR mode may impact the power consumption of a chipset, which may include an image signal processor (ISP). For example, FSR mode may require the ISP to operate at a very high clock rate and voltage to complete image frame processing during the compressed readout time of FSR mode. Furthermore, FSR mode may require memory (e.g., DDR memory) to operate at a high clock rate to enable rapid output of data from the image sensor processor. Therefore, improved techniques for optimizing the power consumption of FSR mode may be useful.

[0038] Therefore, this document describes systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for dynamic camera voting to optimize fast sensor mode power (e.g., FSR mode power). For example, in some examples, the systems and techniques can optimize the power overhead incurred by a camera chipset (such as a system-on-chip (SOC)) when operating a camera in FSR mode. With conventional static clocking mechanisms, the ISP and DDR memory have fixed clock rates to meet the instantaneous performance requirements of a use case, which can result in significant power overhead across the entire use case timeline.

[0039] In one or more aspects, the systems and techniques dynamically increase the ISP and DDR clock rates during a short sensor readout duration in a use case timeline, and reduce the ISP and DDR clock rates immediately after the short sensor readout duration is complete (e.g., such that the clock rates are lower during a large blanking interval in the use case timeline, where no sensor readout is being performed). When the clock rates are adjusted in this manner, the high power overhead requirement can be limited to only the sensor readout portion of the use case timeline (e.g., which is typically only 25% to 50% of the use case timeline).

[0040] Most camera use cases operate at a specified image frame rate that is configured by high-level application software (SW). The system and technique can employ a configurable hardware (HW) timer mechanism (e.g. Figure 12 The dynamic clock voting engine 1250) can trigger a dynamic increase in the clock rate immediately before a new image frame is received from the image sensor, and trigger a dynamic decrease in the clock rate and shared rail voltage immediately after image frame processing is completed.

[0041] Additional aspects of the systems and techniques are described with respect to the accompanying figures.

[0042] Figure 1is a block diagram illustrating the architecture of image capture and processing system 100. Image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., images of scene 110). Image capture and processing system 100 can capture individual images (or photographs) and / or can capture videos comprising multiple images (or video frames) in a particular sequence. Lens 115 of system 100 faces scene 110 and receives light from scene 110. Lens 115 bends the light toward image sensor 130. Light received by lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by image sensor 130.

[0043] The one or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from the image sensor 130 and / or based on information from the image processor 150. The one or more control mechanisms 120 may include a plurality of mechanisms and components; for example, the control mechanisms 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. The one or more control mechanisms 120 may also include additional control mechanisms beyond those illustrated, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0044] Focus control mechanism 125B of control mechanism 120 may obtain a focus setting. In some examples, focus control mechanism 125B stores the focus setting in a memory register. Based on the focus setting, focus control mechanism 125B may adjust the position of lens 115 relative to the position of image sensor 130. For example, based on the focus setting, focus control mechanism 125B may actuate a motor or servo to move lens 115 closer to or further away from image sensor 130, thereby adjusting the focus. In some cases, device 105A may include additional lenses, such as one or more microlenses on each photodiode of image sensor 130, each microlens bending light received from lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus setting may be determined using contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus setting may be determined using control mechanism 120, image sensor 130, and / or image processor 150. The focus setting may be referred to as an image capture setting and / or an image processing setting.

[0045] Exposure control mechanism 125A of control mechanism 120 may obtain an exposure setting. In some cases, exposure control mechanism 125A stores the exposure setting in a memory register. Based on the exposure setting, exposure control mechanism 125A may control the size of the aperture (e.g., aperture size or number of stops), the duration the aperture is open (e.g., exposure time or shutter speed), the sensitivity of image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by image sensor 130, or any combination thereof. The exposure setting may be referred to as an image capture setting and / or an image processing setting.

[0046] Zoom control mechanism 125C of control mechanism 120 may obtain a zoom setting. In some examples, zoom control mechanism 125C stores the zoom setting in a memory register. Based on the zoom setting, zoom control mechanism 125C may control the focal length of an assembly of lens elements (lens assembly) including lens 115 and one or more additional lenses. For example, zoom control mechanism 125C may control the focal length of the lens assembly by actuating one or more motors or servos to move one or more of the lenses relative to one another. The zoom setting may be referred to as an image capture setting and / or an image processing setting. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, this focusing lens may be lens 115) that first receives light from scene 110, where the light then passes through an afocal zoom system between the focusing lens (e.g., lens 115) and image sensor 130 before reaching image sensor 130. In some cases, an afocal zoom system may include two positive (e.g., converging, convex) lenses having equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, zoom control mechanism 125C moves one or more of the lenses in the afocal zoom system, such as the negative lens and one or both of the positive lenses.

[0047] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered by different color filters and, therefore, measure light that matches the color of the color filter covering the photodiode. For example, a Bayer color filter includes red, blue, and green filters, where each pixel of an image is generated based on red light data from at least one photodiode covered by the red filter, blue light data from at least one photodiode covered by the blue filter, and green light data from at least one photodiode covered by the green filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald") filters instead of or in addition to the red, blue, and / or green filters. Some image sensors may lack color filters entirely and, instead, use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity curves, thereby responding to different wavelengths of light. Monochrome image sensors may also lack color filters and, therefore, lack color depth.

[0048] In some cases, image sensor 130 may alternatively or additionally include an opaque mask and / or a reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which may be useful for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output by the photodiode and / or an analog-to-digital converter (ADC) for converting the analog signal output by the photodiode (and / or the analog signal amplified by the analog gain amplifier) into a digital signal. In some cases, certain components or functionality discussed with respect to one or more control mechanisms in control mechanism 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide semiconductor (CMOS), an N-type metal-oxide semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0049] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other types of processors 1510 discussed with respect to the computing system 1500. The host processor 152 may be a digital signal processor (DSP) and / or other types of processors. In some implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system on a chip or SoC) that includes the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G or LTE, 5G, etc.), memory, connectivity components (e.g., Bluetooth ™ , Global Positioning System (GPS), etc.), any combination thereof, and / or other components. The I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as an Inter-Integrated Circuit 2 (I2C) interface, an Inter-Integrated Circuit 3 (I3C) interface, a Serial Peripheral Interface (SPI) interface, a serial general-purpose input / output (GPIO) interface, a Mobile Industry Processor Interface (MIPI) (such as a MIPI CSI-2 physical (PHY) layer port or interface, an Advanced High-Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports. In one illustrative example, the host processor 152 may communicate with the image sensor 130 using an I2C port, and the ISP 154 may communicate with the image sensor 130 using a MIPI port.

[0050] The image processor 150 may perform a number of tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. The image processor 150 may store image frames and / or processed images in a random access memory (RAM) 140 / 1525, a read-only memory (ROM) 145 / 1520, a cache 1512, a memory unit 1515, another storage device 1530, or some combination thereof.

[0051] Various input / output (I / O) devices 160 may be connected to the image processor 150. I / O devices 160 may include a display screen, a keyboard, a keypad, a touch screen, a touchpad, a touch-sensitive surface, a printer, any other output device 1535, any other input device 1545, or some combination thereof. In some cases, subtitles may be entered into the image processing device 105B via a physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad of the touch screen of the I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors that enable wired connections between the device 105B and one or more peripheral devices, over which the device 105B can receive data from and / or send data to the one or more peripheral devices. I / O 160 may also include one or more wireless transceivers that enable wireless connections between the device 105B and one or more peripheral devices, over which the device 105B can receive data from and / or send data to the one or more peripheral devices. Peripheral devices may include any of the types of I / O devices 160 discussed previously, and may themselves be considered I / O devices 160 once they are coupled to a port, jack, wireless transceiver, or other wired and / or wireless connector.

[0052] In some cases, the image capture and processing system 100 can be a single device. In some cases, the image capture and processing system 100 can be two or more separate devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some implementations, the image capture device 105A and the image processing device 105B can be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some implementations, the image capture device 105A and the image processing device 105B can be disconnected from each other.

[0053] like Figure 1 As shown, the vertical dotted line will Figure 1 1. The image capture and processing system 100 is divided into two parts, representing image capture device 105A and image processing device 105B. Image capture device 105A includes lens 115, control mechanism 120, and image sensor 130. Image processing device 105B includes image processor 150 (including ISP 154 and host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, some components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0054] The image capture and processing system 100 may include an electronic device, such as a mobile or landline telephone handset (e.g., a smartphone, a cell phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the image capture and processing system 100 may include one or more wireless transceivers for wireless communication, such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof. In some implementations, the image capture device 105A and the image processing device 105B may be different devices. For example, the image capture device 105A may include a camera device, and the image processing device 105B may include a computing device, such as a mobile handset, a desktop computer, or other computing device.

[0055] Although the image capture and processing system 100 is shown as including certain components, one of ordinary skill will appreciate that the image capture and processing system 100 may include more than Figure 1 . The components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some implementations, the components of the image capture and processing system 100 may include and / or be implemented using electronic circuitry or other electronic hardware that may include one or more programmable electronic circuits (e.g., a microprocessor, GPU, DSP, CPU, and / or other suitable electronic circuitry) and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device that implements the image capture and processing system 100.

[0056] Host processor 152 may configure image sensor 130 with new parameter settings (e.g., via an external control interface such as I2C, I3C, SPI, GPIO, and / or other interfaces). In one illustrative example, host processor 152 may update the exposure settings used by image sensor 130 based on internal processing results of an exposure control algorithm from past image frames.

[0057] In some examples, host processor 152 may perform electronic image stabilization (EIS). For example, host processor 152 may determine motion vectors corresponding to motion compensation for one or more image frames. In some aspects, host processor 152 may position a cropped pixel array ("image window") within the total pixel array. The image window may include pixels used to capture an image. In some examples, the image window may include all pixels in the sensor except for a portion of rows and columns at the sensor's periphery. In some cases, when image capture device 105A is stationary, the image window may be centered on the sensor. In some aspects, the periphery pixels may surround the pixels of the image window, forming a set of buffered pixel rows and columns around the image window. Host processor 152 may implement EIS and shift the image window from frame to frame of the video so that it tracks the same scene across consecutive frames (e.g., assuming the subject is not moving). In some examples where the subject is moving, host processor 152 may determine that the scene has changed.

[0058] In some examples, the image window may include at least 95% (e.g., 95% to 99%) of the pixels on the sensor. A first region of interest (ROI) (e.g., for AF and / or AWB) may include image data within the field of view of at least 95% (e.g., 95% to 99%) of a plurality of imaging pixels in the image sensor 130 of the image capture device 105A. In some aspects, a plurality of buffer pixels at the sensor periphery (outside the image window) may be reserved as a buffer to allow the image window to be shifted to compensate for jitter. In some cases, the image window may be shifted so that the subject remains in the same position within the adjusted image window, even though light from the subject may be projected onto different areas of the sensor. In another example, the buffer pixels may include the ten topmost rows, ten bottommost rows, ten leftmost columns, and ten rightmost columns of pixels on the sensor. In some configurations, when the image capture device 105A is stationary, the buffer pixels are not used for AF, AE, or AWB and are not included in the image output. If the jitter causes the sensor to move left by twice the width of a column of pixels between frames, the EIS algorithm can be used to shift the image window to the right by two columns of pixels so the captured image shows the same scene in the next frame as in the current frame. The host processor 152 can use EIS to smooth the transition from one frame to the next.

[0059] In some aspects, host processor 152 can also dynamically configure parameter settings of ISP 154's internal pipelines or modules to match the settings of one or more input image frames from image sensor 130, so that the image data is correctly processed by ISP 154. The processing (or pipeline) blocks or modules of ISP 154 may include modules for lens / sensor noise correction, demosaicing, color conversion, correction or enhancement / suppression of image properties, denoising filters, sharpening filters, and the like. The settings of the various modules of ISP 154 can be configured by host processor 152. Each module can include a large number of tunable parameter settings. Furthermore, because different modules may affect similar aspects of an image, modules may be interdependent. For example, denoising and texture correction or enhancement may both affect high-frequency aspects of an image. Consequently, a large number of parameters are used by the ISP to generate a final image based on the captured raw image.

[0060] In some cases, the image capture and processing system 100 can automatically perform one or more of the image processing functionalities described above. For example, one or more of the control mechanisms 120 can be configured to perform autofocus operations, auto-exposure operations, and / or auto-white balance operations. In some embodiments, the autofocus functionality allows the image capture device 105A to automatically focus before capturing a desired image. Various autofocus technologies exist. For example, active autofocus techniques typically determine the distance between the camera and the subject of the image via the camera's distance sensor by emitting infrared laser or ultrasonic signals and receiving reflections of these signals. Furthermore, passive autofocus techniques use the camera's own image sensor to focus the camera and therefore do not require the integration of additional sensors into the camera. Passive AF techniques include contrast detection autofocus (CDAF), phase detection autofocus (PDAF), and, in some cases, hybrid systems utilizing both techniques. The image capture and processing system 100 can be equipped with these or any additional types of autofocus technologies.

[0061] Synchronization between image sensor 130 and ISP 154 is important in order to provide an operational image capture system that produces high quality images without interruptions and / or failures. Figure 2 is a block diagram illustrating an example of an image capture and processing system 200 that includes an image processor 250 (including a host processor 252 and an ISP 254 ) in communication with an image sensor 230 . Figure 2The illustrated configuration illustrates a conventional synchronization technique used in camera systems. Generally, the host processor 252 attempts to synchronize the image sensor 230 with the ISP 254 using fixed time periods by communicating with each separately. For example, in conventional camera systems, the host processor 252 communicates with the image sensor 230 (e.g., via an I2C port) and programs the image sensor 230 parameters at a first fixed time period (e.g., two frames before the image frame is processed by the ISP 254). The host processor 252 then communicates with the ISP 254 (e.g., via an internal AHB bus or other interface) and programs the ISP 254 parameter settings at a second fixed time period (e.g., one frame before the image frame is processed by the ISP 254).

[0062] Image sensor 230 may transmit image frames to ISP 254 (e.g., via a MIPI CSI-2 PHY port or interface or other suitable interface). Figure 2 (B to C in

[15] ). However, communication between host processor 252 and image sensor 230 (shown as A to B) is uncertain. Similarly, communication between image sensor 230 and ISP 254 (shown as B to C) and between host processor 252 and ISP 254 (shown as A to C) are also uncertain. For example, programming of image sensor 230 and ISP 254 by host processor 252 may experience varying latency, which may lead to mismatched parameter settings between the sensor and ISP. This latency may be due to high CPU usage, congestion in one or more I / O ports, and / or other factors.

[0063] Figure 3FIG3 is a block diagram of an example device 300 that can be used for dynamic camera voting to optimize fast sensor mode power. Device 300 may include or be coupled to a camera 302, and may also include a processor 306, a memory 308 storing instructions 310, a camera controller 312, a display 316, and multiple input / output (I / O) components 318 including one or more microphones (not shown). Example device 300 may be any suitable device capable of capturing and / or storing images or video, including, for example, wired and wireless communication devices (such as camera phones, smartphones, tablets, security systems, smart home devices, connected home devices, surveillance devices, Internet Protocol (IP) devices, dashcams, laptops, desktop computers, cars, drones, airplanes, etc.), digital cameras (including still cameras, video cameras, etc.), or any other suitable device. Device 300 may include additional features or components not shown. For example, it may include a wireless interface for wireless communication devices, which may include multiple transceivers and a baseband processor. Device 300 may also include or be coupled to additional cameras other than camera 302. The present disclosure should not be limited to any particular example or illustration, including the example device 300 .

[0064] The camera 302 may be capable of capturing individual image frames (such as still images) and / or capturing video (such as a continuous stream of captured image frames). The camera 302 may include one or more image sensors (not shown for simplicity) and a shutter for capturing image frames and providing the captured image frames to the camera controller 312. Although a single camera 302 is shown, any number of cameras or camera assemblies may be included and / or coupled to the device 300. For example, the number of cameras may be increased to achieve greater depth determination capabilities or better resolution for a given FOV.

[0065] The memory 308 may be a non-transitory or non-transitory computer-readable medium that stores computer-executable instructions 310 for performing all or part of one or more operations described in this disclosure. The device 300 may also include a power supply 320 that may be coupled to the device 300 or integrated into the device.

[0066] The processor 306 may be one or more suitable processors capable of executing scripts or instructions of one or more software programs, such as instructions 310, stored within the memory 308. In some aspects, the processor 306 may be one or more general-purpose processors that execute the instructions 310 to cause the device 300 to perform any number of functions or operations. In additional or alternative aspects, the processor 306 may include integrated circuits or other hardware for performing functions or operations without the use of software. Although in Figure 3306, the processor 306, the memory 308, the camera controller 312, the display 316, and the I / O components 318 may be coupled to each other in various arrangements. For example, the processor 306, the memory 308, the camera controller 312, the display 316, and / or the I / O components 318 may be coupled to each other via one or more local buses (not shown for simplicity).

[0067] Display 316 can be any suitable display or screen that allows user interaction and / or presents items (such as captured images and / or video) for viewing by the user. In some aspects, display 316 can be a touch-sensitive display. Display 316 can be part of device 300 or external to the device. Display 316 can include an LCD, LED, OLED, or similar display. I / O components 318 can be or include any suitable mechanism or interface for receiving input (such as commands) from a user and / or providing output to the user. For example, I / O components 318 can include (but are not limited to) a graphical user interface, a keyboard, a mouse, a microphone, and speakers.

[0068] The camera controller 312 may include an image signal processor (ISP) 314, which may be (or may include) one or more image signal processors for processing captured image frames or video provided by the camera 302. For example, the ISP 314 may be configured to perform various processing operations for autofocus (AF), auto white balance (AWB), and / or auto exposure (AE) (which may also be referred to as automatic exposure control (AEC)). Examples of image processing operations include, but are not limited to, cropping, scaling (e.g., scaling to a different resolution), image stitching, image format conversion, color interpolation, image interpolation, color processing, image filtering (e.g., spatial image filtering), and the like.

[0069] In some example implementations, a camera controller 312 (such as an ISP 314) can implement various functionalities, including imaging processing and / or controlling operations of the camera 302. In some aspects, the ISP 314 can execute instructions from a memory (such as instructions 310 stored in the memory 308 or instructions stored in a separate memory coupled to the ISP 314) to control image processing and / or operations of the camera 302. In other aspects, the ISP 314 can include specific hardware for controlling image processing and / or operations of the camera 302. The ISP 314 can alternatively or additionally include a combination of specific hardware and the ability to execute software instructions.

[0070] Although Figure 3Although not shown, in some implementations, the ISP 314 and / or the camera controller 312 may include an AF module, an AWB module, and / or an AE module. The ISP 314 and / or the camera controller 312 may be configured to perform the AF process, the AWB process, and / or the AE process. In some examples, the ISP 314 and / or the camera controller 312 may include hardware-specific circuitry (e.g., an application-specific integrated circuit (ASIC)) configured to perform the AF process, the AWB process, and / or the AE process. In other examples, the ISP 314 and / or the camera controller 312 may be configured to execute software and / or firmware to perform the AF process, the AWB process, and / or the AE process. When configured in software, the code for the AF process, the AWB process, and / or the AE process may be stored in a memory (such as instructions 310 stored in the memory 308 or instructions stored in a separate memory coupled to the ISP 314 and / or the camera controller 312). In other examples, the ISP 314 and / or the camera controller 312 may use a combination of hardware, firmware, and / or software to perform the AF process, the AWB process, and / or the AE process. When configured as software, the AF process, the AWB process, and / or the AE process may include instructions that configure the ISP 314 and / or the camera controller 312 to perform various image processing and device management tasks, including the techniques of the present disclosure.

[0071] Figure 4 4 is a block diagram illustrating the operation of an image signal processing pipeline 402 of an image signal processor (eg, ISP 314). For example, ISP 314 may be configured to execute image signal processing pipeline 402 to process input image data. ISP 314 may be configured to process input image data from Figure 3 The camera 302 and / or the image sensor (not shown) of the camera 302 receives input image data. In some examples, such as Figure 4 As shown, the input image data may include color data and / or any other data (e.g., depth data) of the image / frame. Figure 4 In an example, the color data received for the input image data may be in a Bayer format. Instead of capturing red (R), green (G), and blue (B) values for each pixel of an image, an image sensor (e.g., the image sensor of camera 302) may use a Bayer filter mosaic (or more generally, a color filter array (CFA)), where each photosensor of the digital image sensor captures a different color from the RGB color spectrum. An example filter pattern for a Bayer filter mosaic may include a 50% green filter, a 25% red filter, and a 25% blue filter.

[0072] The Bayer processing unit 410 may perform one or more initial processing techniques on the raw Bayer data received by the ISP 314 , including, for example, subtraction, slip correction, bad pixel correction, black level compensation, and / or denoising.

[0073] The statistics (stats) screening process 412 may determine Bayer levels or Bayer grid (BG) statistics for the received input image data. In some examples, the BG statistics may include a red-to-green ratio (R / G) (which may indicate the presence and amount of red tinting that may be present in the image) and / or a blue-to-green ratio (B / G) (which may indicate the presence and amount of blue tinting that may be present in the image). For example, the (R / G) of an image or a portion / region of an image may be described by the following formula (1):

[0074]

[0075] The image or a portion / region of the image includes pixels 1-N, and each pixel n includes a red value Red(n), a blue value Blue(n), or a green value Green(n) in the RGB space. (R / G) is the sum of the red values of the red pixels in the image divided by the sum of the green values of the green pixels in the image. Similarly, (B / G) of the image or a portion / region of the image can be described by the following formula (2):

[0076]

[0077] In some other example implementations, a different color space may be used, such as Y'UV, where the chromaticity values UV indicate color and / or other indications of tinting or other color temperature effects on an image may be determined.

[0078] The AWB module and / or process 404 may analyze information associated with the received image data to determine the illuminant of the scene from a plurality of possible illuminants, and may determine AWB gains to apply to the received image and / or subsequent images based on the determined illuminant. White balancing is a process that attempts to match the color of an image to a user's perceptual experience of the captured object. As an example, the white balance process may be designed so that white objects appear white in the processed image, and gray objects appear gray in the processed image.

[0079] The illuminant may include the lighting conditions of the scene being captured, the type of light, etc. In some examples, an image capture device (e.g., such as Figure 3A user of a device (e.g., device 300) can select or indicate the illuminant under which the image is to be captured. In other examples, the image capture device itself can automatically determine the most likely illuminant and perform white balancing based on the determined illuminant (e.g., lighting conditions). To better render the colors of the scene in the captured image or video, the AWB algorithm on the device and / or camera may attempt to determine the scene's illuminant and set / adjust the image or video's white balance accordingly.

[0080] During the AWB process 404, device 300 may determine or estimate the color temperature of the received frame (e.g., image). The color temperature may indicate the dominant hue of the image. The true color temperature of a scene captured in a video or image is the color of the scene's light source. If the light is emitted from a perfect blackbody radiator (theoretically ideal for all electromagnetic wavelengths) at a specific color temperature (expressed in Kelvin (K)), and the color temperature is known, then the color temperature of the scene is known. For example, in the color space defined by the International Commission on Illumination (CIE) (since 1931), the chromaticity of radiation from a blackbody radiator with a temperature from 1,000K to 20,000K is the Planckian locus. Colors on the Planckian locus from approximately 2,000K to 20,000K are considered white, with 2,000K being warm white or reddish-white and 20,000K being cool white or bluish-white. Many incandescent light sources include a Planckian radiator (a tungsten filament or another filament used to emit light) that emits warm white light with a color temperature of approximately 2,400K to 3,100K.

[0081] However, other light sources, such as fluorescent lamps, discharge lamps, or light-emitting diodes (LEDs), are not perfect blackbody radiators whose radiation falls along the Planckian locus. For example, LEDs or neon signs emit light through electroluminescence, and the color of the light does not follow the Planckian locus. The color temperature determined for such light sources can be the correlated color temperature (CCT). The CCT is an estimated color temperature of a light source whose color does not fall exactly on the Planckian locus. For example, the CCT of a light source is the closest blackbody color temperature to the radiation emitted by the light source. The CCT can also be expressed in Kelvin.

[0082] CCT can be an approximation of the true color temperature of a scene. For example, CCT can be a simplified colorimetric of chromaticity coordinates in the CIE 1931 color space. Many devices can use AWB to estimate CCT for color balancing.

[0083] CCT can be a temperature rating ranging from warm colors (such as yellows and reds below 3200K) to cool colors (such as blues above 4000K). CCT (or other color temperature) can indicate the coloration that will appear in images captured using such a light source. For example, a CCT of 2700K can indicate red coloration, and a CCT of 5000K can indicate blue coloration.

[0084] Different lighting sources or ambient lighting can illuminate a scene, and the color temperature may be unknown to the device. Therefore, the device may analyze data captured by an image sensor to estimate the color temperature of an image (e.g., a frame). For example, the color temperature may be an estimate of the overall CCT of the light sources in the scene in the image. The data captured by the image sensor for estimating the color temperature of the frame (e.g., an image) may be the captured image itself.

[0085] After device 300 determines the color temperature of the scene (such as during AWB), device 300 may use the color temperature to determine a color balance to correct any tinting in the image. For example, if the color temperature indicates that the image includes red tinting, device 300 may, for example, reduce the red value or increase the blue value for each pixel in the image in RGB space. The color balance may be a color correction (such as a value to reduce the red value or increase the blue value).

[0086] Example inputs to the AWB process 404 may include the Bayer level or Bayer grid (BG) statistics of the received image data determined via the statistics screening process 412, an exposure index (e.g., the brightness of the scene of the received image data), and auxiliary information, which may include contextual information of the scene based on the audio input (as will be discussed in further detail below), depth information, etc. It should be noted that the AWB process 404 may be included as a separate AWB module in Figure 3 within the camera controller 312 .

[0087] The AE process 406 may include procedures for configuring, calculating, and / or storing Figure 3 The AE process 406 may use the audio input and / or contextual information of the scene based on the audio input to more quickly determine and / or apply the exposure settings. It should be noted that the AE process 406 may be included as a separate AE module in the AE process. Figure 3 within the camera controller 312 .

[0088] AF process 408 may include procedures for configuring, calculating, and / or storing Figure 3 The AF process 408 may determine the autofocus settings (e.g., initial lens position, final lens position, etc.) based on the audio input and / or contextual information of the scene based on the audio input. It should be noted that the AF process 408 may be included as a separate AF module in the Figure 3 within the camera controller 312 .

[0089] The demosaicing unit 414 can be configured to convert the processed Bayer image data into RGB values for each pixel of the image. As explained above, the Bayer data may include values for only one color channel (R, G, or B) for each pixel of the image. The demosaicing unit 414 can determine the values of the other color channels of the pixel by interpolating the color channel values of nearby pixels. In some ISP pipelines 402, the demosaicing unit 414 may precede the AWB process 404, the AE process 406, and / or the AF process 408, or after the AWB process 404, the AE process 406, and / or the AF process 408.

[0090] Other processing units 416 may apply additional processing to the image after AWB process 404, AE process 406, and / or AF process 408 and / or demosaicing processing unit 414. The additional processing may include color, tonal, and / or spatial processing of the image.

[0091] As previously mentioned, FSR is a common sensor operating mode used by many OEMs to improve IQ for image processing, as employing FSR can result in a reduction in rolling shutter-related artifacts in the image. Compared to normal readout mode, FSR mode allows for faster readout of image sensor data for an image frame while maintaining the same exposure time as normal readout mode. The faster the readout of image sensor data is performed, the lower the number of rolling shutter-related artifacts that will be present in the rendered image.

[0092] Figure 5 is a diagram comparing the timing of the FSR mode with the timing of the normal readout mode. Specifically, Figure 5 is a diagram illustrating an example of a camera sensor timing sequence 500. Figure 5 , the FSR mode timing diagram 560 is compared with the normal readout mode timing diagram 550.

[0093] exist Figure 5 , a normal readout mode timing diagram 550 illustrates the amount of exposure time for a row (eg, time period 510 ) and the amount of time for a row to be read from one or more camera sensors (eg, Figure 12 sensor 1210a to sensor 1210b) to a camera SoC (which may include one or more ISPs (e.g., Figure 12 ISP 1230a to ISP 1230b)) for an amount of time (e.g., time period 520) to read out the row.

[0094] like Figure 5As shown, the rows are exposed one at a time (e.g., time period 510). In normal readout timing diagram 550, in the first row, the first row is exposed and then read out. After a delay (e.g., time period 510) after the exposure of the first row begins, the second row is exposed and then read out (e.g., time period 520). Readout of the second row can begin immediately after readout of the first row is completed.

[0095] For both the normal readout timing diagram 550 and the FSR mode timing diagram 560, the same amount of exposure time for all rows should be maintained throughout the image frame. This exposure time only occurs on one row at a time. It is desirable to maintain the same exposure time for each row, otherwise the dynamic range of the pixels between rows in the rendered image may not be consistent, which can cause image artifacts. For this reason, the start of the first exposure time for row N plus one (e.g., row N+1) should be delayed by some amount of time after the start of the first exposure time for the previous row (e.g., row N). The start of the first exposure time for row N plus one (e.g., row N+1) should be delayed because the readout of the previous row (e.g., row N) has not yet completed because readout can only occur serially on the bus. With only one MIPI (e.g., row N), the readout time can only occur serially on the bus. Figure 8 820) will camera sensor (eg, Figure 8 The sensor 810) is connected to the SoC (eg, including an ISP such as Figure 8 Due to the inline image processor 830 of the CMOS process, the readout of sensor data is completely serialized on the bus. Therefore, the readout of row N plus one (e.g., row N+1) cannot be performed until the readout of the previous row (e.g., row N) is completed.

[0096] Because readout of row N plus one (e.g., row N+1) cannot occur until readout of the previous row (e.g., row N) is complete, the start of each row's exposure time is offset (e.g., increment 580). The start of the second row's exposure time is delayed by the offset (e.g., increment 580) from the start of the first row's exposure time, and the start of the third row's exposure time is delayed by the offset from the start of the second row's exposure time, and so on. As the entire image frame undergoes the image processing process, as can be seen in normal readout timing diagram 550, there is a significant time lag (e.g., time period 570a) between the start of exposure for the first row in the image frame and the start of exposure for the last row in the image frame. The greater the magnitude of this time lag, the more rolling shutter artifacts will be present in the rendered image.

[0097] For FSR mode timing diagram 560, the time lag between the start of exposure of the first row in the image frame and the start of exposure of the last row in the image frame (e.g., time period 570b) is not as large as the time lag between the start of exposure of the first row in the image frame and the start of exposure of the last row in the image frame (e.g., time period 570a) of normal mode timing diagram 550. The smaller the time lag (e.g., time period 570b), the lower the amount of rolling shutter artifacts in the rendered image.

[0098] However, while FSR mode allows for a reduction in rolling shutter artifacts, there are costs associated with its implementation. Because FSR mode performs a rapid readout of sensor data, the MIPI signal and clock rates need to be increased to accommodate the fast readout. Currently, not all sensors support FSR mode. Typically, only expensive camera sensors support FSR mode. However, to reduce the number of rolling shutter artifacts in rendered images and provide improved IQ, most high-end devices are currently moving towards supporting FSR mode.

[0099] exist Figure 5 , when comparing the normal readout timing diagram 550 with the FSR mode timing diagram 560, it is apparent that each of the gaps (e.g., gaps 540b) in the aggregated readout energy 530b of the FSR mode timing diagram 560 is larger than each of the gaps (e.g., gaps 540a) in the aggregated readout energy 530a of the normal mode timing diagram 550. Therefore, the FSR mode may have a larger vertical blanking period (e.g., one of the vertical blanking periods is gap 540b) than the vertical blanking period of the normal mode (e.g., one of the vertical blanking periods is gap 540a).

[0100] As previously mentioned, the power consumption of FSR mode is lower than that of normal readout mode. FSR mode can result in a reduction in required sensor power (e.g., in some cases, FSR mode can allow for a sensor power savings of 160 to 260 milliwatts compared to normal readout mode). Therefore, utilizing FSR mode for image processing rather than normal readout mode can be beneficial from an IQ perspective as well as a sensor power perspective.

[0101] Figure 6 The power advantage of using FSR mode is shown. Specifically, Figure 6Graph 600 illustrates example power consumption for the entire camera module. On graph 600, the x-axis represents the readout rate per frame, and the y-axis represents total sensor power. Graph 600 shows typical power consumption curves for different readout rates per frame in various camera scenarios, including a single-camera scenario, a dual-camera scenario, and a triple-camera scenario. For the slowest readout rate use case (e.g., a 33 millisecond readout rate per frame), power consumption is shown as highest for all three camera scenarios. As the readout rate increases, for example, to a 16.6 millisecond readout rate and an 8.3 millisecond readout rate per frame, the power consumption curves in graph 600 show a sharp decrease in power consumption.

[0102] Figure 7 Graphs 710, 720, and 730 are included that illustrate example power consumption at the MIPI of a camera. Specifically, graphs 710, 720, and 730 each illustrate an example power consumption waveform captured at the MIPI of the camera. Graphs 710, 720, and 730 each show power consumption waveforms captured at the MIPI for different per-frame readout rates. For example, graph 710 shows a power consumption waveform captured at the MIPI for a per-frame readout rate of 33 milliseconds, graph 720 shows a power consumption waveform captured at the MIPI for a per-frame readout rate of 16.6 milliseconds, and graph 730 shows a power consumption waveform captured at the MIPI for a per-frame readout rate of 8.3 milliseconds.

[0103] Graphs 710, 720, 730 show that the power consumption of the FSR mode is significantly lower (eg, graph 730). Since the FSR mode allows for faster readout, the FSR mode is more efficient during this large vertical blanking period (eg, Figure 5 During the gap 540b of , the camera components not used for the readout process are turned off. Since these camera components are not used for the readout process during this large vertical blanking period (e.g., Figure 5 No power is used during the gap 540b), so the power consumption of the entire camera module can be reduced.

[0104] Typically, camera sensors do not support any dynamic clock and voltage mechanisms. At least by increasing the system's clock rate and completing the sensor data readout faster, there is no voltage loss on the sensor subsystem side, so it is desirable to operate in FSR mode.

[0105] However, as previously mentioned, FSR mode may impact the power consumption of the camera chipset (e.g., SOC), which may include an ISP. For example, FSR mode may require the ISP to operate at a very high clock rate and voltage to complete image frame processing during the compressed readout time of FSR mode. Additionally, FSR mode may require memory (e.g., DDR memory) to operate at a high clock rate to enable rapid output of data from the image sensor processor.

[0106] Figure 8 The data flow in the camera is shown. Specifically, Figure 8 is a diagram illustrating an example of a system 800 for a camera showing data flow. Figure 8 , system 800 is shown as including a sensor 810 (e.g., a camera sensor subsystem for obtaining image frames of a captured scene), an inline image processor 830, an offline image processor 850, and a video processor 860. The sensor 810, the inline image processor 830, the offline image processor 850, and the video processor 860 are all shown in communication with a DDR memory 840.

[0107] During operation of the system 800, the sensor 810 may stream pixels of sensor data to an inline image processor 830 (e.g., an image front-end camera component, which may be a component in a SOC) via the MIPI 820. After the inline image processor 830 receives the pixels from the sensor 810, the inline image processor 830 may process the pixels one row at a time (e.g., as Figure 5 ). After the inline image processor 830 has processed all rows of the image frame, the inline image processor 830 may transfer the processed sensor data to the DDR memory 840. The processing performed by the inline image processor 830 is referred to as inline processing because the inline image processor 830 processes pixels according to the operation of the sensor 810 (e.g., Figure 9 shows an example of inline processing).

[0108] The timing of the inline image processor 830 needs to be strictly maintained because the operating timeline of the inline image processor 830 needs to correspond to the operating timeline of (e.g., be inline with) the sensor 810. Therefore, if the sensor 810 readout occurs within 8.3 milliseconds, the operation of the inline image processor 830 also needs to be completed within 8.3 milliseconds to complete processing of all pixels it receives from the sensor 810 and thus be fully inline.

[0109] In one or more examples, the output of the inline image processor 830 may be scaled down. For example, if the camera includes a forty-eight (48) megapixel camera sensor, all 48 megapixels of sensor data may be streamed to the inline image processor 830. The offline image processor 850 will ultimately process all 48 megapixels, however the video resolution itself may be much smaller. For example, the resulting video may have a full high-definition (FHD) video resolution or an ultra-high-definition (UHD) video resolution. The output of the inline image processor 830 may be scaled down and then transferred to the DDR memory 840. The offline image processor 850 can operate at this scaled-down resolution and continue with the rest of the image processing.

[0110] Figure 9 Graphs 910, 920 each show an example of an activity timeline for an image frame. Figure 9 In graphs 910 and 920 , the x-axis represents time and the y-axis represents activity. Graph 910 shows ISP activity for three image frames. The ISP activity includes activity from the inline image processor 830 (e.g., block 940 of graph 910 ) and activity from the offline image processor 850 (e.g., activity block 930 of graph 910 ). Graph 920 shows sensor activity (e.g., block 950 ), such as that of sensor 810, for the same three image frames.

[0111] exist Figure 9 In Figure 9, the activity of inline image processor 830 in graph 910 is shown inline with the sensor activity in graph 920. An ISP front-end (e.g., inline image processor 830) operating inline with a camera sensor (e.g., sensor 810) needs to run at a very high clock rate to support FSR mode. Running the ISP front-end at a very high clock rate can require high power consumption, which can negatively impact the voltage of the shared rail power supply supporting other cores of the SOC besides the ISP (e.g., the camera components). Even though FSR mode reduces sensor power consumption, SOC power consumption can increase due to the high clock rate required for fast readout. Consequently, the SOC may experience a power penalty when the system operates in FSR mode.

[0112] The systems and techniques provide dynamic voting for cameras to optimize fast sensor mode power (e.g., FSR mode power). In one or more aspects, the systems and techniques provide a dynamic voting mechanism (e.g., Figure 12 A dynamic clock voting engine 1250 is provided, which can optimize the camera chipset (eg, SOC) power consumption overhead incurred during camera operation in FSR mode.

[0113] Figure 10 is a graph 1000 illustrating example power consumption of a camera's chipset power and a camera module. For graph 1000, the x-axis represents sensor readout speed and the y-axis represents power in milliwatts. Graph 1000 illustrates a camera sensor (e.g., Figure 8 The sensor power consumption curve 1030 of the sensor 810) when the dynamic voting mechanism is not adopted, the SOC of the camera (for example, which may include Figure 8 chipset power consumption curve 1010 of the inline graphics processor 830), and when a dynamic voting mechanism is adopted (e.g., Figure 12 A chipset power consumption curve 1020 of the camera's SOC when using the dynamic clock voting engine 1250).

[0114] like Figure 10 As shown in the sensor power consumption curve 1030 , sensor power consumption improves as the sensor readout speed increases (e.g., for operating in FSR mode). In contrast, the chipset power consumption curve 1010 shows that chipset power consumption increases as the sensor readout speed increases (e.g., for operating in FSR mode). However, when compared to the chipset power consumption curve 1010 , the chipset power consumption curve 1020 shows that when the dynamic voting mechanism is used for fast sensor readout (e.g., when operating in FSR mode), chipset power consumption decreases, and therefore, the power consumption penalty for the chipset operating at faster readout speeds is reduced.

[0115] In one or more aspects, to reduce power consumption penalties for a chipset, the systems and techniques dynamically increase the ISP and DDR clock rates before a short sensor readout duration of a use case timeline and immediately reduce the ISP and DDR clock rates after the short sensor readout duration is complete (e.g., such that a large blanking interval in the use case timeline (such as Figure 5 The clock rate is lower during the gap 540b) in which no sensor readout is performed. When the clock rate is adjusted in this way, the large power overhead requirement can be limited to the sensor readout portion of the use case timeline (e.g., which is typically only 25% to 50% of the use case timeline).

[0116] Most camera use cases operate at a specified image frame rate that is configured by high-level application software. The system and technique can employ a configurable hardware timer mechanism (e.g. Figure 12 The dynamic clock voting engine 1250) is configured to trigger a dynamic increase in the clock rate and shared rail voltage immediately before a new image frame is received from the image sensor, and to trigger a dynamic decrease in the clock rate and shared rail voltage immediately after image frame processing is completed.

[0117] Figure 11 An example of a timing diagram for dynamically voting to increase and decrease ISP and DDR clock rates and voltages is shown. Specifically, Figure 11 is a diagram illustrating an example of a timing sequence 1100 for a camera employing a dynamic voting mechanism to optimize fast sensor mode power. Figure 11 , fast sensor readout timing 1110 and inline ISP processing timing 1120 are shown. Figure 11 , using the same camera sensor (e.g. Figure 12 A dynamic voting mechanism (eg, dynamic clock voting engine 1250) operates inline with sensors 1210a to 1210b. The dynamic voting mechanism can increase or decrease ISP and DDR clock rates and associated rail voltage levels.

[0118] Figure 11Timing 1100 shows the activity of FSR mode, which typically operates at a readout rate of 33 milliseconds per frame. A readout period 1140 (e.g., approximately eight milliseconds) for reading out sensor data is shown. Also shown is a large vertical blanking period 1130 (e.g., approximately 24 milliseconds) that occurs when the camera sensor is not streaming any pixels to the inline image processor.

[0119] When a readout cycle 1140 occurs, the dynamic voting mechanism may vote to increase the clock rate and increase the power consumption of the camera front end (e.g., including an inline image processor such as Figure 8 Then, when the vertical blanking period 1130 occurs, the dynamic voting mechanism can vote to reduce the clock rate and reduce the voltage of the shared rail.

[0120] The dynamic voting mechanism can be implemented in software, hardware, or a combination of software and hardware. In some cases, the dynamic voting mechanism can be implemented as a hardware module without any application processor involvement. Therefore, any latency in voting to increase or decrease clock rate and power is negligible. Since latency is minimized, the use case performance requirements can be met. In other cases, the application processor can participate in the dynamic voting mechanism so that the latency of the timing shown in 1120 is met to save time. Figure 10 The power shown in 1020.

[0121] Timer mechanisms (e.g. Figure 12One or more timers 1260 can be built into the dynamic voting mechanism. Typically, camera sensors operate at a specific frame rate, and the SOC does not know when new pixels will arrive from the camera sensor. Even before new pixels arrive from the camera sensor, the SOC (e.g., the inline image processor) must be prepared. To prepare the SOC for the new set of pixels from the camera sensor, the timer mechanism can start a timer (e.g., voting timer 1150). This timer can be configured at the start of a frame but set to expire slightly less than the camera sensor's frame time. Configuring the timer expiration time slightly less than the frame period allows the timer to start before the start of a new frame readout, allowing the SOC to prepare for the higher clock and voltage required for inline pixel processing. Once the SOC receives the first set of pixels of a new frame (at point 1), the voting timer is set for the next frame. After inline processing is complete, the dynamic voting mechanism can quickly vote to reduce the clock rate and voltage without losing any time (at point 2). The voting timer fires at point 4 (before the next frame arrives from the sensor), allowing the SOC to vote to increase the clock rate before the frame. The SOC power can be optimized by reducing the shared rail power between points 3 and 4. In this way, the dynamic voting mechanism (e.g., by voting on increases and decreases in clock rate and voltage) can allow the power loss of the chipset (e.g., SOC) to be reduced.

[0122] Figure 12 is a diagram illustrating an example of a system 1200 employing a dynamic voting mechanism (eg, a dynamic clock voting engine 1250) to optimize fast sensor mode power. Figure 12 In FIG. 1 , system 1200 is shown as including one or more sensors 1210 a - 1210 b (e.g., one or more sensors such as cameras or image sensors), ISPs 1230 a - 1230 b (e.g., one or more ISPs), memory 1240, a dynamic clock voting engine 1250, one or more timers 1260 (in some cases, multiple timers), a system resource voting and aggregation engine 1270, and a clock and voltage control unit 1290 (e.g., core / DDR clock control and power rail voltage control). Dynamic clock voting engine 1250, system resource voting and aggregation engine 1270, and / or clock and voltage control unit 1290 can be implemented in software, hardware, or a combination of software and hardware. Furthermore, additional sensors in system 1200 are optional, as indicated by the dashed outline of the box for sensor 1210 b. For example, in some aspects, system 1200 may include a single sensor 1210 a (in which case, the additional sensors, including sensor 1210 b, are not included in or used by system 1200). In other aspects, the system 1200 may include a plurality of sensors, including sensors 1210a through 1210b (where Figure 12 In the "Sensor (N)" section, "N" is an integer greater than or equal to 2). Furthermore, in some aspects, the system 1200 may include a single ISP 1230a (in which case, additional ISPs including ISP 1230b are not included in or used by the system 1200). In other aspects, the system 1200 may include multiple ISPs, including ISPs 1230a through ISP 1230b (where Figure 12 In "ISP(N)", "N" is an integer greater than or equal to 2).

[0123] In one or more examples, sensors 1210a-1210b may be RGB camera sensors. In some examples, each sensor 1210a-1210b is associated with a corresponding camera (such as an RGB camera). ISPs 1230a-1230b may each include an inline image processor (e.g., Figure 8 Each ISP 1230a-1230b may be associated with a corresponding sensor 1210a-1210b. The memory 1240 may be a DDR memory (e.g., Figure 8 DDR memory 840). Each timer in timer 1260 can support a corresponding camera, and each timer is configured to trigger a vote (increment) before a new image frame is streamed from sensor 1210a to sensor 1210b. In some cases, multiple timers 1260 are included in system 1200 to support multiple cameras.

[0124] exist Figure 12 During operation of system 1200, sensors 1210a-1210b may acquire image frames by capturing a scene. When sensors 1210a-1210b begin streaming pixels of an image frame to their associated ISPs 1230a-1230b, timer 1260 may initiate a timer (e.g., voting timer 1150), which may run at a time period slightly lower than the frame time of the camera sensor, to trigger dynamic clock voting engine 1250 to vote to increase the clock rate and rail voltage before the frame. After dynamic clock voting engine 1250 is triggered to vote to increase, dynamic clock voting engine 1250 may submit (e.g., send) a positive vote to system resource voting and aggregation engine 1270.

[0125] System resource voting and aggregation engine 1270 may receive multiple votes 1280 from multiple components of the camera, which may include dynamic clock voting engine 1250. The multiple components of the camera may be within a camera SOC and may be powered by the same rail (or rails) or different rails. Multiple votes 1280 may include an affirmative vote from dynamic clock voting engine 1250.

[0126] System resource voting and aggregation engine 1270 (which can support multiple clients) receives multiple votes 1280 and maintains the current voting state of each of its clients, as well as logic for voting to increase and decrease client clocks and shared resources (e.g., DDR and power rail voltages). When system resource voting and aggregation engine 1270 determines that there is at least one client that requires a higher operating clock and voltage for the shared rails, system resource voting and aggregation engine 1270 can transmit control signals to clock and voltage control unit 1290 to increase the clock rate and / or increase the voltage on one or more shared rails.

[0127] After the clock and voltage control unit 1290 receives the control signal to increase the clock rate and increase the voltage, the clock and voltage control unit 1290 may first increase the voltage of the main line and then increase the clock rate (e.g., to a higher MHz frequency). After the clock and voltage control unit 1290 has increased the clock rate (e.g., to a higher MHz frequency), the increased clock rate may be applied to the ISPs 1230 a and 1230 b and the memory 1240.

[0128] After the clock rates of the ISPs 1230a-1230b (e.g., one or more ISPs) and the memory 1240 have been increased, the sensors 1210a-1210b (e.g., one or more sensors) may stream pixels of image frames to their respective ISPs 1230a-1230b via the ISP sensor interface 1220. After receiving the pixels of the image frames, the ISPs 1230a-1230b may process the pixels (e.g., perform inline image processing on the pixels). After the ISPs 1230a-1230b have processed the pixels of the image frames, the ISPs 1230a-1230b may transmit (e.g., send) the processed sensor data to the memory 1240. Additionally, after ISPs 1230a through ISP 1230b have processed pixels of an image frame, the dynamic clock voting engine 1250 may submit (eg, send) a negative vote (eg, to reduce the clock rate and rail voltage) to the system resource voting and aggregation engine 1270 .

[0129] The system resource voting and aggregation engine 1270 may receive another plurality of votes 1280 from a plurality of components of the camera, which may include the dynamic clock voting engine 1250. The plurality of votes 1280 may include a negative vote from the dynamic clock voting engine 1250.

[0130] After the system resource voting and aggregation engine 1270 receives multiple votes 1280, if it determines that no more clients require a higher clock rate and voltage for the shared mains, it can send a control signal to the clock and voltage control unit 1290 to reduce the clock rate and reduce the voltage on the shared mains.

[0131] After the clock and voltage control unit 1290 receives the control signal to reduce the clock rate and reduce the voltage, the clock and voltage control unit 1290 may first reduce the clock rate (e.g., to a higher MHz frequency) and then reduce the voltage on one or more rails. After the clock and voltage control unit 1290 has reduced the clock rate (e.g., to a lower MHz frequency), the reduced clock rate may be applied to the ISPs 1230 a and 1230 b and the memory 1240.

[0132] After a period of time (e.g., vertical blanking period), the system 1200 will Figure 11 The sequence 1100 shown repeats the operations previously described.

[0133] Figure 13 is a table 1300 illustrating an example of power savings when dynamic voting is employed to optimize fast sensor mode power. Figure 13 , table 1300 is shown as including four columns, including a component list 1310 (e.g., for a UHD30 camera), an operational parameter list for a non-FSR mode use case 1320 (e.g., normal readout mode), an operational parameter list for an FSR mode use case without dynamic voting 1330, and a parameter list for an FSR mode use case with dynamic voting 1340. Figure 13 As shown in Table 1300, the FSR mode use case 1340 with dynamic voting has a 12% power consumption saving compared to the non-FSR mode use case 1320. The FSR mode use case 1330 without dynamic voting has only a 5.8% power consumption saving compared to the non-FSR mode use case 1320. Therefore, employing dynamic voting with FSR mode can significantly reduce the power loss of the chipset operating in FSR mode.

[0134] Figure 14is a flow chart illustrating an example of a process 1400 for camera dynamic voting to optimize fast sensor mode power. The process 1400 may be performed by a computing device or system, or by a component or system (e.g., a chipset) of the computing device or system. In some aspects, the process 1400 may be performed by Figure 12 The operations of process 1400 may be performed by system 1200 or by a computing device (e.g., a mobile device, a camera device, an extended reality (XR) device, a laptop or desktop computer, a vehicle or a computing device of a vehicle, etc.) that includes system 1200. The operations of process 1400 may be implemented as a processor (e.g., Figure 8 One or more inline image processors 830 and / or offline image processors 850, Figure 12 One or more ISPs 1230a to ISP 1230b or other components, Figure 15 Software components executed and run on the processor 1510 and / or other processors).

[0135] At block 1410, a computing device or system (or a component thereof) may obtain a plurality of votes associated with a plurality of components that share a power source based on performing dynamic voting. Dynamic voting may be performed as described herein, such as with respect to Figure 12 For example, in some aspects, to obtain multiple votes based on performing dynamic voting, a computing device or system (or a component thereof) may perform dynamic voting to generate a vote for at least one current image frame (e.g., using dynamic clock voting engine 1250). The computing device or system (or a component thereof) may obtain the votes generated for at least one previous image frame and may aggregate the votes with the votes (e.g., using system resource voting and aggregation engine 1270) to obtain multiple votes. In some cases, the computing device or system (or a component thereof) may trigger dynamic voting (e.g., based on a timer, such as timer 1260).

[0136] At block 1420, the computing device or system (or components thereof) may determine a voting result based on the plurality of votes (e.g., using dynamic clock voting engine 1250 and / or system resource voting and aggregation engine 1270). In some cases, the power source shared by the plurality of components is or includes one or more power rails.

[0137] At block 1430 , the computing device or system (or components thereof) may increase or decrease (eg, using the clock and voltage control unit 1290 ) the clock rate and voltage of the power supply based on the voting results to produce an updated clock rate and an updated voltage.

[0138] At block 1440, the computing device or system (or a component thereof) may apply the updated clock rate and the updated voltage to an image processor (e.g., ISP 1230a or multiple ISPs, such as ISPs 1230a through 1230b). In some cases, the computing device or system (or a component thereof) may include an image processor (e.g., ISP 1230a or multiple ISPs, such as ISPs 1230a through 1230b). In some aspects, the image processor is a front-end component of a camera. In some cases, the image processor is an inline image processor. In some examples, the image processor is an image signal processor (ISP). In some cases, the image processor and the multiple components are on a system on a chip (SOC).

[0139] In some cases, a computing device or system (or a component thereof) may obtain an image frame of a captured scene via a sensor (e.g., sensor 1210a or from multiple sensors, such as sensors 1210a-1210b). In some cases, a computing device or system (or a component thereof) may include a sensor (e.g., sensor 1210a or multiple sensors, such as sensors 1210a-1210b). In some aspects, an image processor operates inline with the sensor in a temporal manner, as described herein. In some cases, a computing device or system (or a component thereof) may output pixels of the image frame to the image processor (e.g., sensor 1210a may output pixels of the image frame to a respective ISP 1230a via ISP sensor interface 1220, sensors 1210a-1210b may stream pixels of the image frame to their respective ISPs 1230a-1230b via ISP sensor interface 1220, etc.). The computing device or system (or components thereof) may process (e.g., using an image processor such as ISP 1230a) the image frames to generate processed image data. In some cases, the computing device or system (or components thereof) may process multiple image frames from sensors 1210a-1210b using ISPs 1230a-1230b. In some cases, the computing device or system (or components thereof) may send, output, or otherwise provide the processed image data to a memory (e.g., double data rate (DDR) memory or other type of memory).

[0140] As described above, process 1400 may be performed by one or more computing devices or apparatuses. In some illustrative examples, process 1400 may be performed by Figure 1 Image capture and processing system 100, Figure 3 Equipment 300, Figure 12 system 1200 and / or one or more computing devices or systems (e.g., Figure 15The processing steps of process 1400 may be performed by a computing system 1500 (e.g., a computer system 1500). In some cases, such a computing device or apparatus may include a processor, microprocessor, microcomputer, or other component of a device configured to perform the steps of process 1400. In some examples, such a computing device or apparatus may include one or more sensors configured to capture image data. For example, the computing device may include a smartphone, a head-mounted display, a mobile device, a camera, a tablet computer, or other suitable device. In some examples, such a computing device or apparatus may include a camera configured to capture one or more images or videos. In some cases, such a computing device may include a display for displaying images. In some examples, one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such a computing device may further include a network interface configured to communicate the data.

[0141] Components of a computing device can be implemented in circuitry. For example, a component may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein. A computing device may also include a display (as an example of an output device or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive data based on the Internet Protocol (IP) or other types of data.

[0142] Process 1400 is illustrated as a logical flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as limiting, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0143] Additionally, process 1400 may be executed under the control of one or more computer systems configured with executable instructions and may be implemented in hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0144] Figure 15 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Figure 15 An example of a computing system 1500 is illustrated, which can be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1505. Connection 1505 can be a physical connection using a bus, or a direct connection to processor 1510, such as in a chipset architecture. Connection 1505 can also be a virtual connection, a networked connection, or a logical connection.

[0145] In some embodiments, computing system 1500 is a distributed system, in which the functionality described in this disclosure can be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components that each perform some or all of the functionality for which the component is described. In some embodiments, a component can be a physical device or a virtual device.

[0146] Example system 1500 includes at least one processing unit (CPU or processor) 1510 and connections 1505 that couple various system components, including memory units 1515, such as read-only memory (ROM) 1520 and random access memory (RAM) 1525, to processor 1510. Computing system 1500 may include a cache 1512 of high-speed memory directly connected to, in close proximity to, or integrated as part of processor 1510.

[0147] Processor 1510 may include any general-purpose processor and hardware or software services, such as services 1532, 1534, and 1536 stored in storage device 1530, configured to control processor 1510 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1510 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0148] To enable user interaction, the computing system 1500 includes an input device 1545 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1500 may also include an output device 1535 that can be one or more of a plurality of output mechanisms. In some instances, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 1500. The computing system 1500 may include a communication interface 1540, which generally governs and manages user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using wired and / or wireless transceivers, including utilizing an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple ® Lightning ® Ports / plugs, Ethernet port / plug, Optical port / plug, Dedicated wired port / plug, BLUETOOTH ® Wireless signal transmission, Bluetooth ® Low energy (BLE) wireless signal transmission, IBEACON ® The communication interface 1540 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1500 based on one or more signals received from one or more satellites associated with the one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction on operating on any particular hardware arrangement, and thus the base features herein may be readily substituted for improved hardware or firmware arrangements as they are developed.

[0149] The storage device 1530 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data that can be accessed by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a magnetic cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick ® card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin-transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0150] Storage devices 1530 may include software services, servers, services, etc. that, when code defining such software is executed by processor 1510, cause the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1510, connection 1505, output devices 1535, etc.

[0151] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media that can store data and does not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or storage devices. A computer-readable medium may have stored thereon code and / or machine-executable instructions, which may represent a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means, including memory sharing, message passing, token passing, network transmission, and the like.

[0152] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0153] Specific details are provided in the description above to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that embodiments can be put into practice without these specific details. For clarity of explanation, in some cases, the present technology can be presented as comprising separate functional blocks, including functional blocks comprising devices, device components, steps in the method embodied in software or a combination of hardware and software or routines. Additional components other than those components shown in the accompanying drawings and / or described herein can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form to avoid these embodiments becoming difficult to understand in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques can be shown in order to avoid making each embodiment difficult to understand without necessary details.

[0154] Individual embodiments may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or concurrently. Furthermore, the order of the operations may be rearranged. A process is terminated when its operations are completed, but a process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, termination of the process may correspond to the function returning to the calling function or main function.

[0155] The processes and methods according to the examples described above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtained from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0156] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functionality may also be implemented on circuit boards in different chips or different processes executed on a single device.

[0157] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0158] In the foregoing description, various aspects of the present application have been described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. The various features and aspects of the application described above can be used individually or in combination. In addition, without departing from the broader essence and scope of this specification, the embodiments can be used in any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For illustrative purposes, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0159] It should be understood by those skilled in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to (" ") and greater than or equal to (" ) symbol instead.

[0160] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0161] The phrase “coupled to” refers to any component being directly or indirectly physically connected to another component, and / or any component being in direct or indirect communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0162] Claim language or other language that recites "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0163] The various exemplary logic blocks, modules, circuits, and algorithmic steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. Technicians can implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0164] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium containing program code, including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0165] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Therefore, the term "processor," as used herein, may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured for encoding and decoding, or incorporated in a combined system or component (e.g., a system on a chip).

[0166] Illustrative aspects of the present disclosure include:

[0167] Aspect 1. A method for processing image data, the method comprising: obtaining multiple votes associated with multiple components sharing a power supply based on performing dynamic voting; determining voting results based on the multiple votes; increasing or decreasing the clock rate and voltage of the power supply based on the voting results to generate an updated clock rate and an updated voltage; and applying the updated clock rate and the updated voltage to an image processor.

[0168] Aspect 2. A method according to Aspect 1, wherein obtaining the multiple votes based on performing dynamic voting includes: performing the dynamic voting to generate votes for at least one current image frame; obtaining votes generated for at least one previous image frame; and aggregating the votes for at least one current image frame and the votes generated for at least one previous image frame to obtain the multiple votes.

[0169] Aspect 3. The method according to any one of Aspects 1 or 2 further comprises triggering the dynamic voting.

[0170] Aspect 4. The method according to aspect 3, wherein the triggering of the dynamic voting is based on a timer.

[0171] Aspect 5. The method according to any one of aspects 1 to 4, further comprising obtaining an image frame of the captured scene by a sensor.

[0172] Clause 6. The method according to clause 5, wherein the image processor and the sensor operate inline in terms of time sequence.

[0173] Aspect 7. The method according to any one of aspects 5 or 6, further comprising outputting, by the sensor, the pixels of the image frame to the image processor.

[0174] Aspect 8. The method according to any one of aspects 5 to 7, further comprising processing the image frame by the image processor to generate processed image data.

[0175] Aspect 9. The method according to aspect 8, further comprising sending the processed image data to a memory.

[0176] Aspect 10. The method of aspect 9, wherein the memory is a double data rate (DDR) memory.

[0177] Aspect 11. The method according to any one of aspects 1 to 10, wherein the image processor is a front-end component of a camera.

[0178] Aspect 12. The method according to any one of aspects 1 to 11, wherein the image processor is an inline image processor.

[0179] Clause 13. The method according to any one of clauses 1 to 12, wherein the image processor is an image signal processor (ISP).

[0180] Clause 14. The method of any one of clauses 1 to 13, wherein the image processor and the plurality of components are on a system on a chip (SOC).

[0181] Aspect 15. The method according to any one of aspects 1 to 14, wherein the power source shared by the plurality of components is one or more power rails.

[0182] Aspect 16. A device for processing image data, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory and configured to: obtain multiple votes associated with multiple components sharing a power supply based on performing dynamic voting; determine voting results based on the multiple votes; increase or decrease the clock rate and voltage of the power supply based on the voting results to generate an updated clock rate and an updated voltage; and apply the updated clock rate and the updated voltage to an image processor.

[0183] Aspect 17. An apparatus according to Aspect 16, wherein, in order to obtain the multiple votes based on performing the dynamic voting, the at least one processor is configured to: perform the dynamic voting to generate votes for at least one current image frame; obtain votes generated for at least one previous image frame; and aggregate the votes for at least one current image frame and the votes generated for at least one previous image frame to obtain the multiple votes.

[0184] Aspect 18. An apparatus according to any one of aspects 16 or 17, wherein the at least one processor is configured to trigger the dynamic voting.

[0185] Aspect 19. The apparatus according to any one of aspects 16 to 18, wherein the at least one processor is configured to trigger the dynamic voting based on a timer.

[0186] Aspect 20. The apparatus according to any one of aspects 16 to 19, further comprising: a sensor configured to capture an image frame of a scene.

[0187] Clause 21. The apparatus of clause 20, wherein the sensor is configured to output pixels of the image frame to the image processor.

[0188] Aspect 22. The apparatus according to any one of aspects 20 or 21, further comprising the image processor, wherein the image processor operates inline with the sensor in terms of time sequence.

[0189] Aspect 23. An apparatus according to any one of aspects 20 to 22, wherein the image processor is configured to process the image frame to generate processed image data.

[0190] Clause 24. The apparatus according to clause 23, wherein the at least one processor is configured to send the processed image data to the at least one memory.

[0191] Aspect 25. The apparatus of aspect 24, wherein the at least one memory is a double data rate (DDR) memory.

[0192] Clause 26. The apparatus of any one of clauses 16 to 25, wherein the image processor is a front-end component of a camera.

[0193] Clause 27. The apparatus of any one of clauses 16 to 26, wherein the image processor is an inline image processor.

[0194] Clause 28. The apparatus of any one of clauses 16 to 27, wherein the image processor is an image signal processor (ISP).

[0195] Clause 29. The apparatus of any one of clauses 16 to 28, wherein the image processor and the plurality of components are on a system on a chip (SOC).

[0196] Aspect 30. The apparatus of any one of aspects 16 to 29, wherein the power source shared by the plurality of components is one or more power rails.

[0197] Aspect 31. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by at least one processor, causing the at least one processor to perform the operations according to any one of aspects 1 to 15.

[0198] Aspect 32. An apparatus for processing image data, the apparatus comprising one or more components for performing the operations according to any one of aspects 1 to 15.

[0199] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Accordingly, the claims are not intended to be limited to the aspects shown herein, but are to be accorded the full scope consistent with the language of the claims, wherein reference to an element in the singular is not intended to mean "one and only one" unless specifically stated otherwise, but rather "one or more."

Claims

1. A method for processing image data, the method comprising: obtaining a plurality of votes associated with a plurality of components sharing a power source based on performing dynamic voting; determining a voting result based on the plurality of votes; increasing or decreasing a clock rate and a voltage of the power supply based on the voting result to generate an updated clock rate and an updated voltage; as well as The updated clock rate and the updated voltage are applied to an image processor.

2. The method of claim 1 , wherein obtaining the plurality of votes based on performing dynamic voting comprises: performing the dynamic voting to generate a vote for at least one current image frame; obtaining votes generated for at least one previous image frame; as well as The votes for at least one current image frame and the votes generated for at least one previous image frame are aggregated to obtain the plurality of votes. The method according to claim 1 , further comprising triggering the dynamic voting. The method according to claim 3 , wherein the triggering of the dynamic voting is based on a timer. The method of claim 1 , further comprising obtaining, by a sensor, an image frame of the captured scene. The method of claim 5 , wherein the image processor and the sensor operate inline in terms of timing. 7 . The method of claim 5 , further comprising outputting, by the sensor, pixels of the image frame to the image processor.

8. The method of claim 5, further comprising processing the image frame by the image processor to generate processed image data.

9. The method of claim 8, further comprising sending the processed image data to a memory.

10. The method of claim 9, wherein the memory is a double data rate (DDR) memory. The method of claim 1 , wherein the image processor is a front-end component of a camera.

12. The method of claim 1, wherein the image processor is an inline image processor.

13. The method of claim 1, wherein the image processor is an image signal processor (ISP).

14. The method of claim 1, wherein the image processor and the plurality of components are on a system on a chip (SOC).

15. The method of claim 1, wherein the power source shared by the plurality of components is one or more power rails.

16. A device for processing image data, the device comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtaining a plurality of votes associated with a plurality of components sharing a power source based on performing dynamic voting; determining a voting result based on the plurality of votes; increasing or decreasing a clock rate and a voltage of the power supply based on the voting result to generate an updated clock rate and an updated voltage; as well as The updated clock rate and the updated voltage are applied to an image processor.

17. The apparatus of claim 16, wherein to obtain the plurality of votes based on performing the dynamic voting, the at least one processor is configured to: performing the dynamic voting to generate a vote for at least one current image frame; obtaining votes generated for at least one previous image frame; as well as The votes for at least one current image frame and the votes generated for at least one previous image frame are aggregated to obtain the plurality of votes.

18. The apparatus of claim 16, wherein the at least one processor is configured to trigger the dynamic voting.

19. The apparatus of claim 16, wherein the at least one processor is configured to trigger the dynamic voting based on a timer.

20. The apparatus according to claim 16, further comprising: A sensor is configured to capture an image frame of a scene.

21. The apparatus of claim 20, wherein the sensor is configured to output pixels of the image frame to the image processor.

22. The apparatus of claim 20, further comprising the image processor, wherein the image processor operates inline with the sensor in terms of time sequence.

23. The apparatus of claim 20, wherein the image processor is configured to process the image frames to generate processed image data.

24. The apparatus of claim 23, wherein the at least one processor is configured to send the processed image data to the at least one memory.

25. The apparatus of claim 24, wherein the at least one memory is a double data rate (DDR) memory.

26. The apparatus of claim 16, wherein the image processor is a front-end component of a camera.

27. The apparatus of claim 16, wherein the image processor is an inline image processor.

28. The apparatus of claim 16, wherein the image processor is an image signal processor (ISP).

29. The apparatus of claim 16, wherein the image processor and the plurality of components are on a system on a chip (SOC).

30. The apparatus of claim 16, wherein the power source shared by the plurality of components is one or more power rails.

Images