A method and apparatus for optimizing fast sensor mode power based on camera dynamic voting

Abstract

Systems, methods, and computer-readable media are provided for camera dynamic voting to optimize fast sensor mode power. In some examples, a computing device can obtain a plurality of votes associated with a plurality of components sharing a power supply based on performing dynamic voting. The computing device can determine a voting result based on the plurality of votes. The computing device can increase or decrease a clock rate and a voltage of the power supply based on the voting result to produce an updated clock rate and an updated voltage. The computing device can then apply the updated clock rate and the updated voltage to an image processor.

Description

Technical Field

[0001] This application relates to image processing. In some examples, aspects of this application relate to systems and techniques for camera dynamic voting to optimize power in fast sensor modes. Background Technology

[0002] The increasing versatility of digital cameras has allowed them to be integrated into a wide variety of devices, expanding their applications. For example, phones, drones, cars, computers, televisions, and many other devices today are frequently equipped with camera devices. Camera devices allow users to capture images and / or videos (e.g., frames of images) from any system equipped with them. These images and / or videos can be captured for entertainment, professional photography, surveillance, automation, and other applications. Furthermore, camera devices are increasingly equipped with specific features for modifying images or creating artistic effects on them. For example, many camera devices are equipped with image processing capabilities for generating different effects on captured images.

[0003] For image processing, Fast Sensor Readout (FSR) is a common sensor operating mode used by many original equipment manufacturers (OEMs) to improve image quality (IQ) because FSR results in a reduction of shutter-related artifacts in the image. Compared to normal readout mode, FSR mode allows for faster readout of image sensor data from image frames while maintaining the same amount of exposure time. The faster the readout of image sensor data, the lower the amount of shutter-related artifacts present in the rendered image. FSR mode also allows for a reduction in required sensor power (e.g., in some cases, FSR mode can allow for 160 to 260 milliwatts of sensor power savings compared to normal readout mode). Therefore, from both an IQ and sensor power perspective, using FSR for image processing can be advantageous.

[0004] However, FSR mode can impact the power consumption of the chipset (e.g., which may include an image signal processor). For example, FSR mode may require the image signal processor to operate at very high clock rates and voltages to complete image frame processing during the compressed readout time of the FSR. Furthermore, FSR mode may require memory (e.g., double data rate (DDR) memory) to operate at high clock rates to enable rapid output of data received from the image sensor processor. Therefore, improved techniques for optimizing the power consumption of FSR mode may be beneficial. Summary of the Invention

[0005] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all conceptual aspects, nor should it be considered to identify key or decisive elements relating to all conceptual aspects or to depict the scope associated with any particular aspect. Thus, the sole purpose of this summary is to present, in a concise form, certain concepts relating to one or more aspects involving the mechanisms disclosed herein, prior to the detailed description presented below.

[0006] Systems and techniques for camera dynamic voting to optimize power in fast sensor modes are described. According to at least one example, a method for processing image data is provided. The method includes: obtaining multiple votes associated with multiple components sharing a power supply based on performing dynamic voting; determining a voting result based on the multiple votes; increasing or decreasing the clock rate and voltage of the power supply based on the voting result to generate an updated clock rate and updated voltage; and applying the updated clock rate and updated voltage to an image processor.

[0007] In another exemplary example, an apparatus for processing image data is provided. The apparatus includes at least one memory and at least one processor coupled to the at least one memory and configured to: obtain multiple votes associated with multiple components sharing a power supply based on performing dynamic voting; determine a voting result based on the multiple votes; increase or decrease the clock rate and voltage of the power supply based on the voting result to generate an updated clock rate and updated voltage; and apply the updated clock rate and updated voltage to the image processor.

[0008] In another exemplary example, a non-transitory computer-readable medium is provided. The computer-readable medium includes instructions stored thereon that, when executed by at least one processor, cause the at least one processor to: obtain multiple votes associated with multiple components sharing a power supply based on performing dynamic voting; determine a voting result based on the multiple votes; increase or decrease the clock rate and voltage of the power supply based on the voting result to produce an updated clock rate and an updated voltage; and apply the updated clock rate and the updated voltage to an image processor.

[0009] In another exemplary example, an apparatus for processing image data is provided. The apparatus includes: components for obtaining multiple votes associated with multiple components sharing a power supply based on performing dynamic voting; components for determining a voting result based on the multiple votes; components for increasing or decreasing the clock rate and voltage of the power supply based on the voting result to generate an updated clock rate and updated voltage; and components for applying the updated clock rate and updated voltage to an image processor.

[0010] The aspects generally include, as described substantially with reference to the accompanying drawings and description and illustrated as shown in the drawings and description, methods, apparatuses, systems, computer program products, non-transitory computer-readable media, user equipment, user gear, wireless communication equipment and / or processing systems.

[0011] In some aspects, each of the devices described above is a mobile device, a smart or connected device, a camera system, and / or an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), or may be part of or include such devices. In some examples, the device may include a vehicle, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotic device or system, an aviation system, or other devices, or be part of such devices. In some aspects, the device includes one image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the devices described above may include one or more sensors. In some cases, the one or more sensors may be used to determine the location of the device, the state of the device (e.g., tracking state, operating state, temperature, humidity level, and / or another state), and / or for other purposes.

[0012] Some aspects include a device having a processor configured to perform one or more operations of any of the methods outlined above. Further aspects include a processing device for use in the device, the processing device being configured with processor-executable instructions to perform operations of any of the methods outlined above. Further aspects include a non-transitory processor-readable storage medium storing processor-executable instructions thereon, the processor-executable instructions being configured to cause the processor of the device to perform operations of any of the methods outlined above. Further aspects include a device having components for performing functions of any of the methods outlined above.

[0013] The features and technical advantages of the examples according to this disclosure have been summarized rather extensively above in order to better understand the detailed description below. Additional features and advantages will be described below. The disclosed concepts and specific examples can be readily utilized as the basis for modifying or designing other structures for achieving the same purpose of this disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein (both their organization and operation) and their associated advantages will be better understood from the following description when considered in conjunction with the accompanying drawings. Each figure in the drawings is provided for illustrative and descriptive purposes and not as a definition of limitation of the claims. The foregoing, as well as other features and aspects, will become more apparent upon reference to the following specification, claims, and appended drawings.

[0014] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0015] The above and other features and embodiments will become more apparent when the following description, claims and drawings are taken into account. Attached Figure Description

[0016] Examples of specific implementations are described in detail below with reference to the accompanying figures:

[0017] Figure 1 This is a block diagram illustrating an example architecture of an image capture and processing system based on some examples.

[0018] Figure 2 This is a block diagram illustrating examples of interactions between components of an image capture and processing system, based on some examples.

[0019] Figure 3 This is a block diagram of an example device that can be used for camera dynamic voting to optimize power in fast sensor modes, based on some examples.

[0020] Figure 4 This is a block diagram illustrating the operation of an image signal processor pipeline based on some examples.

[0021] Figure 5 This is a diagram illustrating an example of camera sensor timing based on some examples.

[0022] Figure 6 This is a graph showing example power consumption of the entire camera module based on some examples.

[0023] Figure 7These are graphs showing example power consumption waveforms captured at the camera's Mobile Industrial Processor Interface (MIPI) according to some examples.

[0024] Figure 8 This is an example diagram illustrating a camera system for showing data flow, based on some examples.

[0025] Figure 9 These are example graphs showing the activity timeline of some example image frames.

[0026] Figure 10 This is a graph showing the chipset power and sample power consumption of the entire camera module based on some example cameras.

[0027] Figure 11 This is a diagram illustrating examples of timing for cameras that employ a dynamic voting mechanism to optimize power in fast sensor modes, based on some examples.

[0028] Figure 12 This is a diagram illustrating an example of a system that uses a dynamic voting mechanism to optimize the power of a fast sensor mode, based on some examples.

[0029] Figure 13 This is a table showing examples of power savings when using dynamic voting to optimize power in fast sensor modes, based on some examples.

[0030] Figure 14 This is a flowchart illustrating another example of a process for camera dynamic voting to optimize power in fast sensor modes, based on some examples.

[0031] Figure 15 This is a diagram illustrating an example of a system used to implement some of the aspects described in this article. Detailed Implementation

[0032] Certain aspects and embodiments of this disclosure are provided below. Some of these aspects and embodiments may be applied independently, and some may be combined, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation in order to provide a thorough understanding of the various embodiments of this application. However, it will be apparent, however, that the various embodiments may be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0033] The following description provides only exemplary embodiments and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0034] A camera is a device that uses an image sensor to receive light and capture image frames (such as still images or video frames). The terms "image," "image frame," and "frame" are used interchangeably herein. A camera may include a processor (such as an image signal processor (ISP)) that receives and processes one or more image frames. For example, raw image frames captured by a camera sensor may be processed by an ISP to generate a final image. The processing performed by the ISP may be performed by multiple filters or processing blocks applied to the captured image frames, such as denoising or noise filtering, edge enhancement, color balancing, contrast adjustment, intensity adjustment (such as darkening or brightening), tone adjustment, etc. Image processing blocks or modules may include lens / sensor noise correction, Bayer filters, de-mosaicing, color conversion, correction or enhancement / suppression of image attributes, noise reduction filters, sharpening filters, etc.

[0035] Cameras can be configured with various image capture and image processing operations and settings. Different settings produce images with different appearances. Some camera operations are determined and applied before or during image capture, such as automatic exposure control (AEC) and automatic white balance (AWB) processing. Additional camera operations applied before, during, or after image capture include those involving scaling (e.g., zooming in or out), ISO, aperture size, aperture coefficient, shutter speed, and gain. Other camera operations configure post-processing of the image, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color.

[0036] As previously mentioned, for image processing, FSR is a common sensor operating mode used by many OEMs to improve IQ (Input Quality) because it reduces shutter-related artifacts in the image. Compared to normal readout mode, FSR mode allows for faster readout of image sensor data from image frames while maintaining the same amount of exposure time. The faster the readout of image sensor data, the lower the amount of shutter-related artifacts present in the rendered image. FSR mode can also allow for a reduction in required sensor power (e.g., in some cases, FSR mode can allow for 160 to 260 milliwatts of sensor power savings compared to normal readout mode). Therefore, from both an IQ and sensor power perspective, utilizing FSR mode for image processing can be beneficial.

[0037] However, FSR mode can impact the power consumption of a chipset that may include an image signal processor (ISP). For example, FSR mode may require the ISP to operate at very high clock rates and voltages to complete image frame processing during the compressed readout time of FSR mode. Furthermore, FSR mode may require memory (e.g., DDR memory) to operate at high clock rates to enable rapid output of data received from the image sensor processor. Therefore, improved techniques for optimizing the power consumption of FSR mode can be useful.

[0038] Therefore, this document describes systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively, “systems and techniques”) for camera dynamic voting to optimize power consumption in fast sensor modes (e.g., FSR mode power). For example, in some examples, systems and techniques can optimize the power overhead incurred during camera operation in FSR mode by camera chipsets (such as system-on-chip (SOC)). Utilizing conventional static clocking mechanisms, ISPs and DDR memories have fixed clock rates to meet the instantaneous performance requirements of use cases, which can result in significant power overhead required throughout the use case timeline.

[0039] In one or more aspects, the system and technology dynamically increase the ISP and DDR clock rates during the short sensor readout duration of the use case timeline and immediately decrease the ISP and DDR clock rates after the short sensor readout duration has ended (e.g., to make the clock rate lower during the large blanking interval in the use case timeline, where no sensor readout is performed). With such clock rate adjustment, high power overhead requirements can be limited to the sensor readout portion of the use case timeline (e.g., which is typically only 25% to 50% of the use case timeline).

[0040] Most camera use cases operate at a specified image frame rate configured by advanced application software (SW). This system and technology can employ configurable hardware (HW) timer mechanisms (e.g., Figure 12 The dynamic clock voting engine 1250 triggers a dynamic increase in the clock rate immediately before a new image frame is received from the image sensor, and a dynamic decrease in the clock rate and shared trunk voltage immediately after the image frame processing is completed.

[0041] The accompanying drawings are used to describe other aspects of the system and technology.

[0042] Figure 1This is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., an image of scene 110). The image capture and processing system 100 can capture individual images (or photographs) and / or capture video comprising multiple images (or video frames) in a specific sequence. A lens 115 of the system 100 faces scene 110 and receives light from scene 110. The lens 115 bends the light toward an image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0043] One or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or information from image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components; for example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms besides those illustrated, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture attributes.

[0044] The focus control mechanism 125B of the control mechanism 120 can obtain the focus settings. In some examples, the focus control mechanism 125B stores the focus settings in a memory register. Based on the focus settings, the focus control mechanism 125B can adjust the positioning of the lens 115 relative to the positioning of the image sensor 130. For example, based on the focus settings, the focus control mechanism 125B can adjust the focus by moving the lens 115 closer to or further away from the image sensor 130 via an actuated motor or servo. In some cases, the device 105A may include additional lenses, such as one or more microlenses on each photodiode of the image sensor 130, each microlens bending light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus settings can be determined by contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus settings can be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings.

[0045] The exposure control mechanism 125A of the control mechanism 120 can obtain the exposure settings. In some cases, the exposure control mechanism 125A stores the exposure settings in a memory register. Based on the exposure settings, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or aperture order), the duration of aperture opening (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure settings may be referred to as image capture settings and / or image processing settings.

[0046] The zoom control mechanism 125C of the control mechanism 120 can obtain zoom settings. In some examples, the zoom control mechanism 125C stores the zoom settings in a memory register. Based on the zoom settings, the zoom control mechanism 125C can control the focal length of an assembly (lens assembly) comprising lens elements including lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servos to move one or more lenses in the lens relative to each other. The zoom settings may be referred to as image capture settings and / or image processing settings. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, this focusing lens may be lens 115) that first receives light from scene 110, wherein the light then passes through a focusless zoom system between the focusing lens (e.g., lens 115) and image sensor 130 before reaching image sensor 130. In some cases, a focusless zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control mechanism 125C moves one or more lenses in the focusless zoom system, such as the negative lens and one or both positive lenses.

[0047] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus light matching the color of the color filter covering the photodiode can be measured. For example, Bayer color filters include red, blue, and green color filters, where each pixel of the image is generated based on red light data from at least one photodiode covered by the red color filter, blue light data from at least one photodiode covered by the blue color filter, and green light data from at least one photodiode covered by the green color filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald") color filters instead of or supplement red, blue, and / or green color filters. Some image sensors may have no color filters at all and may alternatively use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes throughout the pixel array may have different spectral sensitivity profiles, thereby responding to light of different wavelengths. Monochrome image sensors may also lack color filters and therefore lack color depth.

[0048] In some cases, image sensor 130 may alternatively or additionally include opaque masks and / or reflective masks that block light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which can be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output from the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output from the photodiodes (and / or the analog signal amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed with respect to one or more control mechanisms in control mechanism 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge-coupled device (CCD) sensor, an electron multiplication CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide-semiconductor (CMOS), an N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0049] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other type of processor 1510 discussed with respect to the computing system 1500. The host processor 152 may be a digital signal processor (DSP) and / or other types of processor. In some specific implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) including the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, and connectivity components (e.g., Bluetooth). ™ This includes components such as the Global Positioning System (GPS), any combination thereof, and / or other components. I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as Inter-Integrated Circuit 2 (I2C) interface, Inter-Integrated Circuit 3 (I3C) interface, Serial Peripheral Interface (SPI) interface, Serial General Purpose Input / Output (GPIO) interface, Mobile Industrial Processor Interface (MIPI) (such as MIPI CSI-2 physical (PHY) layer port or interface, Advanced High Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports). In an exemplary example, host processor 152 may use the I2C port to communicate with image sensor 130, and ISP 154 may use the MIPI port to communicate with image sensor 130.

[0050] Image processor 150 can perform multiple tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 / 1525, read-only memory (ROM) 145 / 1520, cache 1512, memory unit 1515, another storage device 1530, or some combination thereof.

[0051] Various input / output (I / O) devices 160 may be connected to the image processor 150. I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, touchpad, touch-sensitive surface, printer, any other output device 1535, any other input device 1545, or some combination thereof. In some cases, text may be input to the image processing device 105B via the physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad on the touchscreen of the I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors enabling a wired connection between the device 105B and one or more peripheral devices, through which the device 105B can receive data from and / or send data to the one or more peripheral devices. I / O 160 may include one or more wireless transceivers enabling a wireless connection between the device 105B and one or more peripheral devices, through which the device 105B can receive data from and / or send data to the one or more peripheral devices. Peripheral devices may include any type of I / O device 160 discussed earlier, and they can be considered as I / O devices 160 themselves once they are coupled to ports, jacks, wireless transceivers or other wired and / or wireless connectors.

[0052] In some cases, the image capture and processing system 100 may be a single device. In other cases, the image capture and processing system 100 may be two or more independent devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some embodiments, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0053] like Figure 1 As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, namely, image capture device 105A and image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, certain components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0054] Image capture and processing system 100 may include electronic devices such as mobile or landline phones (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, image capture and processing system 100 may include one or more wireless transceivers for wireless communication (such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof). In some specific implementations, image capture device 105A and image processing device 105B may be different devices. For example, image capture device 105A may include a camera device, and image processing device 105B may include a computing device, such as a mobile phone, desktop computer, or other computing device.

[0055] Although the image capture and processing system 100 is shown to include certain components, those skilled in the art will understand that the image capture and processing system 100 may include more than [other components]. Figure 1 The components shown are further components. Components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the image capture and processing system 100 may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. Software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 100.

[0056] The host processor 152 can configure the image sensor 130 with new parameter settings (e.g., via an external control interface such as I2C, I3C, SPI, GPIO, and / or other interfaces). In one exemplary example, the host processor 152 can update the exposure settings used by the image sensor 130 based on the internal processing results of the exposure control algorithm from past image frames.

[0057] In some examples, the host processor 152 may perform electronic image stabilization (EIS). For example, the host processor 152 may determine motion vectors corresponding to motion compensation for one or more image frames. In some aspects, the host processor 152 may position a cropped array of pixels (“image window”) within a total pixel array. The image window may include pixels used to capture the image. In some examples, the image window may include all pixels in the sensor except for a portion of rows and columns at the periphery of the sensor. In some cases, the image window may be centered on the sensor when the image capture device 105A is stationary. In some aspects, peripheral pixels may surround the pixels of the image window and form a set of buffered pixel rows and buffered pixel columns around the image window. The host processor 152 may implement EIS and shift the image window from frame to frame of video such that the image window tracks the same scene on consecutive frames (e.g., assuming the subject does not move). In some examples where the subject moves, the host processor 152 may determine that the scene has changed.

[0058] In some examples, the image window may include at least 95% (e.g., 95% to 99%) of the pixels on the sensor. The first region of interest (ROI) (e.g., for AE and / or AWB) may include at least 95% (e.g., 95% to 99%) of the image data within the field of view of a plurality of imaging pixels in the image sensor 130 of the image capture device 105A. In some aspects, a plurality of buffer pixels may be reserved at the periphery of the sensor (outside the image window) as a buffer to allow the image window to shift to compensate for jitter. In some cases, the image window may be moved such that even if light from the subject is projected onto different areas of the sensor, the subject remains in the same position within the adjusted image window. In another example, the buffer pixels may include the top ten, bottom ten, left ten, and right ten columns of pixels on the sensor. In some configurations, when the image capture device 105A is stationary, the buffer pixels are not used for AF, AE, or AWB, and the buffer pixels are not included in the image output. If jitter causes the sensor to shift to the left by twice the width of the pixel column between frames, the EIS algorithm can be used to shift the image window to the right by two columns of pixels, so that the captured image shows the same scene in the current frame in the next frame. The host processor 152 can use EIS to make the transition from one frame to the next smooth.

[0059] In some respects, the host processor 152 can also dynamically configure the parameter settings of the internal pipelines or modules of the ISP 154 to match the settings of one or more input image frames from the image sensor 130, so that the image data is correctly processed by the ISP 154. The processing (or pipeline) blocks or modules of the ISP 154 may include modules for lens / sensor noise correction, demosaicing, color conversion, correction or enhancement / suppression of image attributes, denoising filters, sharpening filters, etc. The settings of different modules of the ISP 154 can be configured by the host processor 152. Each module may include a large number of tunable parameter settings. Furthermore, since different modules may affect similar aspects of the image, the modules can be interdependent. For example, denoising and texture correction or enhancement may both affect the high-frequency aspects of the image. As a result, a large number of parameters are used by the ISP to generate the final image based on the captured raw image.

[0060] In some cases, the image capture and processing system 100 can automatically perform one or more of the image processing functions described above. For example, one or more of the control mechanisms 120 may be configured to perform autofocus operation, auto exposure operation, and / or auto white balance operation. In some embodiments, the autofocus function allows the image capture device 105A to automatically focus before capturing the desired image. Various autofocus techniques exist. For example, active autofocus techniques typically determine the distance between the camera and the subject of the image via the camera's distance sensor by emitting infrared laser or ultrasonic signals and receiving the reflections of these signals. Furthermore, passive autofocus techniques use the camera's own image sensor to focus the camera and therefore do not require the integration of additional sensors into the camera. Passive AF techniques include contrast detection autofocus (CDAF), phase detection autofocus (PDAF), and in some cases, hybrid systems using both techniques. The image capture and processing system 100 may be equipped with these or any additional types of autofocus techniques.

[0061] Synchronization between the image sensor 130 and the ISP 154 is important in order to provide an operational image capture system that can generate high-quality images without interruption and / or failure. Figure 2 This is a block diagram illustrating an example of an image capture and processing system 200, which includes an image processor 250 (including a host processor 252 and an ISP 254) communicating with an image sensor 230. Figure 2The configuration shown illustrates a conventional synchronization technique used in a camera system. Generally, the host processor 252 attempts to provide synchronization between the image sensor 230 and the ISP 254 using fixed time intervals by communicating separately with both the image sensor 230 and the ISP 254. For example, in a conventional camera system, the host processor 252 communicates with the image sensor 230 (e.g., via an I2C port) and programs the parameters of the image sensor 230 for a first fixed time interval (such as a two-frame interval before the image frame will be processed by the ISP 254). The host processor 252 communicates with the ISP 254 (e.g., via an internal AHB bus or other interface) and programs the ISP 254 parameter settings for a second fixed time interval (such as a one-frame interval before the image frame will be processed by the ISP 254).

[0062] Image sensor 230 can transmit image frames to ISP 254, such as via MIPI CSI-2 PHY port or interface or other suitable interface. Figure 2 (See B to C in the diagram). However, communication between the host processor 252 and the image sensor 230 (shown as A to B) is indeterminate. Similarly, communication between the image sensor 230 and the ISP 254 (shown as B to C) and between the host processor 252 and the ISP 254 (shown as A to C) is also indeterminate. For example, the programming of the image sensor 230 and the ISP 254 by the host processor 252 may have varying latency, which can lead to a mismatch in parameter settings between the sensor and the ISP. Latency can be attributed to high CPU utilization, congestion in one or more I / O ports, and / or other factors.

[0063] Figure 3This is a block diagram of an example device 300 that can be used for camera dynamic voting to optimize power in fast sensor modes. Device 300 may include or be coupled to camera 302, and may also include processor 306, memory 308 storing instructions 310, camera controller 312, display 316, and multiple input / output (I / O) components 318 including one or more microphones (not shown). Example device 300 may be any suitable device capable of capturing and / or storing images or videos, including, for example, wired and wireless communication devices (such as camera phones, smartphones, tablets, security systems, smart home devices, connected home devices, surveillance devices, Internet Protocol (IP) devices, dashcams, laptops, desktop computers, automobiles, drones, aircraft, etc.), digital cameras (including still cameras, camcorders, etc.), or any other suitable device. Device 300 may include additional features or components not shown. For example, it may include a wireless interface for wireless communication devices, which may include multiple transceivers and a baseband processor. Device 300 may include or be coupled to an additional camera other than camera 302. This disclosure should not be limited to any particular example or illustration, including example device 300.

[0064] Camera 302 may be capable of capturing individual image frames (such as still images) and / or capturing video (such as a sequence of captured image frames). Camera 302 may include one or more image sensors (not shown for simplicity) and a shutter for capturing image frames and providing the captured image frames to camera controller 312. Although a single camera 302 is shown, any number of cameras or camera components may be included and / or coupled to device 300. For example, the number of cameras may be increased to achieve greater depth determination capability or better resolution for a given field of view (FOV).

[0065] The memory 308 may be a non-transient or non-transitory computer-readable medium storing all or a portion of computer-executable instructions 310 for performing one or more of the operations described in this disclosure. The device 300 may also include a power supply 320 that may be coupled to or integrated into the device 300.

[0066] Processor 306 may be one or more suitable processors capable of executing scripts or instructions of one or more software programs (such as instructions 310) stored in memory 308. In some aspects, processor 306 may be one or more general-purpose processors that execute instructions 310 to cause device 300 to perform any number of functions or operations. In additional or alternative aspects, processor 306 may include integrated circuits or other hardware for performing functions or operations without the use of software. Although in Figure 3The example is shown as coupled to each other via processor 306, but processor 306, memory 308, camera controller 312, display 316 and I / O components 318 can be coupled to each other in various arrangements. For example, processor 306, memory 308, camera controller 312, display 316 and / or I / O components 318 can be coupled to each other via one or more local buses (not shown for simplicity).

[0067] Display 316 may be any suitable display or screen that allows the user to interact with and / or present items (such as captured images and / or videos) for the user to view. In some aspects, display 316 may be a touch-sensitive display. Display 316 may be part of device 300 or external to that device. Display 316 may include LCD, LED, OLED, or similar displays. I / O component 318 may be or may include any suitable mechanism or interface for receiving input (such as commands) from the user and / or providing output to the user. For example, I / O component 318 may include (but is not limited to) a graphical user interface, keyboard, mouse, microphone, and speaker, etc.

[0068] Camera controller 312 may include image signal processor (ISP) 314, which may be (or may include) one or more image signal processors for processing image frames or video captured by camera 302. For example, ISP 314 may be configured to perform various processing operations for autofocus (AF), automatic white balance (AWB), and / or automatic exposure (AE) (which may also be referred to as automatic exposure control (AEC)). Examples of image processing operations include, but are not limited to, cropping, scaling (e.g., scaling to different resolutions), image stitching, image format conversion, color interpolation, image interpolation, color processing, image filtering (e.g., spatial image filtering), etc.

[0069] In some example implementations, the camera controller 312 (such as ISP 314) may implement various functionalities, including image processing and / or control operations of the camera 302. In some aspects, the ISP 314 may execute instructions from memory (such as instructions 310 stored in memory 308 or stored in a separate memory coupled to the ISP 314) to control image processing and / or operation of the camera 302. In other aspects, the ISP 314 may include specific hardware for controlling image processing and / or operation of the camera 302. The ISP 314 may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions.

[0070] Although Figure 3Not shown, but in some implementations, the ISP 314 and / or camera controller 312 may include an AF module, an AWB module, and / or an AE module. The ISP 314 and / or camera controller 312 may be configured to perform AF, AWB, and / or AE processes. In some examples, the ISP 314 and / or camera controller 312 may include dedicated hardware circuitry (e.g., an application-specific integrated circuit (ASIC)) configured to perform the AF, AWB, and / or AE processes. In other examples, the ISP 314 and / or camera controller 312 may be configured to execute software and / or firmware to perform the AF, AWB, and / or AE processes. When configured in software, the code for the AF, AWB, and / or AE processes may be stored in memory (such as instructions 310 stored in memory 308 or instructions stored in a separate memory coupled to the ISP 314 and / or camera controller 312). In other examples, the ISP 314 and / or camera controller 312 may use a combination of hardware, firmware, and / or software to perform AF, AWB, and / or AE processes. When configured as software, the AF, AWB, and / or AE processes may include instructions for configuring the ISP 314 and / or camera controller 312 to perform various image processing and device management tasks, including those using techniques disclosed herein.

[0071] Figure 4 This is a block diagram illustrating the operation of an image signal processing pipeline 402 of an image signal processor (e.g., ISP 314). For example, the ISP 314 may be configured to execute the image signal processing pipeline 402 to process input image data. The ISP 314 can be accessed from... Figure 3 The camera 302 and / or the image sensor (not shown) of the camera 302 receive input image data. In some examples, such as Figure 4 As shown, input image data may include color data and / or any other data (e.g., depth data) of the image / frame. Figure 4 In the example, the color data received for the input image data can be in Bayer format. Instead of capturing the red (R), green (G), and blue (B) values ​​for each pixel of the image, the image sensor (e.g., the image sensor of camera 302) can use a Bayer filter mosaic (or more generally, a color filter array (CFA)). This would allow each photoelectric sensor in the digital image sensor to capture different colors in the RGB color spectrum. An example of a filter pattern used for a Bayer filter mosaic could include a 50% green filter, a 25% red filter, and a 25% blue filter.

[0072] The Bayer processing unit 410 can perform one or more initial processing techniques on the raw Bayer data received by the ISP 314, including, for example, subtraction, slip correction, defective pixel correction, black level compensation and / or denoising.

[0073] The stats filtering process 412 determines the Bayer rank or Bayer grid (BG) statistics of the received input image data. In some examples, BG statistics may include the red to green ratio (R / G) (which indicates the presence of red tinting and the amount of red tinting that may be present in the image) and / or the blue to green ratio (B / G) (which indicates the presence of blue tinting and the amount of blue tinting that may be present in the image). For example, the (R / G) of an image or a portion / region of an image may be described by the following formula (1):

[0074]

[0075] The image or a portion / region of an image comprises pixels 1-N, where each pixel n includes a red value (Red(n), a blue value (Blue(n), or a green value (Green(n))) in the RGB space. (R / G) is the sum of the red values ​​of the red pixels in the image divided by the sum of the green values ​​of the green pixels in the image. Similarly, (B / G) of the image or a portion / region of an image can be described by the following formula (2):

[0076]

[0077] In some other example implementations, different color spaces, such as Y'UV, may be used, where the chromaticity values ​​UV indicate color, and / or other indicators that may determine the tinting or other color temperature effects on an image.

[0078] The AWB module and / or process 404 can analyze information associated with the received image data to determine the scene's illuminators from a plurality of possible illuminators, and can determine the AWB gain based on the determined illuminators to be applied to the received image and / or subsequent images. White balance is a process used to attempt to match the colors of an image with the user's perceptual experience of the captured object. As an example, white balance processing can be designed so that white objects actually appear white in the processed image, and gray objects actually appear gray in the processed image.

[0079] The illuminator may include the lighting conditions of the scene being captured, the type of light, etc. In some examples, the image capture device (e.g., such as...) Figure 3The user of the device (300) can select or specify the illuminators under which the image is captured. In other examples, the image capture device itself can automatically determine the most likely illuminators and perform white balance based on the determined illuminators (e.g., lighting conditions). To better render the colors of the scene in the captured image or video, the AWB algorithm on the device and / or camera can attempt to determine the illuminators of the scene and set / adjust the white balance of the image or video accordingly.

[0080] During AWB process 404, device 300 can determine or estimate the color temperature of a received frame (e.g., an image). Color temperature indicates the dominant hue of an image. The true color temperature of a scene captured in a video or image is the color of the scene's light source. If the light is emitted from a perfect blackbody radiator (theoretically ideal for all electromagnetic wavelengths) at a specific color temperature (expressed in Kelvin (K)), and the color temperature is known, then the color temperature of the scene is known. For example, in the color space defined by the International Commission on Illumination (CIE) (from 1931), the chromaticity of radiation from a blackbody radiator with temperatures ranging from 1,000 K to 20,000 K is the Planck locus. Colors on the Planck locus from approximately 2,000 K to 20,000 K are considered white, where 2,000 K is warm white or reddish white, and 20,000 K is cool white or bluish white. Many incandescent light sources include Planck radiators (tungsten filaments or another filament used for light emission), which emit warm white light with a color temperature of approximately 2,400 K to 3,100 K.

[0081] However, other light sources (such as fluorescent lamps, discharge lamps, or light-emitting diodes (LEDs)) are not perfect blackbody radiators whose radiation falls along the Planck locus. For example, LEDs or neon signs emit light through electroluminescence, and the color of the light does not follow the Planck locus. The color temperature determined for such light sources can be the correlated color temperature (CCT). The CCT is an estimated color temperature of a light source whose color does not fall perfectly along the Planck locus. For example, the CCT of a light source is the blackbody color temperature that is closest to the radiation of the light source. CCT can also be represented in K.

[0082] CCT can be an approximation of the true color temperature of a scene. For example, CCT can be a simplified color measure of the chromaticity coordinates in the CIE 1931 color space. Many devices can use AWB to estimate CCT for color balance.

[0083] CCT can be a temperature rating ranging from warm colors (such as yellow and red below 3200K) to cool colors (such as blue above 4000K). CCT (or other color temperature) indicates the tinting that will appear in an image captured using such a light source. For example, a CCT of 2700K indicates red tinting, and a CCT of 5000K indicates blue tinting.

[0084] Different lighting sources or ambient lighting can illuminate a scene, and the color temperature may be unknown to the device. Therefore, the device can analyze data captured by an image sensor to estimate the color temperature of an image (e.g., a frame). For example, the color temperature could be an estimate of the overall CCT of the light sources in the scene within the image. The data captured by the image sensor for estimating the color temperature of a frame (e.g., an image) could be the captured image itself.

[0085] After device 300 determines the color temperature of the scene (such as during AWB execution), device 300 can use the color temperature to determine the color balance for correcting any shading in the image. For example, if the color temperature indicates that the image includes red shading, device 300 can, for example, decrease the red value or increase the blue value for each pixel of the image in the RGB space. Color balance can be color correction (such as values ​​used to decrease red values ​​or increase blue values).

[0086] Example inputs to the AWB process 404 may include Bayer class or Bayer grid (BG) statistics of the received image data determined via the statistical filtering process 412, exposure index (e.g., the brightness of the scene in the received image data), and auxiliary information, which may include contextual information of the scene based on the audio input (as will be discussed in further detail below), depth information, etc. It should be noted that the AWB process 404 may be included as a separate AWB module. Figure 3 The camera controller 312 is located within it.

[0087] AE process 406 may include configuration, calculation and / or storage. Figure 3 The camera 302 provides instructions for exposure settings. Exposure settings may include the amount of sensor gain to apply, the amount of digital gain to apply, shutter speed and / or exposure time, aperture setting, and / or ISO setting for capturing subsequent images. The AE process 406 can use audio input and / or scene context information based on the audio input to determine and / or apply exposure settings more quickly. It should be noted that the AE process 406 can be included as a separate AE module. Figure 3 The camera controller 312 is located within it.

[0088] AF process 408 may include configuration, calculation and / or storage. Figure 3 The AF process 408 provides instructions for setting the autofocus on camera 302. The AF process 408 can determine the autofocus settings (e.g., initial lens position, final lens position, etc.) based on audio input and / or contextual information of the scene based on the audio input. It should be noted that the AF process 408 can be included as a separate AF module. Figure 3 The camera controller 312 is located within it.

[0089] The demosaicing unit 414 can be configured to convert the processed Bayer image data into RGB values ​​for each pixel of the image. As explained above, the Bayer data may include only the value of one color channel (R, G, or B) for each pixel of the image. The demosaicing unit 414 can determine the values ​​of the other color channels of a pixel by interpolating from the color channel values ​​of neighboring pixels. In some ISP pipelines 402, the demosaicing unit 414 may be located before or after the AWB process 404, AE process 406, and / or AF process 408.

[0090] Other processing units 416 may apply additional processing to the image after the AWB process 404, AE process 406, and / or AF process 408, and / or demosaic processing unit 414. Additional processing may include color, hue, and / or spatial processing of the image.

[0091] As previously mentioned, for image processing, FSR is a common sensor operating mode used by many OEMs to improve IQ because it reduces shutter-related artifacts in the image. Compared to normal readout mode, FSR mode allows for faster readout of image sensor data for image frames while maintaining the same amount of exposure time. The faster the image sensor data readout is performed, the lower the amount of shutter-related artifacts present in the rendered image.

[0092] Figure 5 This is a diagram comparing the timings of FSR mode with those of normal read mode. Specifically, Figure 5 This is a diagram illustrating an example of camera sensor timing 500. Figure 5 In this process, the timing diagram 560 of FSR mode is compared with the timing diagram 550 of normal readout mode.

[0093] exist Figure 5 In the normal readout mode timing diagram 550, the exposure time of the row (e.g., time period 510) and the data from one or more camera sensors (e.g., Figure 12 Sensors 1210a to 1210b) to a camera SoC (which may include one or more ISPs (e.g., Figure 12 The amount of time to read the row for ISP 1230a to ISP 1230b (e.g., time period 520).

[0094] like Figure 5As shown, these rows are exposed one at a time (e.g., time period 510). In the normal readout timing diagram 550, in the first row, the first row is exposed and then read out. After a certain delay (e.g., time period 510) following the start of the exposure of the first row, the second row is exposed and then read out (e.g., time period 520). The reading out of the second row can begin immediately after the reading out of the first row is completed.

[0095] For both normal readout timing diagram 550 and FSR mode timing diagram 560, the same amount of exposure time should be maintained for all rows throughout the entire image frame. This exposure time occurs only on one row at a time. It is desirable to maintain the same exposure time for each row; otherwise, the pixel dynamic range between rows in the rendered image may be inconsistent, which could lead to image artifacts. For this reason, the start of the first exposure time for row N+1 (e.g., row N+1) should be delayed by a certain amount of time after the start of the first exposure time for the previous row (e.g., row N). The start of the first exposure time for row N+1 (e.g., row N+1) should be delayed because the readout of the previous row (e.g., row N) has not yet been completed, since readout can only occur serially on the bus. Only one MIPI (e.g., Figure 8 The 820) will use the camera sensor (e.g., Figure 8 The sensor 810 is connected to the SoC (e.g., including the ISP, such as...) Figure 8 The inline image processor 830 is used, so the readout of sensor data is fully serialized on the bus. Therefore, the readout of row N plus one (e.g., row N+1) cannot be performed before the readout of the previous row (e.g., row N) is completed.

[0096] Since the readout of row N+1 (e.g., row N+1) can only occur after the readout of the previous row (e.g., row N) is complete, there is a certain offset at the start of the exposure time for each row (e.g., an increment of 580). The start of the exposure time for the second row is delayed by an offset from the start of the exposure time for the first row (e.g., an increment of 580), and the start of the exposure time for the third row is delayed by an offset from the start of the exposure time for the second row, and so on. As the entire image frame undergoes image processing, as can be seen in the normal readout timing diagram 550, there is a significant time lag between the start of the exposure time for the first row in the image frame and the start of the exposure time for the last row in the image frame (e.g., time interval 570a). The larger this time lag is, the more rolling shutter artifacts will be present in the rendered image.

[0097] For FSR mode timing diagram 560, the time lag between the start of exposure in the first row of an image frame and the start of exposure time in the last row of an image frame (e.g., time period 570b) is less than the time lag between the start of exposure in the first row of an image frame and the start of exposure time in the last row of an image frame (e.g., time period 570a) in normal mode timing diagram 550. The smaller this time lag (e.g., time period 570b), the lower the amount of rolling shutter artifacts in the rendered image.

[0098] However, while FSR mode allows for a reduction in the amount of rolling shutter artifacts, there are associated costs. Because FSR mode performs fast readout of sensor data, it requires increased MIPI signal and clock rates to accommodate this rapid readout. Currently, not all sensors support FSR mode. Typically, only expensive camera sensors support it. However, to reduce the amount of rolling shutter artifacts in rendered images and provide improved image quality (IQ), most high-end devices are currently moving towards supporting FSR mode.

[0099] exist Figure 5 In this context, when comparing the normal readout timing diagram 550 with the FSR mode timing diagram 560, it is evident that each of the gaps (e.g., gap 540b) in the aggregated readout energy 530b of the FSR mode timing diagram 560 is greater than each of the gaps (e.g., gap 540a) in the aggregated readout energy 530a of the normal mode timing diagram 550. Therefore, the FSR mode can have a larger vertical blanking period (e.g., one of the vertical blanking periods is gap 540b) than the normal mode (e.g., one of the vertical blanking periods is gap 540a).

[0100] As previously mentioned, FSR mode consumes less power than normal readout mode. FSR mode can lead to a reduction in the required sensor power (e.g., in some cases, FSR mode can allow for a sensor power saving of 160 to 260 milliwatts compared to normal readout mode). Therefore, from both an IQ and sensor power perspective, utilizing FSR mode for image processing instead of normal readout mode may be beneficial.

[0101] Figure 6 The power advantages of using the FSR mode are demonstrated. Specifically, Figure 6This is a graph 600 illustrating example power consumption for the entire camera module. On graph 600, the x-axis represents the readout rate per frame, and the y-axis represents the total sensor power. Graph 600 shows typical power consumption curves for different readout rates per frame in various camera scenarios, including single-camera, dual-camera, and triple-camera scenarios. For the slowest readout rate use case (e.g., a readout rate of 33 milliseconds per frame), the power consumption is shown as highest for all three camera scenarios. As the readout rate increases, for example, to a readout rate of 16.6 milliseconds per frame and 8.3 milliseconds per frame, the power consumption curves in graph 600 show a sharp decrease in power consumption.

[0102] Figure 7 The diagrams 710, 720, and 730 illustrate example power consumption at the camera's MIPI. Specifically, each of the diagrams 710, 720, and 730 illustrates an example power consumption waveform captured at the camera's MIPI. Each of the diagrams 710, 720, and 730 shows the power consumption waveform captured at the MIPI for different per-frame readout rates. For example, diagram 710 shows the power consumption waveform captured at the MIPI for a per-frame readout rate of 33 milliseconds, diagram 720 shows the power consumption waveform captured at the MIPI for a per-frame readout rate of 16.6 milliseconds, and diagram 730 shows the power consumption waveform captured at the MIPI for a per-frame readout rate of 8.3 milliseconds.

[0103] Curves 710, 720, and 730 show that the power consumption of the FSR mode is significantly lower (e.g., curve 730). Because the FSR mode allows for faster readouts, this results in significantly lower power consumption during the large vertical blanking cycle (e.g., Figure 5 During the gap 540b), camera components not used in the readout process are shut down. Because these camera components are in this large vertical blanking cycle (e.g., Figure 5 No power is used during the gap 540b, thus reducing the overall power consumption of the camera module.

[0104] Typically, camera sensors do not support any dynamic clock and voltage mechanisms. At least by increasing the system's clock rate and completing sensor data readout more quickly, there is no voltage loss on the sensor subsystem side; therefore, operation in FSR mode is desirable.

[0105] However, as previously mentioned, FSR mode can impact the power consumption of camera chipsets (e.g., SoCs), which may include an image sensor processor (ISP). For example, FSR mode may require the ISP to operate at very high clock rates and voltages to complete image frame processing during the compressed readout time of FSR mode. Furthermore, FSR mode may require memory (e.g., DDR memory) to operate at high clock rates to enable rapid output of data received from the image sensor processor.

[0106] Figure 8 The data stream from the camera is shown. Specifically, Figure 8 This is a diagram illustrating an example of a system 800 used to demonstrate a data stream from a camera. Figure 8 In the diagram, system 800 is shown to include sensor 810 (e.g., a camera sensor subsystem for acquiring image frames of a captured scene), inline image processor 830, offline image processor 850, and video processor 860. Sensor 810, inline image processor 830, offline image processor 850, and video processor 860 are all shown communicating with DDR memory 840.

[0107] During operation of system 800, sensor 810 can stream sensor data pixel by pixel to inline image processor 830 (e.g., an image front-end camera assembly, which may be a component in a SoC) via MIPI 820. After receiving pixels from sensor 810, inline image processor 830 can process pixels one row at a time (e.g., ...). Figure 5 (As shown). After the inline image processor 830 has processed all rows of the image frame, the inline image processor 830 can transfer the processed sensor data to the DDR memory 840. The processing performed by the inline image processor 830 is called inline processing because the inline image processor 830 processes pixels according to the operation of the sensor 810 (e.g., Figure 9 An example of inline processing is shown.

[0108] The timing of the inline image processor 830 needs to be strictly maintained because the operating timeline of the inline image processor 830 needs to correspond to the operating timeline of the sensor 810 (e.g., inline with it). Therefore, if the sensor 810 readout occurs within 8.3 milliseconds, the operation of the inline image processor 830 also needs to be completed within 8.3 milliseconds to complete the processing of all pixels it receives from the sensor 810, thus achieving full inlining.

[0109] In one or more examples, the output of the inline image processor 830 can be downscaled. For example, if the camera includes a 48-megapixel camera sensor, all 48 megapixel sensor data can be streamed to the inline image processor 830. The offline image processor 850 will eventually process all 48 megapixels; however, the video resolution itself may be much smaller. For example, the resulting video may have a Full High Definition (FHD) or Ultra High Definition (UHD) video resolution. The output of the inline image processor 830 can be downscaled and then transferred to DDR memory 840. The offline image processor 850 is able to operate at this downscaled resolution and continue the rest of the image processing.

[0110] Figure 9 These are graphs 910 and 920, each showing an example of an image frame's activity timeline. Figure 9 In graphs 910 and 920, the x-axis represents time, and the y-axis represents activity. Graph 910 shows ISP activity for three image frames. ISP activity includes the activity of the inline image processor 830 (e.g., box 940 of Figure 910) and the activity of the offline image processor 850 (e.g., activity box 930 of Figure 910). Graph 920 shows sensor activity (e.g., box 950) for the same three image frames, such as sensor activity of sensor 810.

[0111] exist Figure 9 In the diagram, the activity of the inline image processor 830 in graph 910 is shown as inline with the sensor activity in graph 920. The ISP front-end (e.g., the inline image processor 830) operating inline with the camera sensor (e.g., sensor 810) needs to run at a very high clock rate to support FSR mode. Running the ISP front-end at a very high clock rate can result in high power consumption, which can negatively impact the voltage of the shared trunk power supply for other cores of the SOC besides the ISP (e.g., camera components). Even if FSR mode reduces sensor power consumption, SOC power consumption may increase due to the high clock rate required for fast readout. Therefore, the SOC may experience power loss when operating in FSR mode.

[0112] This system and technology provide camera dynamic voting to optimize power in fast sensor modes (e.g., FSR mode power). In one or more aspects, this system and technology provide a dynamic voting mechanism (e.g., Figure 12 The dynamic clock voting engine 1250 optimizes the power consumption overhead of the camera chipset (e.g., SOC) during camera operation in FSR mode.

[0113] Figure 10 This is a graph 1000 illustrating the power consumption of the camera's chipset and camera module. In graph 1000, the x-axis represents the sensor readout rate, and the y-axis represents power in milliwatts. Graph 1000 illustrates the power consumption of the camera sensor (e.g., ...). Figure 8 The sensor power consumption curve 1030 of the sensor 810, and the SOC of the camera when the dynamic voting mechanism is not used (e.g., it may include...). Figure 8 The chipset power consumption curve 1010 for the inline graphics processor 830, and when a dynamic voting mechanism is used (e.g., Figure 12 The dynamic clock voting engine 1250) and the power consumption curve of the camera's SOC chipset 1020.

[0114] like Figure 10 As shown in sensor power consumption curve 1030, sensor power consumption improves with increasing sensor readout speed (e.g., for operating FSR mode). Conversely, chipset power consumption curve 1010 shows that chipset power consumption increases with increasing sensor readout speed (e.g., for operating FSR mode). However, compared to chipset power consumption curve 1010, chipset power consumption curve 1020 shows that chipset power consumption is reduced when a dynamic voting mechanism is used for fast sensor readout (e.g., when operating FSR mode), and therefore, the power loss of the chipset for operating at faster readout speeds is reduced.

[0115] In one or more aspects, to reduce power consumption losses in the chipset, the system and technology dynamically increase the ISP and DDR clock rates before the short sensor readout duration in the use case timeline, and immediately decrease the ISP and DDR clock rates after the short sensor readout duration is completed (e.g., to allow for a large blanking interval in the use case timeline, such as...). Figure 5 During the interval 540b), the clock rate is lower, during which no sensor readouts are performed. When the clock rate is adjusted in this way, the high power overhead requirement can be limited to the sensor readout portion of the use case timeline (e.g., which is typically only 25% to 50% of the use case timeline).

[0116] Most camera use cases operate at a specified image frame rate configured by advanced application software. This system and technology can employ configurable hardware timer mechanisms (e.g., Figure 12 The dynamic clock voting engine 1250 triggers a dynamic increase in clock rate and shared trunk voltage immediately before a new image frame is received from the image sensor, and a dynamic decrease in clock rate and shared trunk voltage immediately after image frame processing is completed.

[0117] Figure 11 An example timing diagram is shown for dynamically voting to increase and decrease the ISP and DDR clock rates and voltages. Specifically, Figure 11 This is a diagram illustrating an example of timing 1100 for a camera that employs a dynamic voting mechanism to optimize power in fast sensor modes. Figure 11 The fast sensor readout timing 1110 and inline ISP processing timing 1120 are shown in the diagram. For Figure 11 Employing a camera sensor (e.g., Figure 12 A dynamic voting mechanism (e.g., dynamic clock voting engine 1250) is used for inline operation of sensors 1210a to 1210b. The dynamic voting mechanism can increase or decrease the ISP and DDR clock rates and the associated trunk voltage levels.

[0118] Figure 11Timing 1100 illustrates the activity of FSR mode, which typically operates at a readout rate of 33 milliseconds per frame. A readout period 1140 for reading sensor data is shown (e.g., approximately eight milliseconds). A large vertical blanking period 1130 (e.g., approximately 24 milliseconds) that occurs when the camera sensor is not streaming any pixels to the inline image processor is also shown.

[0119] When readout cycle 1140 occurs, the dynamic voting mechanism can vote to increase the clock rate and increase the power supply to the camera front end (e.g., including an inline image processor, such as...). Figure 8 The voltage of the trunk powering the inline image processor 830. Then, when the vertical blanking cycle 1130 occurs, a dynamic voting mechanism can vote to reduce the clock rate and reduce the voltage of the shared trunk.

[0120] Dynamic voting mechanisms can be implemented in software, hardware, or a combination of both. In some cases, dynamic voting can be implemented as a hardware module without the involvement of any application processor. Therefore, any latency caused by voting to increase or decrease clock rate and power is negligible. Since latency is minimized, use case performance requirements can be met. In other cases, the application processor can participate in the dynamic voting mechanism to achieve latency savings to meet the timing shown in 1120. Figure 10 The power shown in 1020.

[0121] Timer mechanism (e.g., Figure 12One or more timers (1260) can be incorporated into the dynamic voting mechanism. Typically, the camera sensor operates at a specific frame rate, and the SOC is unaware of when new pixels will arrive from the camera sensor. Even before new pixels arrive from the camera sensor, the SOC (e.g., the inline image processor) must be prepared. To prepare the SOC for the new set of pixels from the camera sensor, the timer mechanism can start a timer (e.g., a voting timer 1150) that can be configured at the start of a frame but is set to expire at a time slightly below the frame period of the camera sensor. Configuring the timer's expiration time slightly below the frame period allows the timer to start before the new frame readout begins, making the SOC ready to set the higher clock and voltage required for inline pixel processing. Once the SOC receives the first set of pixels for the new frame (at point 1), the voting timer is set for the next frame. After inline processing is complete (at point 2), the dynamic voting mechanism can quickly vote to reduce the clock rate and voltage without losing any time. The voting timer fires at point 4 (before the next frame arrives from the sensor), allowing the SOC to vote to increase the clock rate before the frame. SOC power can be optimized by reducing the shared trunk power (between points 3 and 4). In this way, a dynamic voting mechanism (e.g., by voting on increases and decreases in clock rate and voltage) can allow for reduction of power loss in the chipset (e.g., SOC).

[0122] Figure 12 This is a diagram illustrating an example of a system 1200 that employs a dynamic voting mechanism (e.g., a dynamic clock voting engine 1250) to optimize power in fast sensor modes. Figure 12 In this diagram, system 1200 is shown to include one or more sensors 1210a to 1210b (e.g., one or more sensors, such as camera or image sensors), ISPs 1230a to 1230b (e.g., one or more ISPs), memory 1240, a dynamic clock voting engine 1250, one or more timers 1260 (in some cases, multiple timers), a system resource voting and aggregation engine 1270, and a clock and voltage control unit 1290 (e.g., core / DDR clock control and power supply line voltage control). The dynamic clock voting engine 1250, the system resource voting and aggregation engine 1270, and / or the clock and voltage control unit 1290 can be implemented in software, hardware, or a combination of software and hardware. Furthermore, additional sensors in system 1200 are optional, as indicated by the dashed outline of the frame of sensor 1210b. For example, in some aspects, system 1200 may include a single sensor 1210a (in which case, additional sensors including sensor 1210b are not included in system 1200 or are not used by the system). In other respects, system 1200 may include multiple sensors, including sensors 1210a to 1210b (wherein) Figure 12 In the context of "sensor (N)", "N" is an integer greater than or equal to 2. Furthermore, in some aspects, system 1200 may include a single ISP 1230a (in which case, additional ISPs including ISP 1230b are not included in system 1200 or are not used by the system). In other aspects, system 1200 may include multiple ISPs, including ISPs 1230a to ISP 1230b (where... Figure 12 In "ISP(N)", "N" is an integer greater than or equal to 2.

[0123] In one or more examples, sensors 1210a to 1210b may be RGB camera sensors. In some examples, each sensor 1210a to 1210b is associated with a corresponding camera (such as an RGB camera). ISPs 1230a to 1230b may each include an inline image processor (e.g., Figure 8 The inline image processor 830). Each ISP 1230a to ISP 1230b may be associated with a corresponding sensor 1210a to sensor 1210b. The memory 1240 may be a DDR memory (e.g., Figure 8 (DDR memory 840). Each timer in timer 1260 can support a corresponding camera, and each timer is configured to trigger a vote (increment) before a new image frame is streamed from sensor 1210a to sensor 1210b. In some cases, system 1200 includes multiple timers 1260 to support multiple cameras.

[0124] exist Figure 12 During the operation of system 1200, sensors 1210a to 1210b can acquire image frames by capturing the scene. When sensors 1210a to 1210b begin streaming the pixels of the image frames to their associated ISPs 1230a to 1230b, timer 1260 can start a timer (e.g., voting timer 1150) capable of running for a period slightly lower than the frame time of the camera sensor to trigger dynamic clock voting engine 1250 to vote to increase the clock rate and trunk voltage before the frame. After dynamic clock voting engine 1250 is triggered to increase the vote, it can submit (e.g., send) a vote of approval to system resource voting and aggregation engine 1270.

[0125] The system resource voting and aggregation engine 1270 can receive multiple votes 1280 from multiple components of the camera, including a dynamic clock voting engine 1250. These multiple components can be within the camera SOC and can be powered by the same (or multiple) trunks or different trunks. The multiple votes 1280 can include affirmative votes from the dynamic clock voting engine 1250.

[0126] The system resource voting and aggregation engine 1270 (which can support multiple clients) receives multiple votes 1280 and maintains the current voting state of each client among its clients, as well as the logic for increasing and decreasing client clocks and shared resources (e.g., DDR and power line voltages). When the system resource voting and aggregation engine 1270 determines that at least one client requires a higher operating clock and voltage on the shared line, the system resource voting and aggregation engine 1270 may transmit control signals to the clock and voltage control unit 1290 to increase the clock rate and / or increase the voltage on one or more shared lines.

[0127] After receiving a control signal to increase the clock rate and voltage, the clock and voltage control unit 1290 may first increase the voltage of the trunk line, and then increase the clock rate (e.g., to a higher megahertz frequency). After the clock and voltage control unit 1290 has increased the clock rate (e.g., to a higher megahertz frequency), the increased clock rate may be applied to ISPs 1230a to ISP 1230b and memory 1240.

[0128] After the clock rates of ISPs 1230a to ISP 1230b (e.g., one or more ISPs) and memory 1240 have been increased, sensors 1210a to ISP 1210b (e.g., one or more sensors) can stream pixels of image frames to their respective ISPs 1230a to ISP 1230b via ISP sensor interface 1220. Upon receiving the pixels of the image frame, ISPs 1230a to ISP 1230b can process these pixels (e.g., perform inline image processing on the pixels). After ISPs 1230a to ISP 1230b have processed the pixels of the image frame, they can transmit the processed sensor data (e.g., send it) to memory 1240. Furthermore, after ISPs 1230a to ISP 1230b have processed the pixels of the image frame, the dynamic clock voting engine 1250 can submit (e.g., send) dissenting votes (e.g., for reducing clock rate and trunk voltage) to the system resource voting and aggregation engine 1270.

[0129] The system resource voting and aggregation engine 1270 can receive additional votes 1280 from multiple components of the camera, including a dynamic clock voting engine 1250. The multiple votes 1280 may include dissenting votes from the dynamic clock voting engine 1250.

[0130] After the system resource voting and aggregation engine 1270 receives multiple votes 1280, if it determines that no client needs a higher clock rate and voltage on the shared trunk, it can send a control signal to the clock and voltage control unit 1290 to reduce the clock rate and the voltage on the shared trunk.

[0131] After receiving a control signal to reduce the clock rate and voltage, the clock and voltage control unit 1290 may first reduce the clock rate (e.g., to a higher megahertz frequency), and then reduce the voltage on one or more trunk lines. After the clock and voltage control unit 1290 has reduced the clock rate (e.g., to a lower megahertz frequency), the reduced clock rate may be applied to ISPs 1230a to ISP 1230b and memory 1240.

[0132] After a period of time (e.g., the vertical blanking time period), system 1200 will, according to... Figure 11 The timing sequence 1100 shown repeats the previously described operation.

[0133] Figure 13 Table 1300 shows examples of power savings when dynamic voting is used to optimize power in fast sensor modes. Figure 13 Table 1300 is shown as comprising four columns, including a component list 1310 (e.g., for a UHD30 camera), an operational parameter list for a non-FSR mode use case 1320 (e.g., normal readout mode), an operational parameter list for an FSR mode use case 1330 without dynamic voting, and a parameter list for an FSR mode use case 1340 with dynamic voting. Figure 13 As shown in Table 1300, the FSR mode use case 1340 with dynamic voting achieves a 12% power saving compared to the non-FSR mode use case 1320. The FSR mode use case 1330 without dynamic voting saves only 5.8% power compared to the non-FSR mode use case 1320. Therefore, employing dynamic voting with FSR mode can significantly reduce chipset power loss when operating in FSR mode.

[0134] Figure 14This is a flowchart illustrating an example of a process 1400 for camera dynamic voting to optimize power in fast sensor modes. Process 1400 may be performed by a computing device or system, or by a component or system of that computing device or system (e.g., a chipset). In some aspects, process 1400 may be performed by… Figure 12 The process 1400 may be executed by system 1200, or by a computing device including system 1200 (e.g., mobile device, camera device, extended reality (XR) device, laptop or desktop computer, vehicle or computing device of vehicle, etc.). The operation of process 1400 may be implemented on one or more processors (e.g., Figure 8 One or more inline image processors 830 and / or offline image processors 850, Figure 12 One or more ISP 1230a to ISP 1230b or other components, Figure 15 Software components that execute and run on processors 1510 and / or other processors.

[0135] At box 1410, a computing device or system (or a component thereof) may obtain multiple votes associated with multiple components sharing a power supply based on performing dynamic voting. Dynamic voting may be performed as described herein, such as regarding... Figure 12 As described. For example, in some aspects, in order to obtain multiple votes based on performing dynamic voting, a computing device or system (or a component thereof) may perform dynamic voting to generate a vote for at least one current image frame (e.g., using a dynamic clock voting engine 1250). The computing device or system (or a component thereof) may obtain a vote generated for at least one previous image frame, and may (e.g., using a system resource voting and aggregation engine 1270) aggregate that vote and these votes to obtain multiple votes. In some cases, the computing device or system (or a component thereof) may trigger dynamic voting (e.g., based on a timer, such as timer 1260).

[0136] At box 1420, the computing device or system (or its components) may determine the voting result based on the multiple votes (e.g., using a dynamic clock voting engine 1250 and / or a system resource voting and aggregation engine 1270). In some cases, the power supply shared by multiple components is or includes one or more power trunks.

[0137] At box 1430, a computing device or system (or a component thereof) may increase or decrease (e.g., using clock and voltage control unit 1290) the clock rate and voltage of the power supply based on the voting result to produce an updated clock rate and updated voltage.

[0138] At box 1440, a computing device or system (or a component thereof) may apply the updated clock rate and the updated voltage to an image processor (e.g., ISP 1230a or multiple ISPs, such as ISP 1230a through ISP 1230b). In some cases, the computing device or system (or a component thereof) may include an image processor (e.g., ISP 1230a or multiple ISPs, such as ISP 1230a through ISP 1230b). In some aspects, the image processor is a front-end component of a camera. In some cases, the image processor is an inline image processor. In some examples, the image processor is an image signal processor (ISP). In some cases, the image processor and the multiple components are on a system-on-a-chip (SoC).

[0139] In some cases, a computing device or system (or its components) may acquire image frames of a captured scene via a sensor (e.g., sensor 1210a or from multiple sensors, such as sensor 1210a to sensor 1210b). In some cases, a computing device or system (or its components) may include a sensor (e.g., sensor 1210a or multiple sensors, such as sensor 1210a to sensor 1210b). In some aspects, the image processor operates inline with the sensor in a timing manner, as described herein. In some cases, the computing device or system (or its components) may (e.g., by the sensor) output pixels of an image frame to an image processor (e.g., sensor 1210a may output pixels of an image frame to a corresponding ISP 1230a via ISP sensor interface 1220, and sensors 1210a to sensor 1210b may stream pixels of an image frame to their respective ISPs 1230a to ISP 1230b via ISP sensor interface 1220, etc.). A computing device or system (or its components) may process (e.g., using an image processor such as ISP 1230a) image frames to produce processed image data. In some cases, the computing device or system (or its components) may use ISP 1230a to ISP 1230b to process multiple image frames from sensors 1210a to 1210b. In some cases, the computing device or system (or its components) may transmit, output, or otherwise provide the processed image data to memory (e.g., double data rate (DDR) memory or other types of memory).

[0140] As described above, process 1400 can be executed by one or more computing devices or apparatuses. In some exemplary examples, process 1400 can be performed by... Figure 1 Image capture and processing system 100 Figure 3 Equipment 300 Figure 12 System 1200 and / or one or more computing devices or systems (e.g., Figure 15The computing system 1500 executes the process. In some cases, such a computing device or apparatus may include a processor, microprocessor, microcomputer, or other components of a device configured to perform the steps of process 1400. In some examples, such a computing device or apparatus may include one or more sensors configured to capture image data. For example, the computing device may include a smartphone, head-mounted display, mobile device, camera, tablet computer, or other suitable device. In some examples, such a computing device or apparatus may include a camera configured to capture one or more images or videos. In some cases, such a computing device may include a display for displaying images. In some examples, one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such a computing device may further include a network interface configured to transmit data.

[0141] Components that enable the implementation of a computing device in a circuit. For example, a component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein. The computing device may also include a display (as an example of an output device or as a supplement to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0142] Process 1400 is illustrated as a logic flowchart, the operations of which represent a sequence of operations that can be implemented by hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0143] Additionally, process 1400 can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0144] Figure 15 This is a diagram illustrating an example of a system used to implement certain aspects of this technology. Specifically, Figure 15 An example of a computing system 1500 is illustrated. This computing system can be any computing device, such as constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1505. Connection 1505 can be a physical connection using a bus, or a direct connection to processor 1510, such as in a chipset architecture. Connection 1505 can also be a virtual connection, a networking connection, or a logical connection.

[0145] In some embodiments, computing system 1500 is a distributed system, wherein the functions described herein may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more system components described represent a plurality of such components that each perform some or all of the functions described for which the component is used. In some embodiments, components may be physical devices or virtual devices.

[0146] Example system 1500 includes at least one processing unit (CPU or processor) 1510 and a connection 1505 that couples various system components, including memory units 1515 such as read-only memory (ROM) 1520 and random access memory (RAM) 1525, to processor 1510. Computing system 1500 may include a cache 1512 of high-speed memory that is directly connected to, closely adjacent to, or integrated into processor 1510.

[0147] Processor 1510 may include any general-purpose processor and hardware or software services, such as services 1532, 1534, and 1536 stored in storage device 1530, which are configured to control processor 1510 and dedicated processors in which software instructions are incorporated into the actual processor design. Processor 1510 may be a substantially completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0148] To enable user interaction, the computing system 1500 includes an input device 1545 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. The computing system 1500 may also include an output device 1535 that can be one or more of a plurality of output mechanisms. In some instances, a multi-mode system allows a user to provide multiple types of input / output to communicate with the computing system 1500. The computing system 1500 may include a communication interface 1540, which typically governs and manages user input and system output. The communication interface can perform or facilitate the receipt and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple... ® Lightning ® Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, BLUETOOTH ® Wireless signal transmission, Bluetooth ® Low-power (BLE) wireless signal transmission, IBEACON ® The communication interface 1540 may include one or more Global Navigation Satellite System (GNSS) receivers or transceivers for determining the location of the computing system 1500 based on one or more signals received from one or more satellites associated with one or more GNSS systems. This includes wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC) wireless signal transmission, microwave access global interoperability (WiMAX) wireless signal transmission, infrared (IR) wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. GNSS systems include, but are not limited to, the U.S. Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There are no limitations on operation on any particular hardware configuration, and therefore the underlying features here can be easily replaced to obtain improved hardware or firmware configurations as they are developed.

[0149] Storage device 1530 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape cassettes, flash memory cards, solid-state storage devices, digital multifunction disks, cassette tapes, floppy disks, flexible disks, hard disks, magnetic tapes, magnetic stripes / strips, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM, rewritable CD, digital video disc (DVD), Blu-ray disc (BDD), holographic disc, another optical medium, secure digital (SD) cards, micro-secure digital (microSD) cards, Memory Stick. ® Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistive random access memory (RRAM / ReRAM), phase-change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette and / or combinations thereof.

[0150] Storage device 1530 may include software services, servers, services, etc., which enable the system to perform functions when the code defining such software is executed by processor 1510. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in computer-readable media connected to necessary hardware components such as processor 1510, connection 1505, output device 1535, etc.

[0151] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and which do not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted using any suitable means, including memory sharing, messaging, token passing, network transmission, etc.

[0152] In some implementations, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as power consumption, carrier signals, electromagnetic waves, and the signals themselves.

[0153] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples presented herein. However, those skilled in the art will understand that embodiments can be practiced without these specific details. For clarity, in some cases, the technology may be presented as comprising individual functional blocks, including functional blocks containing devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid these embodiments becoming obscure with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures and techniques may be shown without necessary detail to avoid obscuring the embodiments.

[0154] Individual implementations may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although flowcharts may describe operations as sequential processes, many operations within an operation may be executed in parallel or concurrently. Furthermore, the order of operations may be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or the main function.

[0155] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. Computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical discs, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0156] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor performs the necessary tasks. Typical examples of form factors include laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0157] Instructions, media for delivering such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0158] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, although exemplary embodiments of this application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in a variety of other ways, and the appended claims are intended to be construed as including such variations, unless limited by prior art. Various features and aspects of the applications described above may be used individually or in combination. Furthermore, without departing from the broader spirit and scope of this specification, the embodiments can be used in any number of environments and applications beyond those described herein. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in a different order than described.

[0159] Those skilled in the art will understand that, without departing from the scope of this description, the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced with less than or equal to (“>”) respectively. ") and greater than or equal to (" The symbol ) is used instead.

[0160] When a component is described as being “configured” to perform certain operations, such a configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0161] The phrase “coupled to” means any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0162] The claim language or other language that states "at least one of" and / or "one or more of" in a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, the claim language stating "at least one of A and B" means A, B, or A and B. In another example, the claim language stating "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the claim language stating "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0163] The various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been broadly described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0164] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0165] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined system or component (e.g., a system-on-a-chip).

[0166] The exemplary aspects of this disclosure include:

[0167] Aspect 1. A method for processing image data, the method comprising: obtaining multiple votes associated with multiple components sharing a power supply based on performing dynamic voting; determining a voting result based on the multiple votes; increasing or decreasing a clock rate and voltage of the power supply based on the voting result to generate an updated clock rate and an updated voltage; and applying the updated clock rate and the updated voltage to an image processor.

[0168] Aspect 2. The method according to aspect 1, wherein obtaining the plurality of votes based on performing dynamic voting includes: performing the dynamic voting to generate a vote for at least one current image frame; obtaining a vote generated for at least one previous image frame; and aggregating the vote for at least one current image frame and the vote generated for at least one previous image frame to obtain the plurality of votes.

[0169] Aspect 3. The method according to any one of Aspect 1 or 2, the method further comprising triggering the dynamic voting.

[0170] Aspect 4. The method according to aspect 3, wherein the triggering of the dynamic voting is based on a timer.

[0171] Aspect 5. The method according to any one of aspects 1 to 4, the method further comprising obtaining image frames of the captured scene by a sensor.

[0172] Aspect 6. The method according to aspect 5, wherein the image processor and the sensor are inter-inlined in time.

[0173] Aspect 7. The method according to any one of Aspects 5 or 6, the method further comprising outputting pixels of the image frame to the image processor by the sensor.

[0174] Aspect 8. The method according to any one of Aspects 5 to 7, the method further comprising processing the image frame by the image processor to generate processed image data.

[0175] Aspect 9. The method according to aspect 8, the method further comprising sending the processed image data to a memory.

[0176] Aspect 10. The method according to aspect 9, wherein the memory is a double data rate (DDR) memory.

[0177] Aspect 11. The method according to any one of Aspects 1 to 10, wherein the image processor is a front-end component of a camera.

[0178] Aspect 12. The method according to any one of Aspects 1 to 11, wherein the image processor is an inline image processor.

[0179] Aspect 13. The method according to any one of Aspects 1 to 12, wherein the image processor is an image signal processor (ISP).

[0180] Aspect 14. The method according to any one of Aspects 1 to 13, wherein the image processor and the plurality of components are on a system-on-a-chip (SOC).

[0181] Aspect 15. The method according to any one of Aspects 1 to 14, wherein the power source shared by the plurality of components is one or more power trunks.

[0182] Aspect 16. An apparatus for processing image data, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: obtain multiple votes associated with multiple components sharing a power supply based on performing dynamic voting; determine a voting result based on the multiple votes; increase or decrease a clock rate and voltage of the power supply based on the voting result to generate an updated clock rate and an updated voltage; and apply the updated clock rate and the updated voltage to the image processor.

[0183] Aspect 17. The apparatus according to aspect 16, wherein, in order to obtain the plurality of votes based on the execution of the dynamic voting, the at least one processor is configured to: execute the dynamic voting to generate a vote for at least one current image frame; obtain a vote generated for at least one previous image frame; and aggregate the vote for at least one current image frame and the vote generated for at least one previous image frame to obtain the plurality of votes.

[0184] Aspect 18. The apparatus according to any one of Aspects 16 or 17, wherein the at least one processor is configured to trigger the dynamic voting.

[0185] Aspect 19. The apparatus according to any one of Aspects 16 to 18, wherein the at least one processor is configured to trigger the dynamic voting based on a timer.

[0186] Aspect 20. The apparatus according to any one of aspects 16 to 19, the apparatus further comprising: a sensor configured to capture image frames of the scene.

[0187] Aspect 21. The apparatus according to aspect 20, wherein the sensor is configured to output pixels of the image frame to the image processor.

[0188] Aspect 22. The apparatus according to any one of Aspects 20 or 21, the apparatus further comprising the image processor, wherein the image processor is sequentially inlined with the sensor.

[0189] Aspect 23. The apparatus according to any one of aspects 20 to 22, wherein the image processor is configured to process the image frame to generate processed image data.

[0190] Aspect 24. The apparatus according to aspect 23, wherein the at least one processor is configured to send the processed image data to the at least one memory.

[0191] Aspect 25. The apparatus according to aspect 24, wherein the at least one memory is a double data rate (DDR) memory.

[0192] Aspect 26. The apparatus according to any one of aspects 16 to 25, wherein the image processor is a front-end component of a camera.

[0193] Aspect 27. The apparatus according to any one of aspects 16 to 26, wherein the image processor is an inline image processor.

[0194] Aspect 28. The apparatus according to any one of aspects 16 to 27, wherein the image processor is an image signal processor (ISP).

[0195] Aspect 29. The apparatus according to any one of Aspects 16 to 28, wherein the image processor and the plurality of components are on a system-on-a-chip (SoC).

[0196] Aspect 30. The apparatus according to any one of aspects 16 to 29, wherein the power source shared by the plurality of components is one or more power trunks.

[0197] Aspect 31. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform any one of aspects 1 to 15.

[0198] Aspect 32. An apparatus for processing image data, the apparatus comprising one or more components for performing operations according to any one of aspects 1 to 15.

[0199] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are to be consistent with the full scope of the language claims, wherein an element referred to in the singular is not intended to mean "one and only one," but rather "one or more" unless specifically stated otherwise.

Claims

1. A method for processing image data, the method comprising: detecting a start of a readout period of an image sensor; obtaining a first plurality of votes associated with a plurality of components sharing a power supply based on the detected start of the readout period of the image sensor; increasing a clock rate and a voltage of the power supply based on the first plurality of votes to produce an updated clock rate and an updated voltage for an image processor; and applying the updated clock rate and the updated voltage to the image processor to process frame data output during the readout period of the image sensor.

2. The method of claim 1, wherein obtaining the first plurality of votes comprises: generating a vote for at least one current image frame; obtaining a vote generated for at least one previous image frame; and aggregating the vote for at least one current image frame and the vote generated for at least one previous image frame to obtain the first plurality of votes.

3. The method of claim 1, further comprising obtaining the first plurality of votes based on a timer.

4. The method of claim 1, further comprising obtaining, by the image sensor, an image frame capturing a scene.

5. The method of claim 4, wherein the image processor is inline in timing with the image sensor.

6. The method of claim 1, further comprising processing, by the image processor, the frame data to produce processed image data.

7. The method of claim 6, further comprising outputting the processed image data to a memory.

8. The method of claim 7, wherein the memory is a double data rate (DDR) memory.

9. The method of claim 1, wherein the image processor is a front-end component of a camera.

10. The method of claim 1, wherein the image processor is an inline image processor.

11. The method of claim 1, wherein the image processor is an image signal processor (ISP).

12. The method of claim 1, wherein the image processor and the plurality of components are on a system on a chip (SOC).

13. The method of claim 1, wherein the power supply shared by the plurality of components is one or more power rails.

14. The method of claim 1, further comprising: applying a reduced clock rate and a reduced voltage to the image processor based on an end of processing of the frame data by the image processor for operation during a blanking interval of the image sensor.

15. The method of claim 1, further comprising: obtaining a second plurality of votes associated with the plurality of components sharing a power supply based on an end of processing of the frame data by the image processor; reducing the clock rate and the voltage of the power supply based on the second plurality of votes to produce a further updated clock rate and a further updated voltage for the image processor; and applying the further updated clock rate and the further updated voltage to the image processor for operation during a blanking interval of the image sensor.

16. An apparatus for processing image data, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: detect a start of a readout period of an image sensor; obtain a first plurality of votes associated with a plurality of components sharing a power supply based on the detected start of the readout period of the image sensor; increase a clock rate and a voltage of the power supply based on the first plurality of votes to produce an updated clock rate and an updated voltage for an image processor; and apply the updated clock rate and the updated voltage to the image processor to process frame data output during the readout period of the image sensor.

17. The apparatus of claim 16, wherein to obtain the first plurality of votes, the at least one processor is configured to: generate a vote for at least one current image frame; obtain a vote generated for at least one previous image frame; and aggregate the vote for at least one current image frame and the vote generated for at least one previous image frame to obtain the first plurality of votes.

18. The apparatus of claim 16, wherein the at least one processor is configured to obtain the first plurality of votes based on a timer.

19. The apparatus of claim 16, further comprising the image sensor.

20. The apparatus of claim 19, further comprising the image processor, wherein the image processor is inline in timing with the image sensor.

21. The apparatus of claim 16, wherein the image processor is configured to process the frame data to produce processed image data.

22. The apparatus of claim 21, wherein the at least one processor is configured to output the processed image data to the at least one memory.

23. The apparatus of claim 22, wherein the at least one memory is a double data rate (DDR) memory.

24. The apparatus of claim 16, wherein the image processor is a front-end component of a camera.

25. The apparatus of claim 16, wherein the image processor is an inline image processor.

26. The apparatus of claim 16, wherein the image processor is an image signal processor (ISP).

27. The apparatus of claim 16, wherein the image processor and the plurality of components are on a system on a chip (SOC).

28. The apparatus of claim 16, wherein the power supply shared by the plurality of components is one or more power rails.

29. The apparatus of claim 16, wherein the at least one processor is configured to: apply a reduced clock rate and a reduced voltage to the image processor for operation during a blanking interval of the image sensor based on an end of processing of the frame data by the image processor.

30. The apparatus of claim 16, wherein the at least one processor is configured to: obtain a second plurality of votes associated with the plurality of components sharing a power supply based on an end of processing of the frame data by the image processor; reduce the clock rate and the voltage of the power supply based on the second plurality of votes to produce a further updated clock rate and a further updated voltage for the image processor; and apply the further updated clock rate and the further updated voltage to the image processor for operation during a blanking interval of the image sensor.

30. The apparatus of claim 16, wherein the at least one processor is configured to: obtain a second plurality of votes associated with the plurality of components sharing a power supply based on an end of processing of the frame data by the image processor; reduce the clock rate and the voltage of the power supply based on the second plurality of votes to produce a further updated clock rate and a further updated voltage for the image processor; and apply the further updated clock rate and the further updated voltage to the image processor for operation during a blanking interval of the image sensor.

Images