Dynamic resource voting (DRV) enhancements for dynamic frames per second (FPS) use cases

Abstract

Systems, apparatuses, processes, and computer-readable media for image processing are disclosed. For example, a computing device can output (e.g., send) a timing indicator for synchronization of a voting engine and a frame rate for processing image frames obtained by a sensor. The computing device can obtain a positive vote result from the voting engine based on the timing indicator. The computing device can increase a clock rate and a voltage of a power supply shared by an image processor and at least one memory based on the positive vote result to produce an updated clock rate and an updated voltage. The computing device can apply the updated clock rate and the updated voltage to the image processor and the at least one memory.

Description

Technical Field

[0001] This disclosure relates in general to image processing. For example, aspects of this disclosure relate to dynamic resource voting (DRV) enhancements for dynamic frames per second (FPS) use cases. Background Technology

[0002] The increasing versatility of digital cameras has allowed them to be integrated into a wide variety of devices, expanding their applications. For example, telephones, cars, computers, televisions, and many other devices today are frequently equipped with camera devices. Camera devices allow users to capture images and / or video (e.g., frames of an image) from any system equipped with them. Images and / or video can be captured for entertainment, professional photography, surveillance, automation, and other applications. Furthermore, camera devices are increasingly equipped with specific functionalities for modifying images or creating artistic effects on them. For example, many camera devices are equipped with image processing capabilities for generating different effects on captured images.

[0003] For image processing, Fast Sensor Readout (FSR) is a common sensor operating mode used by many original equipment manufacturers (OEMs) to improve image quality (IQ) because FSR results in a reduction of shutter-related artifacts in the image. Compared to normal readout mode, FSR mode allows for faster readout of image sensor data from image frames while maintaining the same amount of exposure time. The faster the readout of image sensor data, the lower the amount of shutter-related artifacts present in the rendered image. FSR mode also allows for a reduction in required sensor power (e.g., in some cases, FSR mode can allow for 160 to 260 milliwatts of sensor power savings compared to normal readout mode). Therefore, from both an IQ and sensor power perspective, using FSR for image processing can be advantageous. Summary of the Invention

[0004] The following is a simplified summary of the invention relating to one or more aspects disclosed herein. Therefore, this summary should not be considered an exhaustive overview relating to all conceived aspects, nor should it be considered to identify key or decisive elements relating to all conceived aspects or to depict the scope associated with any particular aspect. Thus, the sole purpose of this summary is to present, in a simplified form, certain concepts relating to one or more aspects involving the mechanisms disclosed herein, prior to the detailed description presented below.

[0005] Systems, apparatuses, methods, and computer-readable media for dynamic resource voting (DRV) enhancements for dynamic frames per second (FPS) use cases are disclosed. According to at least one exemplary example, an apparatus for image processing is provided. The apparatus includes at least one memory and at least one processor coupled to the at least one memory and configured to: output a timing indicator for synchronizing a voting engine and a frame rate used to process image frames acquired by a sensor; obtain a positive voting result from the voting engine based on the timing indicator; increase the clock rate and voltage of a power supply shared by the image processor and the at least one memory based on the positive voting result to generate an updated clock rate and updated voltage; and apply the updated clock rate and updated voltage to the image processor and the at least one memory.

[0006] In another exemplary example, a method for image processing is provided. The method includes: sending a timing indicator from a sensor to a voting engine for synchronization of the voting engine and a frame rate used to process image frames acquired by the sensor; the voting engine determining a positive voting result based on the timing indicator; increasing the clock rate and voltage of a power supply shared by an image processor and a memory based on the positive voting result to generate an updated clock rate and updated voltage; and applying the updated clock rate and updated voltage to the image processor and the memory.

[0007] In another exemplary example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: output a timing indicator for synchronizing a voting engine and a frame rate for processing image frames acquired by a sensor; obtain a positive voting result from the voting engine based on the timing indicator; increase the clock rate and voltage of a power supply shared by an image processor and a memory based on the positive voting result to produce an updated clock rate and updated voltage; and apply the updated clock rate and updated voltage to the image processor and the at least one memory.

[0008] In another exemplary example, an apparatus for image processing is provided. The apparatus includes: components for transmitting a timing indicator for synchronizing a voting engine and a frame rate for processing image frames acquired by a sensor; components for determining a positive voting result based on the timing indicator; components for increasing the clock rate and voltage of a power supply shared by an image processor and a memory based on the positive voting result to generate an updated clock rate and updated voltage; and components for applying the updated clock rate and updated voltage to the image processor and the memory.

[0009] The aspects generally include, as described substantially with reference to the accompanying drawings and description and illustrated as shown in the drawings and description, methods, apparatus, systems, computer program products, non-transitory computer-readable media, user equipment, user gear, wireless communication equipment, and / or processing systems.

[0010] In some aspects, one or more of the devices described herein are mobile devices, smart or connected devices, camera systems, and / or extended reality (XR) devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), and may be part of or include such devices. In some examples, the device may include a vehicle, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotic device or system, an aviation system, or other equipment, or be part of such equipment. In some aspects, the device may include one image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device may include one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device may include one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the device may include one or more sensors. In some cases, one or more sensors may be used to determine the location of the device, the state of the device (e.g., tracking state, operating state, temperature, humidity level, and / or other states), and / or for other purposes.

[0011] Some aspects include a device having a processor configured to perform one or more operations of any of the methods outlined above. Further aspects include a processing device for use in the device, configured using processor-executable instructions to perform operations of any of the methods outlined above. Further aspects include a non-transitory processor-readable storage medium storing processor-executable instructions thereon configured to cause the device's processor to perform operations of any of the methods outlined above. Further aspects include a device having components for performing functions of any of the methods outlined above.

[0012] The features and technical advantages of the examples according to this disclosure have been summarized quite extensively above in order to better understand the detailed description below. Additional features and advantages will be described below. The disclosed concepts and specific examples can be readily utilized as the basis for modifying or designing other structures for achieving the same purpose of this disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The characteristics of the concepts disclosed herein (both their organization and operation) and their associated advantages will be better understood from the following description when considered in conjunction with the accompanying drawings. Each figure in the drawings is provided for illustrative and descriptive purposes and not as a definition of limitation of the claims. The foregoing, as well as other features and aspects, will become more apparent upon reference to the following specification, claims, and appended drawings.

[0013] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to define the scope of the claimed subject matter. This subject matter should be understood with reference to the appropriate portions of the entire specification, any or all drawings, and each claim.

[0014] Based on the accompanying drawings and detailed description, other objects and advantages associated with the aspects disclosed herein will be apparent to those skilled in the art. Attached Figure Description

[0015] The exemplary aspects of this application are described in detail below with reference to the following figures:

[0016] Figure 1 This is a block diagram illustrating an example architecture of an image capture and processing system based on some examples.

[0017] Figure 2 This is a block diagram illustrating examples of interactions between components of an image capture and processing system, based on some examples.

[0018] Figure 3 This is a block diagram of an example device that can be used for camera dynamic voting to optimize power in fast sensor modes, based on some examples.

[0019] Figure 4 This is a block diagram illustrating the operation of an image signal processor pipeline based on some examples.

[0020] Figure 5 This is an illustration illustrating an example of timing based on some sample camera sensors.

[0021] Figure 6 This is an illustration of an example of timing using a camera in Fast Readout Sensor (FRS) mode, based on some examples.

[0022] Figure 7This is a diagram illustrating an example of using Dynamic Resource Voting (DRV) for camera timing in an FRS use case, based on some examples.

[0023] Figure 8 This is a diagram illustrating an example of a system that uses DRV for Fast Sensor Readout (FSR) based on some examples.

[0024] Figure 9 This is a diagram illustrating an example of using DRV for camera timing in an FRS use case, based on some examples, where the DRV timer value and frame rate are synchronized.

[0025] Figure 10 This is a diagram illustrating an example of using DRV for camera timing in an FRS use case, based on some examples, where the DRV timer value and frame rate are asynchronous.

[0026] Figure 11 This is a diagram illustrating an example of using DRV for camera timing in an FRS use case, based on some examples, where timing indicators are employed.

[0027] Figure 12 This is an illustration of an example system that uses DRV for FSR for the Dynamic Frames Per Second (FPS) use case, based on some examples, where the system employs timing including timing indicators.

[0028] Figure 13A This is an illustration of an example of a timing indicator for a camera using DRV for FSR in dynamic FPS use cases, based on some examples, where the timing indicator appears at a time before the start of an image frame.

[0029] Figure 13B This is an illustration of an example of a timing indicator for a camera using DRV for FSR in dynamic FPS use cases, based on some examples, where the timing indicator appears after the start of an image frame.

[0030] Figure 14A This is a graph illustrating different power requirements for different sensor readout rates when using DRV for FSR in dynamic FPS use cases, based on some examples.

[0031] Figure 14B It is a summary based on some examples. Figure 14A A table showing the different power requirements for different sensor readout speeds.

[0032] Figure 15 This is a flowchart illustrating an example of the DRV process for FSR for dynamic FPS use cases, based on some examples.

[0033] Figure 16 This is a diagram illustrating an example of a system used to implement some of the aspects described in this article. Detailed Implementation

[0034] Certain aspects of this disclosure are provided below for illustrative purposes. Alternative aspects may be devised without departing from the scope of this disclosure. Additionally, well-known elements of this disclosure will not be described in detail or will be omitted so as not to obscure the relevant details of this disclosure. Some aspects described herein can be applied independently, and some of them can be combined, as will be apparent to those skilled in the art. Specific details are set forth in the following description for purposes of explanation to provide a thorough understanding of various aspects of this application. However, it will be apparent that various aspects can be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0035] The following description provides only exemplary aspects and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the following description of the exemplary aspects will provide those skilled in the art with a description that can be used to implement the exemplary aspects. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of this application as set forth in the appended claims.

[0036] The terms “exemplary” and / or “example” are used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” and / or “example” is not necessarily to be construed as superior to or better than other aspects. Similarly, the term “aspects of this disclosure” does not require that all aspects of this disclosure include the features, advantages, or modes of operation discussed.

[0037] A camera is a device that uses an image sensor to receive light and capture image frames (such as still images or video frames). The terms "image," "image frame," and "frame" are used interchangeably herein. A camera may include a processor (such as an image signal processor (ISP)) that receives and processes one or more image frames. For example, raw image frames captured by a camera sensor may be processed by an ISP to generate a final image. The processing performed by the ISP may be performed by multiple filters or processing blocks applied to the captured image frames, such as denoising or noise filtering, edge enhancement, color balancing, contrast adjustment, intensity adjustment (such as darkening or brightening), tone adjustment, etc. Image processing blocks or modules may include lens / sensor noise correction, Bayer filters, de-mosaicing, color conversion, correction or enhancement / suppression of image attributes, noise reduction filters, sharpening filters, etc.

[0038] Cameras can be configured with various image capture and image processing operations and settings. Different settings produce images with different appearances. Some camera operations are determined and applied before or during image capture, such as automatic exposure control (AEC) and automatic white balance (AWB) processing. Additional camera operations applied before, during, or after image capture include those involving scaling (e.g., zooming in or out), ISO, aperture size, aperture coefficient, shutter speed, and gain. Other camera operations configure post-processing of the image, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or color.

[0039] As previously mentioned, for image processing, FSR is a common sensor operating mode used by many OEMs to improve IQ (Input Quality) because it reduces shutter-related artifacts in the image. Compared to normal readout mode, FSR mode allows for faster readout of image sensor data from image frames while maintaining the same amount of exposure time. The faster the readout of image sensor data, the lower the amount of shutter-related artifacts present in the rendered image. FSR mode can also allow for a reduction in required sensor power (e.g., in some cases, FSR mode can allow for 160 to 260 milliwatts of sensor power savings compared to normal readout mode). Therefore, from both an IQ and sensor power perspective, utilizing FSR mode for image processing can be beneficial.

[0040] However, FSR mode can impact the power consumption of the chipset, which may include an image signal processor (ISP). For example, FSR mode may require the ISP to operate at very high clock rates and voltages to complete image frame processing during the compressed readout time of FSR mode. Furthermore, FSR mode may require memory (e.g., double data rate (DDR) memory) to operate at high clock rates to enable rapid output of data received from the image sensor processor.

[0041] Dynamic voting (e.g., Dynamic Resource Voting (DRV)) can be used to optimize power consumption in fast sensor modes (e.g., FSR mode power consumption). Dynamic voting can be used to optimize the power overhead of camera chipsets (such as system-on-a-chip (SoC)) during camera operation in FSR mode. Using conventional static clocking mechanisms, ISPs and DDR memories have fixed clock rates to meet the instantaneous performance requirements of use cases, which can result in significant power overhead over the entire use case timeline.

[0042] Dynamic voting (performed by the DRV engine, for example) can dynamically increase the ISP and DDR clock rates during the short sensor readout duration of the use case timeline (e.g., by controlling ISP and DDR voting) and immediately decrease the ISP and DDR clock rates after the short sensor readout duration has ended (e.g., so that the clock rates are lower during the large blanking intervals in the use case timeline when no sensor readouts are performed). With such clock rate adjustments, high power overhead requirements can be limited to the sensor readout portion of the use case timeline (e.g., which is typically only 25% to 50% of the use case timeline).

[0043] Most camera use cases operate at a specified image frame rate configured by the advanced application software (SW). Dynamic voting employs a configurable hardware (HW) timer mechanism (e.g., Figure 8 The DRV engine 830 dynamically increases the clock rate immediately before receiving a new image frame from the image sensor and dynamically decreases the clock rate and shared rail voltage immediately after processing of the image frame is complete. Therefore, dynamic voting can allow for reduced chipset power overhead in FSR mode.

[0044] When the FPS is constant throughout the use case, existing DRV schemes operate adequately. However, in some cases, such as when lighting conditions or ambient light change dynamically, the software (SW) typically programs the per-frame exposure ratio to the sensor via a Camera Control Interface (CCI) (such as an I2C interface). This change in the per-frame exposure ratio alters the frame rate (e.g., FPS). Thus, this is an example of a use case where the frame rate (e.g., FPS) remains dynamically changing based on ambient lighting conditions. In parallel, the SW also programs a timer value into the DRV. This timer value is typically programmed into an Advanced High-Performance Bus (AHB) interface. For proper functionality, the DRV timer value programmed by the SW should match the corresponding frame rate being transmitted by the sensor.

[0045] The sensor and DRV are completely asynchronous hardware, as are the AHB and CCI interfaces. Therefore, there is a high probability of a mismatch between the DRV timer value and the corresponding frame rate being transmitted by the sensor. If such a mismatch occurs, it can lead to image frame loss and hardware hangs. Therefore, improved techniques for DRV in FSRs for dynamic FPS use cases may be beneficial.

[0046] Therefore, this document describes systems, apparatuses, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for DRV enhancements for dynamic FPS use cases. For example, in some examples, the systems and techniques employ timing indicators (such as a Pre-Start-of-Frame (PRE-SOF) indicator) to stabilize the voltage and clock levels of the ISP (e.g., the Image Front-End (IFE) component) and DDR after an increase in the clock rate and power (e.g., due to DRV voting in favor of the ISP and DDR). A sensor (e.g., a camera's) transmits this timing indicator to the DRV engine. The DRV engine will vote in favor of the ISP and DDR based on the timing indicator. Because the sensor transmits the timing indicator to the DRV engine, both the DRV vote and the frame rate (e.g., the image frame rate) will be synchronized, which avoids the possibility of any mismatch between the DRV timer value and the corresponding frame rate. In one or more examples, the timing indicator (e.g., PRE-SOF) may appear before or after the start-of-frame (SOF) of multiple image frames being processed.

[0047] Additional aspects of this disclosure are described in more detail below.

[0048] Figure 1 This is a block diagram illustrating the architecture of an image capture and processing system 100. The image capture and processing system 100 includes various components for capturing and processing images of a scene (e.g., an image of scene 110). The image capture and processing system 100 can capture individual images (or photographs) and / or capture video comprising multiple images (or video frames) in a specific sequence. A lens 115 of the system 100 faces scene 110 and receives light from scene 110. The lens 115 bends the light toward an image sensor 130. The light received by the lens 115 passes through an aperture controlled by one or more control mechanisms 120 and is received by the image sensor 130.

[0049] One or more control mechanisms 120 may control exposure, focus, and / or zoom based on information from image sensor 130 and / or information from image processor 150. One or more control mechanisms 120 may include multiple mechanisms and components; for example, control mechanism 120 may include one or more exposure control mechanisms 125A, one or more focus control mechanisms 125B, and / or one or more zoom control mechanisms 125C. One or more control mechanisms 120 may also include additional control mechanisms besides those illustrated, such as controls for analog gain, flash, HDR, depth of field, and / or other image capture properties.

[0050] The focus control mechanism 125B of the control mechanism 120 can obtain the focus settings. In some examples, the focus control mechanism 125B stores the focus settings in a memory register. Based on the focus settings, the focus control mechanism 125B can adjust the positioning of the lens 115 relative to the positioning of the image sensor 130. For example, based on the focus settings, the focus control mechanism 125B can adjust the focus by moving the lens 115 closer to or further away from the image sensor 130 via an actuated motor or servo. In some cases, the device 105A may include additional lenses, such as one or more microlenses on each photodiode of the image sensor 130, each microlens bending light received from the lens 115 toward the corresponding photodiode before it reaches the photodiode. The focus settings can be determined by contrast detection autofocus (CDAF), phase detection autofocus (PDAF), or some combination thereof. The focus settings can be determined using the control mechanism 120, the image sensor 130, and / or the image processor 150. The focus settings may be referred to as image capture settings and / or image processing settings.

[0051] The exposure control mechanism 125A of the control mechanism 120 can obtain the exposure settings. In some cases, the exposure control mechanism 125A stores the exposure settings in a memory register. Based on the exposure settings, the exposure control mechanism 125A can control the aperture size (e.g., aperture size or aperture coefficient), the duration of the aperture opening (e.g., exposure time or shutter speed), the sensitivity of the image sensor 130 (e.g., ISO speed or film speed), the analog gain applied by the image sensor 130, or any combination thereof. The exposure settings may be referred to as image capture settings and / or image processing settings.

[0052] The zoom control mechanism 125C of the control mechanism 120 can obtain zoom settings. In some examples, the zoom control mechanism 125C stores the zoom settings in a memory register. Based on the zoom settings, the zoom control mechanism 125C can control the focal length of an assembly (lens assembly) comprising lens elements including lens 115 and one or more additional lenses. For example, the zoom control mechanism 125C can control the focal length of the lens assembly by actuating one or more motors or servos to move one or more lenses in the lens relative to each other. The zoom settings may be referred to as image capture settings and / or image processing settings. In some examples, the lens assembly may include a parfocal zoom lens or a variable focal length zoom lens. In some examples, the lens assembly may include a focusing lens (in some cases, this focusing lens may be lens 115) that first receives light from scene 110, where the light then passes through a focusless zoom system between the focusing lens (e.g., lens 115) and image sensor 130 before reaching image sensor 130. In some cases, a focusless zoom system may include two positive (e.g., converging, convex) lenses with equal or similar focal lengths (e.g., within a threshold difference), with a negative (e.g., diverging, concave) lens between them. In some cases, the zoom control mechanism 125C moves one or more lenses in the focusless zoom system, such as the negative lens and one or both positive lenses.

[0053] Image sensor 130 includes one or more arrays of photodiodes or other photosensitive elements. Each photodiode measures the amount of light that ultimately corresponds to a specific pixel in the image generated by image sensor 130. In some cases, different photodiodes may be covered by different color filters, and thus light matching the color of the color filter covering the photodiode can be measured. For example, Bayer color filters include red, blue, and green color filters, where each pixel of the image is generated based on red light data from at least one photodiode covered by the red color filter, blue light data from at least one photodiode covered by the blue color filter, and green light data from at least one photodiode covered by the green color filter. Other types of color filters may use yellow, magenta, and / or cyan (also known as "emerald green") color filters as alternatives to or complements to red, blue, and / or green color filters. Some image sensors may have no color filters at all and may alternatively use different photodiodes (in some cases stacked vertically) throughout the pixel array. Different photodiodes in the pixel array can have different spectral sensitivity profiles, thus responding to light of different wavelengths. Monochrome image sensors may also lack color filters and therefore lack color depth.

[0054] In some cases, image sensor 130 may alternatively or additionally include an opaque mask and / or a reflective mask that blocks light from reaching certain photodiodes or portions of certain photodiodes at certain times and / or from certain angles, which can be used for phase detection autofocus (PDAF). Image sensor 130 may also include an analog gain amplifier for amplifying the analog signal output from the photodiodes and / or an analog-to-digital converter (ADC) for converting the analog signal output from the photodiodes (and / or amplified by the analog gain amplifier) ​​into a digital signal. In some cases, certain components or functions discussed relative to one or more control mechanisms in control mechanism 120 may alternatively or additionally be included in image sensor 130. Image sensor 130 may be a charge-coupled device (CCD) sensor, an electron-multiplying CCD (EMCCD) sensor, an active pixel sensor (APS), a complementary metal-oxide-semiconductor (CMOS), an N-type metal-oxide-semiconductor (NMOS), a hybrid CCD / CMOS sensor (e.g., sCMOS), or some other combination thereof.

[0055] The image processor 150 may include one or more processors, such as one or more image signal processors (ISPs) (including ISP 154), one or more host processors (including host processor 152), and / or one or more of any other type of processor 1610 discussed with respect to the computing system 1600. The host processor 152 may be a digital signal processor (DSP) and / or other types of processor. In some specific implementations, the image processor 150 is a single integrated circuit or chip (e.g., referred to as a system-on-a-chip or SoC) including the host processor 152 and the ISP 154. In some cases, the chip may also include one or more input / output ports (e.g., input / output (I / O) port 156), a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), a broadband modem (e.g., 3G, 4G, or LTE, 5G, etc.), memory, and connectivity components (e.g., Bluetooth). ™This includes components such as the Global Positioning System (GPS), any combination thereof, and / or other components. I / O port 156 may include any suitable input / output port or interface according to one or more protocols or specifications, such as Inter-Integrated Circuit 2 (I2C) interface, Inter-Integrated Circuit 3 (I3C) interface, Serial Peripheral Interface (SPI) interface, Serial General Purpose Input / Output (GPIO) interface, Mobile Industrial Processor Interface (MIPI) (such as MIPI CSI-2 physical (PHY) layer port or interface, Advanced High Performance Bus (AHB) bus, any combination thereof, and / or other input / output ports). In an exemplary example, host processor 152 may use the I2C port to communicate with image sensor 130, and ISP 154 may use the MIPI port to communicate with image sensor 130.

[0056] Image processor 150 can perform multiple tasks, such as demosaicing, color space conversion, image frame downsampling, pixel interpolation, automatic exposure (AE) control, automatic gain control (AGC), CDAF, PDAF, automatic white balance, merging image frames to form an HDR image, image recognition, object recognition, feature recognition, receiving input, managing output, managing memory, or some combination thereof. Image processor 150 can store image frames and / or processed images in random access memory (RAM) 140 / 1625, read-only memory (ROM) 145 / 1620, cache 1612, memory unit 1615, another storage device 1630, or some combination thereof.

[0057] Various input / output (I / O) devices 160 may be connected to the image processor 150. I / O devices 160 may include a display screen, keyboard, keypad, touchscreen, touchpad, touch-sensitive surface, printer, any other output device 1635, any other input device 1645, or some combination thereof. In some cases, text may be input to the image processing device 105B via the physical keyboard or keypad of the I / O device 160, or via a virtual keyboard or keypad on the touchscreen of the I / O device 160. I / O 160 may include one or more ports, jacks, or other connectors enabling a wired connection between the device 105B and one or more peripheral devices, through which the device 105B can receive data from and / or send data to the one or more peripheral devices. I / O 160 may include one or more wireless transceivers enabling a wireless connection between the device 105B and one or more peripheral devices, through which the device 105B can receive data from and / or send data to the one or more peripheral devices. Peripheral devices may include any type of I / O device 160 discussed earlier, and they can be considered I / O devices 160 in themselves once they are coupled to ports, jacks, wireless transceivers or other wired and / or wireless connectors.

[0058] In some cases, the image capture and processing system 100 may be a single device. In other cases, the image capture and processing system 100 may consist of two or more independent devices, including an image capture device 105A (e.g., a camera) and an image processing device 105B (e.g., a computing device coupled to the camera). In some embodiments, the image capture device 105A and the image processing device 105B may be coupled together, for example, via one or more wires, cables, or other electrical connectors, and / or wirelessly coupled together via one or more wireless transceivers. In some embodiments, the image capture device 105A and the image processing device 105B may be disconnected from each other.

[0059] like Figure 1 As shown, the vertical dashed line will Figure 1 The image capture and processing system 100 is divided into two parts, namely, image capture device 105A and image processing device 105B. Image capture device 105A includes a lens 115, a control mechanism 120, and an image sensor 130. Image processing device 105B includes an image processor 150 (including an ISP 154 and a host processor 152), RAM 140, ROM 145, and I / O 160. In some cases, certain components illustrated in image capture device 105A (such as ISP 154 and / or host processor 152) may be included in image capture device 105A.

[0060] Image capture and processing system 100 may include electronic devices such as mobile or landline phones (e.g., smartphones, cellular phones, etc.), desktop computers, laptop or notebook computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some examples, image capture and processing system 100 may include one or more wireless transceivers for wireless communication (such as cellular network communication, 802.11 Wi-Fi communication, wireless local area network (WLAN) communication, or some combination thereof). In some specific implementations, image capture device 105A and image processing device 105B may be different devices. For example, image capture device 105A may include a camera device, and image processing device 105B may include a computing device, such as a mobile phone, desktop computer, or other computing device.

[0061] Although the image capture and processing system 100 is shown to include certain components, those skilled in the art will understand that the image capture and processing system 100 may include more than [other components]. Figure 1 The components shown herein are additional components. Components of the image capture and processing system 100 may include software, hardware, or one or more combinations of software and hardware. For example, in some embodiments, components of the image capture and processing system 100 may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, CPUs, and / or other suitable electronic circuits); and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. Software and / or firmware may include one or more instructions stored on a computer-readable storage medium and executable by one or more processors of an electronic device implementing the image capture and processing system 100.

[0062] The host processor 152 can configure the image sensor 130 with new parameter settings (e.g., via an external control interface such as I2C, I3C, SPI, GPIO, and / or other interfaces). In one exemplary example, the host processor 152 can update the exposure settings used by the image sensor 130 based on the internal processing results of the exposure control algorithm from past image frames.

[0063] In some examples, the host processor 152 may perform electronic image stabilization (EIS). For example, the host processor 152 may determine motion vectors corresponding to motion compensation for one or more image frames. In some aspects, the host processor 152 may position a cropped array of pixels (“image window”) within a total pixel array. The image window may include pixels used to capture the image. In some examples, the image window may include all pixels in the sensor except for a portion of rows and columns at the periphery of the sensor. In some cases, the image window may be centered on the sensor when the image capture device 105A is stationary. In some aspects, peripheral pixels may surround the pixels of the image window and form a set of buffered pixel rows and buffered pixel columns around the image window. The host processor 152 may implement EIS and shift the image window from frame to frame of video such that the image window tracks the same scene on consecutive frames (e.g., assuming the subject does not move). In some examples where the subject moves, the host processor 152 may determine that the scene has changed.

[0064] In some examples, the image window may include at least 95% (e.g., 95% to 99%) of the pixels on the sensor. The first region of interest (ROI) (e.g., for AE and / or AWB) may include at least 95% (e.g., 95% to 99%) of the image data within the field of view of a plurality of imaging pixels in the image sensor 130 of the image capture device 105A. In some aspects, a plurality of buffer pixels may be reserved at the periphery of the sensor (outside the image window) as a buffer to allow the image window to shift to compensate for jitter. In some cases, the image window may be moved such that even if light from the subject is projected onto different areas of the sensor, the subject remains in the same position within the adjusted image window. In another example, the buffer pixels may include the top ten, bottom ten, left ten, and right ten columns of pixels on the sensor. In some configurations, when the image capture device 105A is stationary, the buffer pixels are not used for AF, AE, or AWB, and the buffer pixels are not included in the image output. If jitter causes the sensor to shift to the left by twice the width of the pixel column between frames, the EIS algorithm can be used to shift the image window to the right by two columns of pixels, so that the captured image shows the same scene in the current frame in the next frame. The host processor 152 can use EIS to make the transition from one frame to the next smooth.

[0065] In some respects, the host processor 152 can also dynamically configure the parameter settings of the internal pipelines or modules of the ISP 154 to match the settings of one or more input image frames from the image sensor 130, so that the image data is correctly processed by the ISP 154. The processing (or pipeline) blocks or modules of the ISP 154 may include modules for lens / sensor noise correction, demosaicing, color conversion, correction or enhancement / suppression of image attributes, denoising filters, sharpening filters, and so on. The settings of different modules of the ISP 154 can be configured by the host processor 152. Each module may include a large number of tunable parameter settings. Additionally, since different modules may affect similar aspects of the image, the modules can be interdependent. For example, denoising and texture correction or enhancement may both affect the high-frequency aspects of the image. As a result, a large number of parameters are used by the ISP to generate the final image based on the captured raw image.

[0066] In some cases, the image capture and processing system 100 can automatically perform one or more of the image processing functions described above. For example, one or more of the control mechanisms 120 may be configured to perform autofocus operation, auto exposure operation, and / or auto white balance operation. In some embodiments, the autofocus function allows the image capture device 105A to automatically focus before capturing the desired image. Various autofocus techniques exist. For example, active autofocus techniques typically determine the distance between the camera and the subject of the image via the camera's distance sensor by emitting infrared laser or ultrasonic signals and receiving the reflections of these signals. Furthermore, passive autofocus techniques use the camera's own image sensor to focus the camera and therefore do not require the integration of additional sensors into the camera. Passive AF techniques include contrast detection autofocus (CDAF), phase detection autofocus (PDAF), and in some cases, hybrid systems using both techniques. The image capture and processing system 100 may be equipped with these or any additional types of autofocus techniques.

[0067] Synchronization between the image sensor 130 and the ISP 154 is important in order to provide an operational image capture system that can generate high-quality images without interruption and / or failure. Figure 2 This is a block diagram illustrating an example of an image capture and processing system 200, which includes an image processor 250 (including a host processor 252 and an ISP 254) communicating with an image sensor 230. Figure 2The configuration shown illustrates a conventional synchronization technique used in a camera system. Generally, the host processor 252 attempts to provide synchronization between the image sensor 230 and the ISP 254 using fixed time intervals by communicating separately with both the image sensor 230 and the ISP 254. For example, in a conventional camera system, the host processor 252 communicates with the image sensor 230 (e.g., via an I2C port) and programs the parameters of the image sensor 230 for a first fixed time interval (such as a two-frame interval before the image frame will be processed by the ISP 254). The host processor 252 communicates with the ISP 254 (e.g., via an internal AHB bus or other interface) and programs the ISP 254 parameter settings for a second fixed time interval (such as a one-frame interval before the image frame will be processed by the ISP 254).

[0068] Image sensor 230 can transmit image frames to ISP 254, such as via MIPI CSI-2 PHY port or interface or other suitable interface. Figure 2 (See B to C in the diagram). However, communication between the host processor 252 and the image sensor 230 (shown as A to B) is indeterminate. Similarly, communication between the image sensor 230 and the ISP 254 (shown as B to C) and between the host processor 252 and the ISP 254 (shown as A to C) is also indeterminate. For example, the programming of the image sensor 230 and the ISP 254 by the host processor 252 may have varying latency, which can lead to a mismatch in parameter settings between the sensor and the ISP. Latency can be attributed to high CPU utilization, congestion in one or more I / O ports, and / or other factors.

[0069] Figure 3This is a block diagram of an example device 300 that can be used for camera dynamic voting to optimize power in fast sensor modes. Device 300 may include or be coupled to camera 302, and may also include processor 306, memory 308 storing instructions 310, camera controller 312, display 316, and multiple input / output (I / O) components 318 including one or more microphones (not shown). Example device 300 may be any suitable device capable of capturing and / or storing images or video, including, for example, wired and wireless communication devices (such as camera phones, smartphones, tablets, security systems, smart home devices, connected home devices, surveillance devices, Internet Protocol (IP) devices, dashcams, laptops, desktop computers, automobiles, etc.), digital cameras (including still cameras, camcorders, etc.), or any other suitable device. Device 300 may include additional features or components not shown. For example, it may include a wireless interface for wireless communication devices, which may include multiple transceivers and a baseband processor. Device 300 may include or be coupled to an additional camera other than camera 302. This disclosure should not be limited to any particular example or illustration, including example device 300.

[0070] Camera 302 may be able to capture individual image frames (such as still images) and / or capture video (such as a sequence of captured image frames). Camera 302 may include one or more image sensors (not shown for simplicity) and a shutter for capturing image frames and providing the captured image frames to camera controller 312. Although a single camera 302 is shown, any number of cameras or camera components may be included and / or coupled to device 300. For example, the number of cameras may be increased to achieve greater depth determination capability or better resolution for a given field of view (FOV).

[0071] Memory 308 may be a non-transient or non-transitory computer-readable medium storing computer-executable instructions 310 for performing all or a portion of one or more operations described in this disclosure. Device 300 may also include a power source 320 that may be coupled to or integrated into device 300.

[0072] Processor 306 may be one or more suitable processors capable of executing scripts or instructions of one or more software programs (such as instructions 310) stored in memory 308. In some aspects, processor 306 may be one or more general-purpose processors that execute instructions 310 to cause device 300 to perform any number of functions or operations. In additional or alternative aspects, processor 306 may include integrated circuits or other hardware for performing functions or operations without the use of software. Although in Figure 3The example is shown as coupled to each other via processor 306, but processor 306, memory 308, camera controller 312, display 316 and I / O components 318 can be coupled to each other in various arrangements. For example, processor 306, memory 308, camera controller 312, display 316 and / or I / O components 318 can be coupled to each other via one or more local buses (not shown for simplicity).

[0073] Display 316 may be any suitable display or screen that allows the user to interact with and / or present items (such as captured images and / or videos) for the user to view. In some aspects, display 316 may be a touch-sensitive display. Display 316 may be part of device 300 or external to that device. Display 316 may include LCD, LED, OLED, or similar displays. I / O component 318 may be or may include any suitable mechanism or interface for receiving input (such as commands) from the user and / or providing output to the user. For example, I / O component 318 may include (but is not limited to) a graphical user interface, keyboard, mouse, microphone, and speaker, etc.

[0074] Camera controller 312 may include image signal processor (ISP) 314, which may be (or may include) one or more image signal processors for processing image frames or video captured by camera 302. For example, ISP 314 may be configured to perform various processing operations for autofocus (AF), automatic white balance (AWB), and / or automatic exposure (AE) (which may also be referred to as automatic exposure control (AEC)). Examples of image processing operations include, but are not limited to, cropping, scaling (e.g., scaling to different resolutions), image stitching, image format conversion, color interpolation, image interpolation, color processing, image filtering (e.g., spatial image filtering), etc.

[0075] In some example implementations, the camera controller 312 (such as ISP 314) may implement various functionalities, including image processing and / or control operations of the camera 302. In some aspects, the ISP 314 may execute instructions from memory (such as instructions 310 stored in memory 308 or stored in a separate memory coupled to the ISP 314) to control image processing and / or operation of the camera 302. In other aspects, the ISP 314 may include specific hardware for controlling image processing and / or operation of the camera 302. The ISP 314 may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions.

[0076] Although Figure 3Not shown, but in some implementations, the ISP 314 and / or camera controller 312 may include an AF module, an AWB module, and / or an AE module. The ISP 314 and / or camera controller 312 may be configured to perform AF, AWB, and / or AE processes. In some examples, the ISP 314 and / or camera controller 312 may include dedicated hardware circuitry (e.g., an application-specific integrated circuit (ASIC)) configured to perform the AF, AWB, and / or AE processes. In other examples, the ISP 314 and / or camera controller 312 may be configured to execute software and / or firmware to perform the AF, AWB, and / or AE processes. When configured in software, the code for the AF, AWB, and / or AE processes may be stored in memory (such as instructions 310 stored in memory 308 or instructions stored in a separate memory coupled to the ISP 314 and / or camera controller 312). In other examples, the ISP 314 and / or camera controller 312 may use a combination of hardware, firmware, and / or software to perform AF, AWB, and / or AE processes. When configured as software, the AF, AWB, and / or AE processes may include instructions for configuring the ISP 314 and / or camera controller 312 to perform various image processing and device management tasks, including those using techniques disclosed herein.

[0077] Figure 4 This is a block diagram illustrating the operation of an image signal processing pipeline 402 of an image signal processor (e.g., ISP 314). For example, the ISP 314 may be configured to execute the image signal processing pipeline 402 to process input image data. The ISP 314 can be accessed from... Figure 3 The camera 302 and / or the image sensor (not shown) of the camera 302 receive input image data. In some examples, such as Figure 4 As shown, input image data may include color data and / or any other data (e.g., depth data) of the image / frame. Figure 4 In the example, the color data received for the input image data can be in Bayer format. Instead of capturing the red (R), green (G), and blue (B) values ​​for each pixel of the image, the image sensor (e.g., the image sensor of camera 302) can use a Bayer filter mosaic (or more generally, a color filter array (CFA)). This would allow each photoelectric sensor in the digital image sensor to capture different colors in the RGB color spectrum. An example of a filter pattern used for a Bayer filter mosaic could include a 50% green filter, a 25% red filter, and a 25% blue filter.

[0078] The Bayer processing unit 410 can perform one or more initial processing techniques on the raw Bayer data received by the ISP 314, including, for example, subtraction, slip correction, defective pixel correction, black level compensation and / or denoising.

[0079] The stats filtering process 412 determines the Bayer rank or Bayer grid (BG) statistics of the received input image data. In some examples, BG statistics may include the red to green ratio (R / G) (which indicates the presence of red tinting and the amount of red tinting that may be present in the image) and / or the blue to green ratio (B / G) (which indicates the presence of blue tinting and the amount of blue tinting that may be present in the image). For example, the (R / G) of an image or a portion / region of an image may be described by the following formula (1): (1)

[0080] The image or a portion / region of an image comprises pixels 1-N, where each pixel n includes a red value (Red(n), a blue value (Blue(n), or a green value (Green(n))) in the RGB space. (R / G) is the sum of the red values ​​of the red pixels in the image divided by the sum of the green values ​​of the green pixels in the image. Similarly, (B / G) of the image or a portion / region of an image can be described by the following formula (2): (2)

[0081] In some other example implementations, different color spaces, such as Y'UV, may be used, where the chromaticity values ​​UV indicate color, and / or other indicators that may determine the tinting or other color temperature effects on an image.

[0082] The AWB module and / or process 404 can analyze information associated with the received image data to determine the scene's illuminators from a plurality of possible illuminators, and can determine the AWB gain based on the determined illuminators to apply to the received image and / or subsequent images. White balance is a process used to attempt to match the colors of an image with the user's perceptual experience of the captured object. As an example, white balance processing can be designed so that white objects actually appear white in the processed image, and gray objects actually appear gray in the processed image.

[0083] The illuminator may include the lighting conditions of the scene being captured, the type of light, etc. In some examples, the image capture device (e.g., such as...) Figure 3The user of the device (300) can select or specify the illuminators under which the image is captured. In other examples, the image capture device itself can automatically determine the most likely illuminators and perform white balance based on the determined illuminators (e.g., lighting conditions). To better render the colors of the scene in the captured image or video, the AWB algorithm on the device and / or camera can attempt to determine the illuminators of the scene and set / adjust the white balance of the image or video accordingly.

[0084] During AWB process 404, device 300 can determine or estimate the color temperature of a received frame (e.g., an image). Color temperature indicates the dominant hue of an image. The true color temperature of a scene captured in a video or image is the color of the scene's light source. If the light is emitted from a perfect blackbody radiator (theoretically ideal for all electromagnetic wavelengths) at a specific color temperature (expressed in Kelvin (K)), and the color temperature is known, then the color temperature of the scene is known. For example, in the color space defined by the International Commission on Illumination (CIE) (from 1931), the chromaticity of radiation from a blackbody radiator with temperatures ranging from 1,000 K to 20,000 K is the Planck locus. Colors on the Planck locus from approximately 2,000 K to 20,000 K are considered white, where 2,000 K is warm white or reddish white, and 20,000 K is cool white or bluish white. Many incandescent light sources include Planck radiators (tungsten filaments or another filament used for light emission), which emit warm white light with a color temperature of approximately 2,400 K to 3,100 K.

[0085] However, other light sources (such as fluorescent lamps, discharge lamps, or light-emitting diodes (LEDs)) are not perfect blackbody radiators whose radiation falls along the Planck locus. For example, LEDs or neon signs emit light through electroluminescence, and the color of the light does not follow the Planck locus. The color temperature determined for such light sources can be the correlated color temperature (CCT). The CCT is an estimated color temperature of a light source whose color does not fall perfectly along the Planck locus. For example, the CCT of a light source is the blackbody color temperature that is closest to the radiation of the light source. CCT can also be represented in K.

[0086] CCT can be an approximation of the true color temperature of a scene. For example, CCT can be a simplified color measure of the chromaticity coordinates in the CIE 1931 color space. Many devices can use AWB to estimate CCT for color balance.

[0087] CCT can be a temperature rating ranging from warm colors (such as yellow and red below 3200K) to cool colors (such as blue above 4000K). CCT (or other color temperature) indicates the tinting that will appear in an image captured using such a light source. For example, a CCT of 2700K indicates red tinting, and a CCT of 5000K indicates blue tinting.

[0088] Different lighting sources or ambient lighting can illuminate a scene, and the color temperature may be unknown to the device. Therefore, the device can analyze data captured by an image sensor to estimate the color temperature of an image (e.g., a frame). For example, the color temperature could be an estimate of the overall CCT of the light sources in the scene within the image. The data captured by the image sensor for estimating the color temperature of a frame (e.g., an image) could be the captured image itself.

[0089] After device 300 determines the color temperature of the scene (such as during AWB execution), device 300 can use the color temperature to determine the color balance for correcting any shading in the image. For example, if the color temperature indicates that the image includes red shading, device 300 can, for example, decrease the red value or increase the blue value for each pixel of the image in the RGB space. Color balance can be color correction (such as values ​​used to decrease red values ​​or increase blue values).

[0090] Example inputs to the AWB process 404 may include Bayer class or Bayer grid (BG) statistics of the received image data determined via the statistical filtering process 412, exposure index (e.g., the brightness of the scene in the received image data), and auxiliary information, which may include contextual information of the scene based on the audio input (as will be discussed in further detail below), depth information, etc. It should be noted that the AWB process 404 may be included as a separate AWB module. Figure 3 The camera controller 312 is located within it.

[0091] AE process 406 may include configuration, calculation and / or storage. Figure 3 The camera 302 provides instructions for exposure settings. Exposure settings may include the amount of sensor gain to apply, the amount of digital gain to apply, shutter speed and / or exposure time, aperture setting, and / or ISO setting for capturing subsequent images. The AE process 406 can use audio input and / or scene context information based on the audio input to determine and / or apply exposure settings more quickly. It should be noted that the AE process 406 can be included as a separate AE module. Figure 3 The camera controller 312 is located within it.

[0092] AF process 408 may include configuration, calculation and / or storage. Figure 3 The AF process 408 provides instructions for setting the autofocus on camera 302. The AF process 408 can determine the autofocus settings (e.g., initial lens position, final lens position, etc.) based on audio input and / or contextual information of the scene based on the audio input. It should be noted that the AF process 408 can be included as a separate AF module. Figure 3 The camera controller 312 is located within it.

[0093] The demosaicing unit 414 can be configured to convert processed Bayer image data into RGB values ​​for each pixel of the image. As explained above, the Bayer data may include only the value of one color channel (R, G, or B) for each pixel of the image. The demosaicing unit 414 can determine the values ​​of the other color channels of a pixel by interpolating from the color channel values ​​of neighboring pixels. In some ISP pipelines 402, the demosaicing unit 414 may be located before or after the AWB process 404, AE process 406, and / or AF process 408.

[0094] Other processing units 416 may apply additional processing to the image after the AWB process 404, AE process 406, and / or AF process 408, and / or demosaic processing unit 414. Additional processing may include color, hue, and / or spatial processing of the image.

[0095] As previously mentioned, for image processing, FSR is a common sensor operating mode used by many OEMs to improve IQ because it reduces shutter-related artifacts in the image. Compared to normal readout mode, FSR mode allows for faster readout of image sensor data for image frames while maintaining the same amount of exposure time. The faster the image sensor data readout is performed, the lower the amount of shutter-related artifacts present in the rendered image.

[0096] Figure 5 This is a diagram comparing the timing of FSR mode with the timing of normal read mode. Specifically, Figure 5 This is an example illustration of a camera sensor timing 500. Figure 5 In this paper, the timing diagram 560 of FSR mode is compared with the timing diagram 550 of normal readout mode.

[0097] exist Figure 5 In the normal readout mode timing diagram 550, the exposure time (e.g., time period 510) and the data from one or more camera sensors (e.g., Figure 8 Sensor 870 or Figure 12 The sensor 1220) is connected to the camera SOC (which may include one or more ISPs). Figure 8 IFE 840 or Figure 12 The IFE 1250) reads the time of the line (e.g., time period 520). Therefore, the time of the line read (e.g., time period 520) is the exposure time of the image frame.

[0098] like Figure 5As shown, these rows are exposed one at a time (e.g., time period 520). In the normal readout timing diagram 550, in the first row, the first row is exposed, and then the second row is read. Then, the second row is exposed (e.g., time period 510), and then the second row is read (e.g., time period 520).

[0099] For both normal readout timing diagram 550 and FSR mode timing diagram 560, the same amount of exposure time should be maintained for all rows throughout the entire image frame. This exposure time occurs only on one row at a time. It is desirable to maintain the same exposure time for each row; otherwise, many rolling shutter artifacts may exist in the rendered image. For this reason, the start of the first exposure time for row N+1 (e.g., row N+1) should be delayed by a certain amount of time after the start of the first exposure time for the previous row (e.g., row N). The start of the first exposure time for row N+1 (e.g., row N+1) should be delayed because the readout of the previous row (e.g., row N) has not yet been completed, since readout can only occur serially on the bus. Only one MIPI will connect the camera sensor (e.g., Figure 8 Sensor 870 or Figure 12 The sensor 1220 is connected to the SOC (e.g., including an ISP, such as...). Figure 8 IFE 840 or Figure 12 (IFE 1250), and therefore the readout of sensor data is fully serialized on the bus. Therefore, the readout of row N plus one (e.g., row N+1) cannot occur until the readout of the previous row (e.g., row N) is completed.

[0100] Since the readout of row N+1 (e.g., row N+1) can only occur after the readout of the previous row (e.g., row N) is complete, there is a certain offset at the start of the exposure time for each row (e.g., an increment of 580). The start of the exposure time for the second row is delayed by an offset from the start of the exposure time for the first row (e.g., an increment of 580), and the start of the exposure time for the third row is delayed by an offset from the start of the exposure time for the second row, and so on. As the entire image frame undergoes the image processing, as can be seen in the normal readout timing diagram 550, there is a significant time lag between the start of the exposure of the first row in the image frame and the start of the exposure time for the last row in the image frame (e.g., time interval 570a). The larger this time lag is, the more rolling shutter artifacts will be present in the rendered image.

[0101] For FSR mode timing diagram 560, the time lag between the start of exposure in the first row of the image frame and the start of exposure time in the last row of the image frame (e.g., time period 570b) is less than the time lag between the start of exposure in the first row of the image frame and the start of exposure time in the last row of the image frame (e.g., time period 570a) in normal mode timing diagram 550. The smaller this time lag (e.g., time period 570b), the lower the amount of rolling shutter artifacts in the rendered image.

[0102] However, while FSR mode allows for a reduction in the amount of rolling shutter artifacts, there are associated costs. Because FSR mode performs fast readout of sensor data, it requires increased MIPI signal and clock rates to accommodate this rapid readout. Currently, not all sensors support FSR mode. Typically, only expensive camera sensors support it. However, to reduce the amount of rolling shutter artifacts in rendered images and provide improved image quality (IQ), most high-end devices are currently moving towards supporting FSR mode.

[0103] exist Figure 5 When comparing the normal readout timing diagram 550 with the FSR mode timing diagram 560, it is evident that each of the gaps (e.g., gap 540b) in the aggregated readout energy 530b of the FSR mode timing diagram 560 is larger than each of the gaps (e.g., gap 540a) in the aggregated readout energy 530a of the normal mode timing diagram 550. Therefore, the FSR mode can have a larger blanking period (e.g., one of the blanking periods is gap 540b) than the normal mode (e.g., one of the blanking periods is gap 540a).

[0104] As previously mentioned, FSR mode consumes less power than normal readout mode. FSR mode can lead to a reduction in the required sensor power (e.g., in some cases, FSR mode can allow for a sensor power saving of 160 to 260 milliwatts compared to normal readout mode). Therefore, from both an IQ and sensor power perspective, utilizing FSR mode for image processing instead of normal readout mode may be beneficial.

[0105] Figure 6 This is an illustration demonstrating an example of timing using a camera in FRS mode. Figure 6The image shows sensor timing 610 and IFE and DDR timing 620. For FRS mode, the duration of active frames transmitting images and pixels is very short compared to a full inter-frame time period. Typically, the use case is a 30 FPS use case, where the inter-frame time 630 is thirty-three (33) milliseconds (ms). For example, in sensor timing 610, the first frame (e.g., during the duration of the 33 ms inter-frame time 630) occurs, followed by subsequent frames, each with a duration of 33 ms. The frames themselves are actually transmitted only during the first eight (8) ms of the inter-frame time 630 (e.g., active frame time 640). The remaining twenty-five (25) ms of the inter-frame time 630 (e.g., idle time 650) are idle periods during which the sensor does not transmit any data.

[0106] As previously mentioned, FSR mode can affect the power consumption of the chipset, which may include the image sensor processor (ISP). For example, FSR mode may require the ISP to operate at very high clock rates and voltages to complete image frame processing during the compressed readout time of FSR mode. Additionally, FSR mode may require memory (e.g., DDR memory) to operate at high clock rates to enable rapid output of data received from the image sensor processor.

[0107] Dynamic voting (e.g., DRV) can be used to optimize FSR mode power consumption. Dynamic voting can also be used to optimize camera chipset (e.g., SOC) power overhead during camera operation in FSR mode. Using conventional static clocking mechanisms, ISPs and DDR memories have fixed clock rates to meet the instantaneous performance requirements of use cases, which can require significant power overhead over the entire use case timeline.

[0108] (For example, by DRV engines (such as...) Figure 8 The dynamic voting performed by the DRV engine 830 can dynamically increase (e.g., by controlling ISP and DDR voting) the ISP and DDR clock rates during the short sensor readout duration of the use case timeline, and immediately decrease the ISP and DDR clock rates after the short sensor readout duration is complete (e.g., so that the clock rates are lower during the large blanking intervals in the use case timeline when no sensor readouts are performed). With such clock rate adjustments, high power overhead requirements can be limited to the sensor readout portion of the use case timeline, which is typically only 25% to 50% of the use case timeline.

[0109] Most camera use cases operate at a specified image frame rate configured by the high-level application switch (SW). Dynamic voting employs a configurable HW timer mechanism (e.g., Figure 8The DRV engine 830 dynamically increases the clock rate immediately before receiving a new image frame from the image sensor (e.g., via a vote of approval), and dynamically decreases the clock rate and shared rail voltage immediately after processing of the image frame is complete (e.g., via a vote of rejection). Therefore, dynamic voting can reduce chipset power overhead in FSR mode.

[0110] Figure 7 This is a diagram illustrating an example of using DRV for camera timing in an FRS use case. Figure 7 The diagram shows the Start of Frame (SOF) timing 710, vote-for-favor timing 720, End of Frame (EOF) timing 730, vote-for-no-decision timing 740, and vote level timing 750.

[0111] In the FRS use case, DRV is used to reduce SOC power by controlling ISP (e.g., IFE) and DDR voting. The SW can configure a DRV timer (e.g., TIMER_VAL in SOF timer 710) at the beginning of each use case. The timer (e.g., TIMER_VAL) can begin counting at each SOF (e.g., each SOF is represented by each pulse of SOF timer 710). When the timer (e.g., TIMER_VAL) expires, there is a vote of approval from the IFE and DDR resources (e.g., each vote of approval is represented by each pulse of vote-approval timer 720), making the IFE and DDR resources ready to receive the next SOF.

[0112] The voting level of a resource (e.g., as shown in voting level timing 750) will rise and fall based on a vote in favor (e.g., a pulse) in vote-for timing 720 and a vote against (e.g., a pulse) in vote-against timing 740. Therefore, when a timer (e.g., TIMER_VAL) expires, the voting level of voting level timing 750 rises. Thus, when the next SOF arrives at SOF timing 710, the IFE and DDR resources are ready to receive data.

[0113] When an EOF is received (e.g., each EOF is represented by each pulse of EOF timing 730), there is a corresponding veto of IFE and DDR resources (e.g., each veto is represented by each pulse of veto timing 740) to save power. When a veto exists in veto timing 740, the voting level of voting level timing 750 will drop.

[0114] In summary, when the timer (e.g., TIMER_VAL) of SOF timer 710 expires, the voting level of voting level timer 750 rises. When EOF occurs in EOF timer 730, a corresponding veto exists in voting veto 740, and therefore the voting level of voting level timer 750 falls, and this cycle continues to repeat.

[0115] Figure 8 This is a diagram illustrating an example of a system 800 using DRV for Fast Sensor Readout (FSR). Figure 8 In the diagram, system 800 is shown to include SW 810, AHB interface 820, DRV engine 830, IFE 840, DDR 850, CCI 860 (e.g., I2C or I3C interface) and sensor 870.

[0116] When the FPS is constant throughout the use case, existing DRV schemes operate adequately. However, in some cases, such as when lighting conditions or ambient light change dynamically, the SW 810 typically programs a per-frame exposure ratio 855 (e.g., Exp_ratio) to the sensor 870 via a CCI 860 (e.g., an I2C interface). This change in the per-frame exposure ratio 855 causes a change in the frame rate (e.g., FPS). This is an example of a use case where the frame rate (e.g., FPS) remains dynamically changing based on ambient lighting conditions. In parallel, the SW 810 also programs a timer value 815 (e.g., Timer_value) in the DRV 830. This timer value 815 is programmed in the AHB interface 820. For proper functionality, the DRV timer value 815 programmed by the SW 810 should match the corresponding frame rate being transmitted by the sensor 870.

[0117] Sensor 870 and DRV 830 are fully asynchronous hardware, as are AHB interface 820 and CCI 860. Therefore, there is a high probability that a mismatch may exist between the DRV timer value 815 and the corresponding frame rate being transmitted by sensor 870. If such a mismatch occurs, the discrepancy between the DRV timer value 815 and the frame rate may result in image frame loss and hardware hang.

[0118] Figure 9 and Figure 10 The timing of DRV for the FRS use case is shown, where the DRV timer value and frame rate are synchronous and asynchronous, respectively. Specifically, Figure 9 This is a diagram illustrating an example of timing 900 for a camera using DRV in an FRS use case, where the DRV timer value and frame rate are synchronized. Figure 9The diagram shows the SOF timing 910, the vote-for timing 920, the EOF timing 930, the vote-for-no timing 940, and the vote level timing 950.

[0119] Under varying lighting conditions, the software programs DRV timer values ​​(e.g., T1, T2, and T3) for each of three frames (e.g., as shown in SOF timing 910). The timer values ​​(e.g., T1, T2, or T3) should precisely match the corresponding data rate or frame rate (e.g., FPS) being transmitted by the sensor. This matching ensures that a vote of approval (in vote of approval timing 920) occurs in time before the next SOF (e.g., in SOF timing 910) arrives.

[0120] exist Figure 9 In this process, for each of the three frames (e.g., represented by pulses in SOF timing 910), the FPS (e.g., the time between SOFs represented by pulses in SOF timing 910) remains variable. For example, the time between the first SOF and the second SOF is greater than the time between the second SOF and the third SOF. Therefore, T1 is greater than T2.

[0121] for Figure 9 In the example shown, timer 900 functions correctly. For example, the software will program T1, T2, and T3. When a timer (e.g., T1) expires, the corresponding vote in vote-for-grant timer 920 occurs. The vote level of vote level timer 950 will rise after the vote in vote-for-grant timer 920. When an EOF (e.g., a pulse) occurs in EOF timer 930, the corresponding vote-reject timer 940 will occur. The vote level of vote level timer 950 will fall after the vote-reject timer 940. In this example, T1 and T2 are correctly programmed such that the vote level of vote level timer 950 rises before the second SOF in SOF timer 910 arrives.

[0122] Figure 10 This is a diagram illustrating an example of using DRV for a camera's timing 1000 in an FRS use case, where the DRV timer value and frame rate are asynchronous. Figure 10 The diagram shows SOF timing 1010, voting approval timing 1020, EOF timing 1030, voting rejection timing 1040, and voting level timing 1050.

[0123] In this example, due to DRV (e.g., Figure 8 The DRV engine 830) and sensors (e.g., Figure 8Due to the asynchronous nature between the sensors (870), the DRV timer (e.g., T1) is mismatched with the corresponding frame rate of the sensor. This asynchronous nature causes a delay within the AHB programming. Therefore, timer T1 is also used for the second and third frames.

[0124] When the first timer (e.g., the first T1 in SOF timer 1010) expires, a vote of approval occurs in vote approval timer 1020. The vote level in vote level timer 1050 follows this vote of approval, and therefore the second frame in SOF timer 1010 can be received correctly.

[0125] However, the second timer (e.g., the second T1 in SOF timer 1010) expires later, causing the vote to pass after the arrival of the third frame in SOF timer 1010. When the third frame arrives in SOF timer 1010, the corresponding voting level of voting level timer 1050 is low, and therefore IFE and DDR are not ready to receive the third frame. Therefore, the third frame is discarded.

[0126] Furthermore, a second vote-agreement (e.g., a second pulse) occurs in the middle of the second frame of SOF timing 1010 during voting-agreement timing 1020. If the hardware (e.g., IFE) receives in the middle of a frame (e.g., receives an incomplete frame), this can lead to hardware violations and hardware hangs (e.g., fatal errors), and therefore the software will need to reset the hardware (e.g., IFE). Therefore, improved techniques for DRV of FSR for dynamic FPS use cases may be useful.

[0127] In one or more aspects, the system and technology provide DRV enhancements for dynamic FPS use cases. For example, in some examples, the system and technology employ timing indicators (such as a Pre-Start-of-Frame (PRE-SOF) indicator) to stabilize the voltage and clock levels of the ISP (e.g., the image front-end component) and DDR after an increase in the clock rate and power (e.g., due to DRV voting in favor of the ISP and DDR). The sensor (e.g., a camera's) transmits this timing indicator to the DRV engine. The DRV engine will vote in favor of the ISP and DDR based on the timing indicator. Because the sensor transmits the timing indicator to the DRV engine, both the DRV vote and the frame rate (e.g., the image frame rate) will be synchronized, which avoids the possibility of any mismatch between the DRV timer value and the corresponding frame rate. In one or more examples, the timing indicator (e.g., PRE-SOF) may appear before or after the start-of-frame (SOF) of multiple image frames being processed.

[0128] Figure 11This is a diagram illustrating an example of timing 1100 for a camera using DRV in an FRS use case, where a timing indicator (e.g., shown as PRE-SOF timing 1110) is employed. Figure 11 The diagram shows PRE-SOF timing 1110, SOF timing 1120, EOF timing 1130, voting decision timing 1140, and voting level timing 1150.

[0129] exist Figure 11 In the diagram, PRE-SOF timing 1110 is shown as including timing indicators (e.g., each PRE-SOF represented by a pulse in PRE-SOF timing 1110). Each PRE-SOF in PRE-SOF can be detected by a sensor (e.g., Figure 12 The sensor 1220 is transmitted via user-defined (UD) packets in the MIPI protocol. In one or more examples, each PRE-SOF may be transmitted prior to the SOF (e.g., represented by a pulse in SOF timing 1120). Each PRE-SOF may be used to vote in favor of the IFE and DDR voting levels (e.g., in voting level timing 1150).

[0130] In PRE-SOF timing 1110, PRE-SOF can be transmitted at a time T (e.g., 0.5 ms) before SOF (e.g., before the start of an image frame). This time T can be based on the amount of time required by resources (e.g., hardware, such as IFE and DDR) to ensure that the voltage and clock levels are stable in the hardware before the next frame (e.g., SOF) arrives. The value of time T can be programmed internally to the sensor and can be based on SOC requirements and the technical specifications of the specific SOC that has been manufactured.

[0131] In this example, both the DRV vote (e.g., via PRE-SOF) and the frame rate (e.g., determined by the exposure ratio) originate from the same source: the sensor. Therefore, the DRV vote and frame rate will be synchronized together. Since the DRV vote and frame rate are synchronized, there should be no mismatch between them. This synchronization should prevent any frame loss or hardware malfunction. Therefore, this timing 1100 allows DRV to be used in dynamic FPS use cases to save significant power.

[0132] Figure 12 This is a diagram illustrating an example of a system 1200 using DRV for an FSR (Functional System Responsibility) for dynamic FPS use cases, where the system employs timing including timing indicators (e.g., PRE-SOF 1231). Figure 12In the diagram, system 1200 is shown to include SW 1210, DRV engine 1240 (e.g., a voting engine), IFE 1250 (e.g., a front-end component of a camera), DDR 1260 (e.g., memory), sensor 1220, and camera serial interface (CSI) decoder 1230. In one or more examples, DRV engine 1240, IFE 1250, DDR 1260, sensor 1220, and / or CSI decoder 1230 may be implemented on a SoC.

[0133] During operation of system 1200, sensor 1220 can acquire image frames of the captured scene. Depending on varying lighting conditions, SW 1210 can dynamically program the exposure ratio 1215 (e.g., Exp_ratio) within sensor 1220, which will change the frame rate (e.g., FPS). Sensor 1220 can (e.g., via CSI decoder 1230) transmit (e.g., send) a timing indicator 1231 (e.g., PRE-SOF) to DRV engine 1240 for synchronization of DRV engine 1240 and the frame rate (e.g., FPS) used to process the image frames acquired by sensor 1220. In one or more examples, sensor 1220 may transmit a timing indicator 1231 (e.g., (SOF)) at Tms preceding each image frame in image frame 1221. The timing indicator 1231 (e.g., PRE-SOF) may be transmitted via PRE-SOF packets (such as UD packets with a specific data type (DT)). CSI decoder 1230 may use the DT defined within MIPI to distinguish whether the data it receives from sensor 1220 is the timing indicator 1231 (e.g., PRE-SOF) or the actual image (e.g., image frame). Therefore, when CSI decoder 1230 receives the timing indicator 1231 (e.g., PRE-SOF), CSI decoder 1230 may direct the timing indicator 1231 to DRV engine 1240.

[0134] Once the DRV engine 1240 receives the timing indicator 1231 (e.g., PRE-SOF), the DRV engine 1240 can vote in favor of IFE 1250 and DDR 1260. Therefore, the DRV engine 1240 can determine the vote in favor (e.g., affirmative vote) based on the timing indicator 1231 (e.g., PRE-SOF). The clock rate and voltage of the power supply shared by IFE 1250 and DDR 1260 can be increased based on the vote in favor (e.g., affirmative vote) to produce an updated clock rate and updated voltage. The updated clock rate and updated voltage can then be applied to IFE 1250 and DDR 1260.

[0135] Sensor 1220 can transmit (e.g., send) the SOF and remaining image frames to CSI 1230 decoder. CSI decoder 1230 can transmit (e.g., send) the SOF and remaining image frames to IFE 1250. IFE 1250 can process the image frames to produce processed image data. IFE 1250 can transmit (e.g., send) the processed image data (e.g., processed image frames) to DDR 1260. DDR 1260 (e.g., memory) can store the processed image data (e.g., processed image frames). Since IFE 1250 and DDR 1260 have voted in favor, IFE 1250 and DDR 1260 are respectively ready to receive image frames and processed image data.

[0136] In one or more aspects, the time between the timing indicator 1231 (e.g., PRE-SOF) and the start of the frame (e.g., SOF) can be configurable. Figure 13A and Figure 13B Examples of timing indicators appearing before and after the start of an image frame (e.g., SOF) are shown respectively. Specifically, Figure 13A This is an example of a timing indicator (shown as T) used for timing cameras in an FSR for dynamic FPS use cases, where DRV is used. PRE-SOF Example 1300 (1302) illustrates a timing indicator that appears at the beginning of an image frame (shown as T). SOF Time T prior to 1304. At the start of the use case, SW (e.g., Figure 12 The SW1210 can be configured with sensors (e.g., Figure 12 The value of time T within the sensor 1220. This time T is the time before the sensor should transmit (e.g., send) the PRE-SOF packet before SOF. Time T can be based on the updated clock rate and the updated clock voltage in the IFE (e.g., Figure 12 IFE 1250) and DRV (e.g., Figure 12 The amount of time required for stabilization in the DRV engine (1260).

[0137] Dynamically, between image frames, SW can change the exposure ratio (e.g., FPS) within the sensor. Based on the FPS, the sensor can calculate (e.g., using the formula: T) PRE-SOF =T SOF -T) When should SOF (or the next frame) be transmitted? The sensor can transmit at time T. PRE-SOF The PRE-SOF is transmitted at this time T. PRE-SOF The time T of SOF SOF The previous time T. Therefore, time T can be configured by SW.

[0138] Figure 13B This is an example of a timing indicator (shown as T) used for timing cameras in an FSR for dynamic FPS use cases, where DRV is used. PRE-SOF Example 1310 (1314) illustrates a timing indicator that appears at the beginning of an image frame (shown as T). SOF 1312) after time t. Dynamically, between image frames, SW (e.g., Figure 12 The SW 1210 can be programmed to set the exposure ratio and timer value (e.g., equal to time t) within the sensor (e.g., sensor 1220). Time t can be based on the lighting conditions of the scene in the image frame acquired by the sensor. The timer value (e.g., equal to time t) starts when the sensor transmits the SOF of the current frame. When the timer value expires, the sensor can transmit the PRE-SOF (e.g., via UD packets).

[0139] This enhanced DRV mechanism, using timing indicators (e.g., PRE-SOF), can be reliably used with dynamic FPS use cases to provide significant power savings, such as... Figure 14A and Figure 14B As shown. Due to the use of timing indicators (e.g., PRE-SOF), power consumption may increase slightly (e.g., 1.3%). However, this slight increase in power will be offset by the very large power savings achieved in the sensor, IFE, and DDR (e.g., due to the enhanced DRV mechanism). Figure 14A and Figure 14B An example of reducing power by employing a timing indicator (e.g., PRE-SOF) is shown. Specifically, Figure 14A This is graph 1400, illustrating examples of different power requirements for different sensor readout rates when using DRV for FSR in dynamic FPS use cases. Figure 14A In the graph 1400, the x-axis represents the readout speed, and the y-axis represents the power in milliwatts (mW).

[0140] As shown in graph 1400 of Figure 14, the power consumption is approximately 1800mW during a normal use case with a read speed of 33ms. When the enhanced DRV mechanism is used in an FSR with twice the read speed (2X), the power consumption is reduced to approximately 1722ms. When the enhanced DRV mechanism is used in an FSR with four times the read speed (4X), the power consumption is further reduced to approximately 1585ms.

[0141] Figure 14B It is a summary Figure 14A Table 1410 shows the different power requirements for different sensor readout speeds. Figure 14BAs shown in Table 1410, when the read speed is doubled (e.g., 2X read speed), there may be approximately 4.33% power savings. When the read speed is 4X, there may be approximately 12% power savings.

[0142] Figure 15 This is a flowchart illustrating an example of process 1500 of DRV for FSR used in dynamic FPS use cases. Process 1500 may be executed by a computing device or by components or systems of the computing device (e.g., chipset, one or more processors (such as graphics processor, CPU, GPU, DSP, NPU, etc.), one or more memories, any components thereof, and / or other components or systems). The operation of process 1500 may be implemented on one or more processors (e.g., Figure 16 Software components that execute and run on a processor (such as the 1610 or other processors).

[0143] At box 1510, the computing device (or a component thereof) may output a timing indicator for synchronization of the voting engine and frame rate (e.g., Figure 11 PRE-SOF timing 1110, Figure 12 The timing indicator 1231 Figure 13A T Pre-SOF 1302 Figure 13B T Pre-SOF (1314 or other timing indicators), this frame rate is used to process image frames acquired by the sensor. In some cases, the computing device (or a component thereof) can acquire image frames of the captured scene from the sensor. In some aspects, the computing device may include a sensor and a voting engine. For example, the sensor of the computing device may acquire (e.g., capture) image frames of the captured scene, and / or may send timing indicators to the voting engine. In some examples, the timing indicators are sent (or output for transmission) via user-defined (UD) packets.

[0144] In some respects, the timing indicator appears before the start of an image frame. For example, as... Figure 13A As shown, the timing indicator (shown as T) is used for timing the camera. PRE-SOF 1302) appears at the beginning of the image frame (shown as T). SOF The timing indicator appears at time T before the start of the image frame (1304). In some respects, the timing indicator appears at a time after the start of the image frame. For example, as... Figure 13B As shown, the timing indicator (shown as T) is used for timing the camera. PRE-SOF 1314) appears at the beginning of an image frame (e.g., shown as T). SOFAt time t after (1312). In some cases, time is based on the amount of time required for the updated clock rate and updated voltage to stabilize in the image processor (e.g., the computing device) and at least one memory (e.g., the computing device). In some cases, time is based on the lighting conditions of the scene in the image frame.

[0145] At box 1520, the computing device (or a component thereof) can obtain a positive vote result from the voting engine based on a timing indicator. For example, the voting engine can determine the positive vote result based on a timing indicator. In some cases, the voting engine is a Dynamic Resource Voting (DRV) engine (e.g., Figure 12 (DRV 1240).

[0146] At box 1530, the computing device (or a component thereof) may, based on a positive voting result, increase the clock rate and voltage of the power supply shared by the image processor and at least one memory of the computing device to produce an updated clock rate and updated voltage. In some aspects, the computing device may include an image processor. In some cases, the image processor is a front-end component of a camera (e.g., an image front-end (IFE), such as... Figure 12 (IFE 1250). In some examples, at least one memory is double data rate (DDR) memory (e.g., Figure 12 (DDR 1260).

[0147] At box 1540, the computing device (or a component thereof) can apply an updated clock rate and an updated voltage to the image processor and at least one memory. For example, as Figure 12 As shown, the DRV engine 1240 can receive a timing indicator 1231 and can vote in favor of the IFE 1250 and DDR 1260 based on the timing indicator 1231. Based on the vote in favor (e.g., affirmative vote result), the clock rate and voltage of the power supply shared by the IFE 1250 and DDR 1260 can be increased to produce an updated clock rate and updated voltage. The updated clock rate and updated voltage can then be applied to the IFE 1250 and DDR 1260.

[0148] In some respects, an image processor can process image frames to produce processed image data. In such respects, the image processor can output the processed image data to at least one memory. In some cases, the image processor and at least one memory are on a system-on-a-chip (SoC). For example, as per [reference to...] Figure 12As described, sensor 1220 can transmit (e.g., send) the SOF and remaining image frames to CSI 1230 decoder, which in turn can transmit (e.g., send) the SOF and remaining image frames to IFE 1250. IFE 1250 can process the image frames to produce processed image data. IFE 1250 can transmit (e.g., send) the processed image data (e.g., processed image frames) to DDR 1260. DDR 1260 (e.g., memory) can store the processed image data (e.g., processed image frames). Because IFE 1250 and DDR 1260 were previously voted in favor, IFE 1250 and DDR 1260 are respectively ready to receive image frames and processed image data.

[0149] In some examples, process 1500 may be executed by one or more computing devices or apparatuses. In some exemplary examples, process 1500 may be executed by... Figure 1 Image capture and processing system 100 Figure 3 Equipment 300 Figure 12 System 1200 and / or one or more computing devices or systems (e.g., Figure 16 The computing device or apparatus (1600) executes the process. In some cases, such a computing device or apparatus may include a processor, microprocessor, microcomputer, or other components of a device configured to perform the steps of process 1500. In some examples, such a computing device or apparatus may include one or more sensors configured to capture image data. For example, the computing device may include a smartphone, head-mounted display, mobile device, camera, tablet computer, or other suitable device. In some examples, such a computing device or apparatus may include a camera configured to capture one or more images or videos. In some cases, such a computing device may include a display for displaying images. In some examples, one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such a computing device may further include a network interface configured to transmit data.

[0150] Components that enable the implementation of a computing device in a circuit. For example, a component may include electronic circuitry or other electronic hardware, and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, graphics processing unit (GPU), digital signal processor (DSP), central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein, and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein. The computing device may also include a display (as an example of an output device or as a supplement to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0151] Process 1500 is illustrated as a logic flowchart, whose operations represent a sequence of operations that can be implemented by hardware, computer instructions, or combinations thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which the operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the process.

[0152] Additionally, process 1500 can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes jointly on one or more processors. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0153] Figure 16 This is a block diagram illustrating an example of a computing system 1600 that can be employed by a system disclosed for DRV enhancements targeting dynamic FPS use cases. Specifically, Figure 16An example of computing system 1600 is illustrated. This computing system can be any computing device, such as constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using connection 1605. Connection 1605 can be a physical connection using a bus, or a direct connection to processor 1610, such as in a chipset architecture. Connection 1605 can also be a virtual connection, a networking connection, or a logical connection.

[0154] In some aspects, computing system 1600 is a distributed system in which the functions described herein can be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a plurality of such components, each of which performs some or all of the functions described for that component. In some aspects, the components can be physical or virtual devices.

[0155] Example system 1600 includes at least one processing unit (CPU or processor) 1610 and a connection 1605 that communicatively couples various system components, including system memories 1615 such as read-only memory (ROM) 1620 and random access memory (RAM) 1625, to processor 1610. Computing system 1600 may include a cache 1612 of high-speed memory that is directly connected to, closely proximate to, or integrated into processor 1610.

[0156] Processor 1610 may include any general-purpose processor and hardware or software services (such as services 1632, 1634, and 1636 stored in storage device 1630 and configured to control processor 1610), as well as dedicated processors in which software instructions are incorporated into the actual processor design. Processor 1610 may be a substantially completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0157] To enable user interaction, the computing system 1600 includes an input device 1645 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice input, etc. The computing system 1600 may also include an output device 1635 that can be one or more of a plurality of output mechanisms. In some instances, a multi-mode system allows a user to provide multiple types of input / output to communicate with the computing system 1600.

[0158] The computing system 1600 may include a communication interface 1640, which typically controls and manages user input and system output. The communication interface may perform or facilitate the receiving and / or transmitting of wired or wireless communications using wired and / or wireless transceivers, including utilizing audio jacks / plugs, microphone jacks / plugs, Universal Serial Bus (USB) ports / plugs, Apple... ™ Lightning ™ Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, dedicated wired ports / plugs, 3G, 4G, 5G and / or other cellular data network wireless signal transmission, Bluetooth ™ Wireless signal transmission, Bluetooth ™ Low-power (BLE) wireless signal transmission, IBEACON ™ Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), microwave access global interoperability (WiMAX), infrared (IR) wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, self-organizing network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or those communications in some combination thereof.

[0159] The communication interface 1640 may also include one or more distance sensors (e.g., light-based sensors, radio frequency (RF)-based sensors, ultrasonic sensors, and infrared (IR) sensors) configured to collect data and provide measurements to the processor 1610, thereby configuring the processor 1610 to perform determinations and calculations required to obtain various measurements from the one or more distance sensors. In some examples, measurements may include time of flight, wavelength, azimuth, elevation, distance, linear velocity, and / or angular velocity, or any combination thereof. The communication interface 1640 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the position of the computing system 1600 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the U.S. GPS, the Russian GLONASS, the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS. There are no limitations on operation on any particular hardware arrangement, and therefore the basic features here can be easily replaced to obtain improved hardware or firmware arrangements as they are developed.

[0160] Storage device 1630 may be a non-volatile and / or non-transitory and / or computer-readable storage device, and may be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as magnetic tape, flash memory cards, solid-state storage devices, digital versatile discs, cartridges, floppy disks, hard disks, magnetic tapes, magnetic stripes, any other magnetic storage media, flash memory, memristor memory, any other solid-state storage, CD-ROM, rewritable CD, digital video disc (DVD), Blu-ray disc (BDD), holographic disc, another optical medium, secure digital card (SD card), micro secure digital card (microSD card), Memory Stick. ®Cards, smart card chips, EMV chips, Subscriber Identity Module (SIM) cards, mini / micro / nano / micro SIM cards, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM, cache memory (e.g., layer 1 (L1) cache, layer 2 (L2) cache, layer 3 (L3) cache, layer 4 (L4) cache, layer 5 (L5) cache, or other (L#) cache), resistive random access memory (RRAM / ReRAM), phase change memory (PCM), spin-transfer torque RAM (STT-RAM), another memory chip or cassette and / or combinations thereof.

[0161] Storage device 1630 may include software services, servers, services, etc., which enable the system to perform functions when the code defining such software is executed by processor 1610. In some aspects, hardware services performing specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as processor 1610, connection 1605, output device 1635, etc. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data can be stored and which does not include carrier waves and / or transient electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., may be passed, forwarded, or transmitted via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0162] Specific details have been provided in the foregoing description to offer a thorough understanding of the aspects and examples presented herein, but those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative aspects of this application have been described in detail herein, it is to be understood that the various inventive concepts may be embodied and employed in various other ways, and the appended claims are not intended to be construed as including these variations unless limited by prior art. The various features and aspects of the applications described above may be used individually or in combination. Furthermore, without departing from the broader scope of this specification, aspects may be used in any number of environments and applications beyond those described herein. Therefore, the specification and drawings should be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that, in alternative aspects, the methods may be performed in a different order than described.

[0163] For clarity, in some instances, this technology may be presented as comprising various functional blocks, which include devices, device components, steps, or routines embodied in a method, either in software or a combination of hardware and software. Additional components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring these aspects in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the aspects.

[0164] Furthermore, those skilled in the art will understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this disclosure.

[0165] Various aspects described above can be presented as processes or methods, depicted as flowcharts, diagrams, data flow graphs, structure diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations within an operation can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but a process may have additional steps not included in the accompanying diagrams. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0166] The processes and methods described in the examples above can be implemented using stored computer-executable instructions or computer-executable instructions otherwise obtainable from a computer-readable medium. Such instructions may include, for example, instructions and data that configure, or otherwise configure, a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. The portion may be accessible via a network of the computer resources used. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, or source code. Examples of computer-readable media that can be used to store the instructions, the information used, and / or information created during the methods according to the described examples include disks or optical discs, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0167] In some respects, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0168] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the above description may, in some cases, be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or light particles, or any combination thereof, depending in part on the specific application, in part on the desired design, in part on the corresponding technology, etc.

[0169] The various exemplary logic blocks, modules, and circuits described in conjunction with the aspects disclosed herein may be implemented or performed using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any form factor of various form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Examples of form factors include: laptop computers, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, self-contained devices, etc. The functionality described herein may also be embodied in peripheral devices or interlocking cards. By further example, such functionality may also be implemented on circuit boards of different chips or different processes executed on a single device.

[0170] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0171] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods, algorithms, and / or operations described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0172] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein.

[0173] Those skilled in the art will understand that, without departing from the scope of this description, the less than (“<”) and greater than (“>”) symbols or terms used herein may be replaced with less than or equal to (“>”) respectively. ") and greater than or equal to (" The symbol ) is used instead.

[0174] When a component is described as being “configured” to perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., microprocessors or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0175] The phrase “coupled to” or “communicatively coupled to” means that any component is physically connected directly or indirectly to another component, and / or that any component is in communication with another component directly or indirectly (e.g., connected to that other component via a wired or wireless connection and / or other suitable communication interface).

[0176] Claim language or other languages ​​that state "at least one of" and / or "one or more of" in a set indicate that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language stating "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language stating "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, A and B and C, or any repeating information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" and / or "one or more of" in a set does not limit the set to the items listed in the set. For example, the language of a claim stating "at least one of A and B" or "at least one of A or B" may refer to A, B, or A and B, and may additionally include items not listed in the set of A and B. The phrases "at least one" and "one or more" are used interchangeably herein.

[0177] Claims using phrases such as "at least one processor, the at least one processor being configured to," "at least one processor being configured to," "one or more processors, the one or more processors being configured to," or "one or more processors being configured to," or other languages, indicate that one or more processors (in any combination) are capable of performing associated operations. For example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" means that a single processor can be used to perform operations X, Y, and Z; or that multiple processors are each assigned a specific subset of tasks to perform operations X, Y, and Z, such that the multiple processors together perform X, Y, and Z; or that a group of multiple processors work together to perform operations X, Y, and Z. In another example, a claim using the phrase "at least one processor, the at least one processor being configured to: X, Y, and Z" could mean that any single processor can perform only at least one subset of operations X, Y, and Z.

[0178] When referring to one or more elements that perform functions (e.g., steps of a method), one element may perform all functions, or more than one element may jointly perform these functions. When more than one element jointly performs these functions, each function does not need to be performed by every single element (e.g., different functions may be performed by different elements), and / or each function does not need to be performed by only one element as a whole (e.g., different elements may perform different sub-functions of a function). Similarly, when referring to one or more elements configured to cause another element (e.g., a device) to perform functions, one element may be configured to cause another element to perform all functions, or more than one element may be jointly configured to cause another element to perform these functions.

[0179] When referring to an entity that performs or is configured to perform functions (e.g., steps of a method) (e.g., any entity or device described herein), the entity may be configured to cause one or more elements (individually or collectively) to perform those functions. One or more components of the entity may include at least one memory, at least one processor, at least one communication interface, another component configured to perform one or more of those functions, and / or any combination thereof. When referring to an entity that performs functions, the entity may be configured to cause one component to perform all functions, or to cause more than one component to perform those functions collectively. When the entity is configured to cause more than one component to perform those functions collectively, each function does not need to be performed by every single component (e.g., different functions may be performed by different components), and / or each function does not need to be performed by only one component as a whole (e.g., different components may perform different sub-functions of a function).

[0180] The various exemplary logic blocks, modules, engines, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, engines, modules, circuits, and steps have been broadly described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this application.

[0181] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices (mobile phones), or integrated circuit devices with multiple uses, including applications in wireless communication devices (mobile phones) and other devices. Any feature described as an engine, module, or component can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium can form part of a computer program product, which may include packaging material. The computer-readable medium may include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or conveys program code in the form of instructions or data structures that can be accessed, read and / or executed by a computer, such as propagated signals or waves.

[0182] The program code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in alternatives, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or means suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0183] The exemplary aspects of this disclosure include:

[0184] Aspect 1. An apparatus for image processing, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to: output a timing indicator for synchronization of a voting engine and a frame rate for processing image frames acquired by a sensor; obtain a positive voting result from the voting engine based on the timing indicator; increase a clock rate and voltage of a power supply shared by the image processor and the at least one memory based on the positive voting result to generate an updated clock rate and an updated voltage; and apply the updated clock rate and the updated voltage to the image processor and the at least one memory.

[0185] Aspect 2. The apparatus according to aspect 1, wherein the timing indicator is output for transmission via user-defined (UD) packets.

[0186] Aspect 3. The apparatus according to any one of Aspect 1 or 2, wherein the timing indicator appears at a time before the start of the image frame of the image frame.

[0187] Aspect 4. The apparatus according to aspect 3, wherein the time is based on the amount of time required for the updated clock rate and the updated voltage to stabilize in the image processor and the at least one memory.

[0188] Aspect 5. The apparatus according to any one of Aspects 1 to 4, wherein the timing indicator appears at a time after the start of the image frame of the image frame.

[0189] Aspect 6. The apparatus according to aspect 5, wherein the time is based on the lighting conditions of the scene of the image frame.

[0190] Aspect 7. The apparatus according to any one of Aspects 1 to 6, wherein the at least one processor is configured to obtain the image frame of the captured scene from the sensor.

[0191] Aspect 8. The apparatus according to any one of Aspects 1 to 7, wherein the image processor is a front-end component of a camera.

[0192] Aspect 9. The apparatus according to any one of Aspects 1 to 8, wherein the at least one memory is a double data rate (DDR) memory.

[0193] Aspect 10. The apparatus according to any one of Aspects 1 to 9, wherein the voting engine is a Dynamic Resource Voting (DRV) engine.

[0194] Aspect 11. The apparatus according to any one of Aspects 1 to 10, wherein the at least one processor includes the image processor.

[0195] Aspect 12. The apparatus according to aspect 11, wherein the image processor is configured to process the image frame to generate processed image data.

[0196] Aspect 13. The apparatus according to aspect 12, wherein the image processor is configured to output the processed image data to the at least one memory.

[0197] Aspect 14. The apparatus according to any one of Aspects 11 to 13, wherein the image processor and the at least one memory are on a system-on-a-chip (SoC).

[0198] Aspect 15. The apparatus according to any one of Aspects 1 to 14, the apparatus further comprising the sensor and the voting engine.

[0199] Aspect 16. A method for image processing, the method comprising: sending a timing indicator from a sensor to a voting engine for synchronization of the voting engine and a frame rate, the frame rate being used to process image frames acquired by the sensor; determining a positive voting result by the voting engine based on the timing indicator; increasing a clock rate and voltage of a power supply shared by an image processor and a memory based on the positive voting result to generate an updated clock rate and an updated voltage; and applying the updated clock rate and the updated voltage to the image processor and the memory.

[0200] Aspect 17. The method according to aspect 16, the method further comprising processing the image frame by the image processor to generate processed image data.

[0201] Aspect 18. The method according to aspect 17, the method further comprising sending the processed image data to the memory by the image processor.

[0202] Aspect 19. The method according to any one of aspects 16 to 18, wherein the timing indicator is transmitted via a user-defined (UD) packet.

[0203] Aspect 20. The method according to any one of aspects 16 to 19, wherein the timing indicator appears at a time before the start of the image frame of the image frame.

[0204] Aspect 21. The method according to aspect 20, wherein the time is based on the amount of time required for the updated clock rate and the updated voltage to stabilize in the image processor and the memory.

[0205] Aspect 22. The method according to any one of aspects 16 to 21, wherein the timing indicator appears at a time after the start of the image frame of the image frame.

[0206] Aspect 23. The method according to aspect 22, wherein the time is based on the lighting conditions of the scene of the image frame.

[0207] Aspect 24. The method according to any one of aspects 16 to 23, the method further comprising obtaining the image frame of the captured scene by the sensor.

[0208] Aspect 25. The method according to any one of Aspects 16 to 24, wherein the image processor is a front-end component of a camera.

[0209] Aspect 26. The method according to any one of Aspects 16 to 25, wherein the memory is a double data rate (DDR) memory.

[0210] Aspect 27. The method according to any one of Aspects 16 to 26, wherein the voting engine is a Dynamic Resource Voting (DRV) engine.

[0211] Aspect 28. The method according to any one of Aspects 16 to 27, wherein the image processor and the memory are on a system-on-a-chip (SoC).

[0212] Aspect 29. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform any one of aspects 16 to 28.

[0213] Aspect 30. An apparatus for image processing, the apparatus comprising one or more components for performing operations according to any one of aspects 16 to 28.

[0214] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but are to be consistent with the full scope of the language claims, wherein an element referred to in the singular is not intended to mean "one and only one," but rather "one or more" unless specifically stated otherwise.

Claims

1. An apparatus for image processing, the apparatus comprising: At least one memory; and At least one processor, the at least one processor being coupled to the at least one memory and being configured to: Output a timing indicator for synchronizing the voting engine and the frame rate, which is used to process image frames acquired by the sensor; The positive voting result is obtained from the voting engine based on the timing indicator; Based on the affirmative voting results, the clock rate and voltage of the power supply shared by the image processor and the at least one memory are increased to generate an updated clock rate and updated voltage. as well as The updated clock rate and the updated voltage are applied to the image processor and the at least one memory.

2. The apparatus of claim 1, wherein the timing indicator is output for transmission via user-defined (UD) packets.

3. The apparatus of claim 1, wherein the timing indicator appears at a time before the start of the image frame of the image frame.

4. The apparatus of claim 3, wherein the time is based on the amount of time required for the updated clock rate and the updated voltage to stabilize in the image processor and the at least one memory.

5. The apparatus of claim 1, wherein the timing indicator appears at a time after the start of the image frame of the image frame.

6. The apparatus of claim 5, wherein the time is based on the lighting conditions of the scene of the image frame.

7. The apparatus of claim 1, wherein the at least one processor is configured to obtain the image frame of the captured scene from the sensor.

8. The apparatus of claim 1, wherein the image processor is a front-end component of a camera.

9. The apparatus of claim 1, wherein the at least one memory is a double data rate (DDR) memory.

10. The apparatus of claim 1, wherein the voting engine is a Dynamic Resource Voting (DRV) engine.

11. The apparatus of claim 1, wherein the at least one processor comprises the image processor.

12. The apparatus of claim 11, wherein the image processor is configured to process the image frame to generate processed image data.

13. The apparatus of claim 12, wherein the image processor is configured to output the processed image data to the at least one memory.

14. The apparatus of claim 11, wherein the image processor and the at least one memory are on a system-on-a-chip (SoC).

15. The apparatus of claim 1, further comprising the sensor and the voting engine.

16. A method for image processing, the method comprising: The sensor sends a timing indicator to the voting engine for synchronization between the voting engine and the frame rate, which is used to process image frames acquired by the sensor. The voting engine determines the affirmative voting result based on the timing indicator; Based on the affirmative voting results, the clock rate and voltage of the power supply shared by the image processor and memory are increased to produce an updated clock rate and updated voltage. as well as The updated clock rate and the updated voltage are applied to the image processor and the memory.

17. The method of claim 16, further comprising processing the image frame by the image processor to generate processed image data.

18. The method of claim 17, further comprising sending the processed image data to the memory by the image processor.

19. The method of claim 16, wherein the timing indicator is transmitted via a user-defined (UD) packet.

20. The method of claim 16, wherein the timing indicator appears at a time prior to the start of the image frame of the image frame.

21. The method of claim 20, wherein the time is based on the amount of time required for the updated clock rate and the updated voltage to stabilize in the image processor and the memory.

22. The method of claim 16, wherein the timing indicator appears at a time after the start of the image frame of the image frame.

23. The method of claim 22, wherein the time is based on the lighting conditions of the scene of the image frame.

24. The method of claim 16, further comprising obtaining the image frame of the captured scene by the sensor.

25. The method of claim 16, wherein the image processor is a front-end component of a camera.

26. The method of claim 16, wherein the memory is a double data rate (DDR) memory.

27. The method of claim 16, wherein the voting engine is a Dynamic Resource Voting (DRV) engine.

28. The method of claim 16, wherein the image processor and the memory are on a system-on-a-chip (SoC).

Images