Circuit for performing optical image stabilization prior to lens shading correction

Abstract

An image processing circuit for performing optical image stabilization prior to lens shading correction. The image processing circuitry determines, for each row of pixels in an image captured by an image sensor receiving light through a lens having an optical center having a first offset in a first direction and a second offset in a second direction orthogonal to the first direction, the first offset and coordinates of the row of pixels in the second direction using the first offset and coordinates of the row of pixels in the second direction The first mesh is offset in the first direction. The image processing circuitry determines, for each row of pixels in the image, a second grid offset in the second direction using the second offset and the coordinates of the row of pixels in the second direction. The image processing circuitry generates an adjusted grid of the pixels of the image using the first grid offset and the second grid offset.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 112,808, filed February 22, 2023, which is incorporated by reference herein in its entirety. BACKGROUND TECHNICAL FIELD

[0003] The present disclosure relates to circuitry for processing image data, and more specifically, to circuitry for performing optical image stabilization.

[0004] Description of Related Art

[0005] Image data captured by an image sensor or received from other data sources is often processed in an image processing pipeline before further processing or consumption. For example, raw image data can be corrected, filtered, or otherwise modified before being provided to a subsequent component, such as a video encoder. To perform corrections or enhancements on captured image data, various components, units, or modules can be employed.

[0006] Such image processing pipelines can be structured to enable corrections or enhancements on captured image data to be performed in a favorable manner without consuming other system resources. While many image processing algorithms can be performed by executing software programs on a central processing unit (CPU), executing these programs on the CPU would consume a large amount of bandwidth of the CPU and other peripheral resources as well as increase power consumption. Thus, image processing pipelines are often implemented as hardware components separate from the CPU and dedicated to performing one or more image processing algorithms.

[0007] Image processing pipelines often include lens shading correction, which represents applying a gain per pixel to compensate for an intensity drop that is approximately proportional to the distance from the optical center of the lens. However, a shift can occur between the optical center of the lens and the center axis of an image sensor that captures light passing through the lens. This shift can cause an inappropriate gain to be applied to the lens shading correction, resulting in a shift of the lens shading correction and a brightness flicker in the captured image frame. SUMMARY

[0008] Implementations relate to an image processing circuit for performing optical image stabilization prior to lens shading correction. The image processing circuit includes a first processor circuit, a second processor circuit, and a grid adjustment circuit coupled to the first processor circuit and the second processor circuit. The first processor circuit determines, for each row of pixels of an image captured by an image sensor that receives light passing through a lens including an optical center having a first offset along a first direction and a second offset along a second direction orthogonal to the first direction, a first grid offset along the first direction using the first offset and a coordinate of the row of pixels along the second direction. The second processor circuit determines, for each row of pixels of the image, a second grid offset along the second direction using the second offset and the coordinate of the row of pixels along the second direction. The grid adjustment circuit generates adjusted grids of the pixels of the image using the first grid offset and the second grid offset. BRIEF DESCRIPTION OF DRAWINGS

[0009] FIG. 1 is a high-level diagram of an electronic device in accordance with an implementation.

[0010] FIG. 2 is a block diagram illustrating components in an electronic device in accordance with an implementation.

[0011] FIG. 3 is a block diagram illustrating an image processing pipeline implemented using an image signal processor in accordance with an implementation.

[0012] FIG. 4 is a block diagram illustrating a raw processing stage with optical compensation circuit coupled to a lens shading correction circuit in accordance with an implementation.

[0013] FIG. 5 is a block diagram illustrating a detailed view of the optical compensation circuit in FIG. 4 in accordance with an implementation.

[0014] FIG. 6 is a conceptual diagram illustrating optical image stabilization compensation performed by the optical compensation circuit in FIG. 4 in accordance with an implementation.

[0015] FIG. 7 is a conceptual diagram illustrating grid adjustment for three rows of pixels as part of the optical image stabilization compensation performed by the optical compensation circuit in FIG. 4 in accordance with an implementation.

[0016] FIG. 8 is a flow diagram illustrating a method of performing optical image stabilization prior to lens shading correction in accordance with an implementation.

[0017] The accompanying drawings depict and the detailed description describes various non-limiting embodiments, for purposes of illustration only. DETAILED DESCRIPTION

[0018] Reference will now be made in detail to implementations, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described implementations. However, the described implementations can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the implementations.

[0019] Embodiments of the present disclosure relate to a circuit for compensation of optical image stabilization (OIS) drift, which is applied prior to lens shading correction of image data. OIS drift is the drift between the optical center of a lens in a camera device and the center axis of an image sensor in the camera device. Compensation of OIS drift can be performed prior to lens shading correction of image data to mitigate lens shading correction shift and brightness flicker in captured image frames. The optical center can be adjusted for each image frame by applying an appropriate grid offset as a function of pixel row position. The grid offset can be computed for each row of pixels along two spatial dimensions, e.g., along the x-axis and the y-axis. After applying the grid offset to all rows of pixels, gain interpolation can be applied as a function of pixel position (which is now adjusted by the grid offset) for lens shading correction.

[0020] Exemplary Electronic Device

[0021] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communications device, such as a mobile telephone, that also contains other functions, such as personal digital assistant (PDA) and / or music player functions. Exemplary embodiments of portable multifunction devices include, without limitation, the iPhone®, iPod touch®, and iPad® devices from Apple Inc. of Cupertino, California. Other portable electronic devices, such as wearable devices, laptop computers, or tablet computers, can also be used. iPod Apple devices, and devices. Other portable electronic devices, such as wearable devices, laptop computers, or tablet computers, can also be used. In some embodiments, the device is not a portable communications device, but is a desktop computer or other computing device that is not designed for portable use. In some embodiments, the disclosed electronic device can include a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). Some embodiments are described below with reference to specific FIG. 1The example electronic devices described (e.g., device 100) can include a touch-sensitive surface for receiving user input. These devices also can include one or more other physical user-interface devices, such as a physical keyboard, a mouse, and / or a joystick.

[0022] FIG. 1 is a high-level diagram of an electronic device 100 in accordance with an embodiment. The device 100 can include one or more physical buttons, such as a "home" or menu button 104. The menu button 104, for example, is used to navigate to any of a set of applications that are executed on the device 100. In some embodiments, the menu button 104 includes a fingerprint sensor that identifies a fingerprint on the menu button 104. The fingerprint sensor can be used to determine whether a finger on the menu button 104 has a fingerprint that matches a fingerprint stored as unlocking the device 100. Alternatively, in some embodiments, the menu button 104 is implemented as a soft key in a graphical user interface (GUI) displayed on a touch screen.

[0023] In some embodiments, the device 100 includes a touch screen 150, a menu button 104, a volume adjustment button 108, a subscriber identity module (SIM) card slot 110, a FIG. 1components not shown in FIG. 1A, such as an ambient light sensor, a dot projector, and a flood illuminator.

[0024] Device 100 is merely one example of an electronic device, and device 100 can have more or fewer components than listed above, some of which can be combined or have a different configuration or arrangement. Various components of device 100 listed above are embodied as hardware, software, firmware, or a combination thereof, including one or more signal processing and / or application specific integrated circuits (ASICs). While FIG. 1 Components in FIG. 1A are shown as being located generally on the same side as touch screen 150, but one or more components can also be located on an opposite side of device 100. For example, the front side of device 100 can include infrared image sensor 164 for facial recognition and another image sensor 164 as a front-facing camera of device 100. The back side of device 100 can also include two additional image sensors 164 as back-facing cameras of device 100.

[0025] FIG. 2 is a block diagram illustrating components of a device 100, in accordance with one embodiment. Device 100 can perform various operations including image processing. For this and other purposes, device 100 can include image sensors 202, system on a chip (SOC) components 204, system memory 230, persistent storage (e.g., flash memory) 228, motion sensors 234, and display 216, among other components. As FIG. 2 The illustrated components are merely exemplary. For instance, device 100 can include other components not illustrated in FIG. 1A, such as a speaker or microphone. Additionally, some components (such as motion sensors 234) can be omitted from device 100. FIG. 2 The illustrated components are merely exemplary. For instance, device 100 can include other components not illustrated in FIG. 1A, such as a speaker or microphone. Additionally, some components (such as motion sensors 234) can be omitted from device 100.

[0026] Image sensors 202 are components for capturing image data. Each of image sensors 202 can be embodied, for example, as a complementary metal-oxide-semiconductor (CMOS) active pixel sensor, a camera, a video camera, or other device. Image sensors 202 generate raw image data, which is sent to SOC components 204 for further processing. In some embodiments, the image data processed by SOC components 204 is displayed on display 216, stored in system memory 230, persistent storage 228, or sent to a remote computing device via a network connection. The raw image data generated by image sensors 202 can be in a Bayer color filter array (CFA) pattern (also referred to as “Bayer pattern” hereinafter). Image sensors 202 can also include optical and mechanical components that assist image-sensing components (e.g., pixels) in capturing images. The optical and mechanical components can include an aperture, a lens system, and an actuator that controls the focal length of image sensors 202.

[0027] The motion sensor 234 is a component or set of components for sensing motion of the device 100. The motion sensor 234 can generate sensor signals indicative of an orientation and / or acceleration of the device 100. The sensor signals are sent to the SOC component 204 for various operations, such as turning on the device 100 or rotating an image displayed on the display 216.

[0028] The display 216 is a component for displaying images generated by the SOC component 204. The display 216 can include, for example, a liquid crystal display (LCD) device or an organic light-emitting diode (OLED) device. Based on data received from the SOC component 204, the display 216 can display various images, such as a menu, a selected operating parameter, an image captured by the image sensor 202 and processed by the SOC component 204, and / or other information received from a user interface of the device 100 (not shown).

[0029] The system memory 230 is a component for storing instructions executed by the SOC component 204 and for storing data processed by the SOC component 204. The system memory 230 can be embodied as any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or a combination thereof. In some embodiments, the system memory 230 can store pixel data or other image data or statistical values in various formats.

[0030] The persistent storage 228 is a component for storing data in a non-volatile manner. The persistent storage 228 retains data even when power is unavailable. The persistent storage 228 can be embodied as read-only memory (ROM), flash memory, or other non-volatile random access memory devices.

[0031] The SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing processes. The SOC component 204 can include subcomponents such as an image signal processor (ISP) 206, a central processor unit (CPU) 208, a network interface 210, a motion sensor interface 212, a display controller 214, a graphics processor unit (GPU) 220, a memory controller 222, a video encoder 224, a storage controller 226, and various other input / output (I / O) interfaces 218, and a bus 232 connecting these subcomponents. The SOC component 204 can include more or less subcomponents than those shown in FIG. 2. FIG. 2 The SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing processes. The SOC component 204 can include subcomponents such as an image signal processor (ISP) 206, a central processor unit (CPU) 208, a network interface 210, a motion sensor interface 212, a display controller 214, a graphics processor unit (GPU) 220, a memory controller 222, a video encoder 224, a storage controller 226, and various other input / output (I / O) interfaces 218, and a bus 232 connecting these subcomponents. The SOC component 204 can include more or less subcomponents than those shown in FIG. 2.

[0032] The ISP 206 is hardware that performs stages of an image processing pipeline. In some embodiments, the ISP 206 can receive raw image data from the image sensor 202 and process the raw image data into a form usable by other sub-components of the SOC component 204 or components of the device 100. The ISP 206 can perform various image processing operations, such as image translation operations, horizontal and vertical scaling, color space conversion, and / or image stabilization transforms, as discussed below with reference to FIG. 3 The detailed description.

[0033] The CPU 208 can be implemented using any suitable instruction set architecture and can be configured to execute instructions defined in that instruction set architecture. The CPU 208 can be a general- or embedded-purpose processor using any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, RISC, ARM, or MIPS ISAs, or any other suitable ISA. Although illustrated as a single processor, the SOC component 204 can include multiple CPUs. In a multiprocessor system, each of the CPUs can collectively implement the same ISA, but this is not required. FIG. 2 The CPU 208 can be implemented using any suitable instruction set architecture and can be configured to execute instructions defined in that instruction set architecture. The CPU 208 can be a general- or embedded-purpose processor using any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, RISC, ARM, or MIPS ISAs, or any other suitable ISA. Although illustrated as a single processor, the SOC component 204 can include multiple CPUs. In a multiprocessor system, each of the CPUs can collectively implement the same ISA, but this is not required.

[0034] The GPU 220 is graphics processing circuitry for performing operations on graphics data. For example, the GPU 220 can render objects to be displayed into a frame buffer (e.g., a frame buffer including pixel data for an entire frame). The GPU 220 can include one or more graphics processors that can execute graphics software to perform some or all of the graphics operations or hardware acceleration of certain graphics operations.

[0035] The I / O interface 218 is hardware, software, firmware, or a combination thereof for interfacing with various input / output components in the device 100. The I / O components can include devices such as a keyboard, buttons, audio devices, and sensors such as a global positioning system. The I / O interface 218 handles data for sending to such I / O components or handles data received from these I / O components.

[0036] The network interface 210 is a sub-component that supports the exchange of data between the device 100 and other devices via one or more networks (e.g., carrier or proxy devices). For example, video or other image data can be received from other devices via the network interface 210 and stored in the system memory 230 for subsequent processing (e.g., via a back-end interface to the image signal processor 206 as discussed below in FIG. 3 The network can include, but is not limited to, a local area network (LAN) (e.g., an Ethernet or corporate network) and a wide area network (WAN). Image data received via the network interface 210 can be subjected to an image processing process by the ISP 206.

[0037] Motion sensor interface 212 is circuitry for interfacing with motion sensor 234. Motion sensor interface 212 receives sensor information from motion sensor 234 and processes the sensor information to determine an orientation or movement of device 100.

[0038] Display controller 214 is circuitry for sending image data to be displayed on display 216. Display controller 214 receives image data from ISP 206, CPU 208, a graphics processor, or system memory 230 and processes the image data into a format suitable for display on display 216.

[0039] Memory controller 222 is circuitry for communicating with system memory 230. Memory controller 222 can read data from system memory 230 for processing by ISP 206, CPU 208, GPU 220, or other subcomponents of SOC components 204. Memory controller 222 can also write data to system memory 230 received from various subcomponents of SOC components 204.

[0040] Video encoder 224 is hardware, software, firmware, or a combination thereof, for encoding video data into a format suitable for storage in persistent storage 228 or for transmission to another device over network interface 210 for transmission over a network to another device.

[0041] In some embodiments, one or more subcomponents of SOC components 204 or some functionality of these subcomponents can be performed by software components executing on ISP 206, CPU 208, or GPU 220. Such software components can be stored in system memory 230, persistent storage 228, or another device in communication with device 100 via network interface 210.

[0042] Image data or video data can flow through various data paths within SOC components 204. In one example, raw image data can be generated from image sensor 202 and processed by ISP 206, then sent to system memory 230 via bus 232 and memory controller 222. After the image data is stored in system memory 230, the image data can be accessed by video encoder 224 for encoding or by display 216 for display via bus 232.

[0043] In another example, image data is received from a source other than the image sensor 202. For example, video data may be streamed, downloaded, or otherwise transferred to the SOC component 204 via a wired or wireless network. The image data may be received via the network interface 210 and written to the system memory 230 via the memory controller 222. The image data may then be obtained from the system memory 230 by the ISP 206 and processed through one or more image processing pipeline stages, as described below with reference to FIG. FIG. 3 The image data may then be returned to system memory 230 or sent to video encoder 224 , display controller 214 (for display on display 216 ), or storage controller 226 for storage in persistent storage 228 .

[0044] Example Image Signal Processing Pipeline

[0045] FIG. 3 is a block diagram illustrating an image processing pipeline implemented using ISP 206 according to one embodiment. FIG. 3 In one embodiment, ISP 206 is coupled to image sensor system 201, which includes one or more image sensors 202A through 202N (hereinafter collectively referred to as "image sensors 202" or individually as "image sensors 202") to receive raw image data. Image sensor system 201 may include one or more subsystems that independently control image sensors 202. In some cases, each image sensor 202 may operate independently, while in other cases, image sensors 202 may share some components. For example, in one embodiment, two or more image sensors 202 may share a common circuit board that controls the mechanical components of the image sensors (e.g., an actuator that changes the focus of each image sensor). The image sensing components of image sensors 202 may include different types of image sensing components that may provide raw image data to ISP 206 in different forms. For example, in one embodiment, the image sensing components may include a plurality of focus pixels for autofocus and a plurality of image pixels for capturing images. In another embodiment, the image sensing pixels may be used for both autofocus and image capture purposes.

[0046] The ISP 206 implements an image processing pipeline that can include a set of stages that process image information from creation, capture, or reception to output. Among other components, the ISP 206 can include a sensor interface 302, a central control 320, front-end pipeline stages 330, back-end pipeline stages 340, an image statistics module 304, a vision module 322, a back-end interface 342, an output interface 316, and autofocus circuits 350A-350N (hereinafter referred to collectively as “autofocus circuits 350” or individually as “autofocus circuit 350”). The ISP 206 can include other components not shown in FIG. 3 or can omit one or more of the components shown in FIG. 3. FIG. 3 The ISP 206 can include other components not shown in FIG. 3 or can omit one or more of the components shown in FIG. 3. FIG. 3 The ISP 206 can include other components not shown in FIG. 3 or can omit one or more of the components shown in FIG. 3.

[0047] In one or more embodiments, different components of the ISP 206 process image data at different rates. In FIG. 3 In embodiments of the image sensor 202, the front-end pipeline stages 330 (e.g., raw processing stage 306 and resampling processing stage 308) can process image data at an initial rate. Thus, various different techniques, adjustments, modifications, or other processing operations are performed by these front-end pipeline stages 330 at the initial rate. For example, if the front-end pipeline stages 330 process two pixels per clock cycle, then raw processing stage 306 operations (e.g., black level compensation, highlight recovery, and defective pixel correction) can process two pixels of image data at a time. In contrast, one or more of the back-end pipeline stages 340 can process image data at a different rate that is less than the initial data rate. For example, in FIG. 3 In embodiments of the image sensor 202, the back-end pipeline stages 340 (e.g., noise processing stage 310, color processing stage 312, and output rescaling 314) can be processed at a reduced rate (e.g., one pixel per clock cycle).

[0048] Raw image data captured by the image sensor 202 can be transmitted to different components of the ISP 206 in different ways. In one embodiment, raw image data for focus pixels can be sent to the autofocus circuits 350, while raw image data corresponding to image pixels can be sent to the sensor interface 302. In another embodiment, raw image data corresponding to both types of pixels can be sent to both the autofocus circuits 350 and the sensor interface 302 at the same time.

[0049] The auto-focus circuit 350 can include hardware circuitry that analyzes raw image data to determine an appropriate focal length for each image sensor 202. In one embodiment, the raw image data can include data transferred from image sensing pixels dedicated for image focus. In another embodiment, raw image data from image capture pixels can also be used for auto-focus purposes. The auto-focus circuit 350 can perform various image processing operations to generate data that determines the appropriate focal length. The image processing operations can include cropping, merging, image compensation, scaling to generate data for auto-focus purposes. The auto-focus data generated by the auto-focus circuit 350 can be fed back to the image sensor system 201 to control the focal length of the image sensor 202. For example, the image sensor 202 can include control circuitry that analyzes the auto-focus data to determine command signals to send to actuators associated with the lens system of the image sensor 202 to change the focal length of the image sensor 202. The data generated by the auto-focus circuit 350 can also be sent to other components of the ISP 206 for other image processing purposes. For example, some data can be sent to the image statistics module 304 to determine information about auto-exposure.

[0050] The auto-focus circuit 350 can be a separate circuit from other components such as the image statistics module 304, the sensor interface 302, the front-end 330, and the back-end 340. This allows the ISP 206 to perform auto-focus analysis independent of other image processing pipelines. For example, the ISP 206 can analyze raw image data from the image sensor 202A to adjust the focal length of the image sensor 202A using the auto-focus circuit 350A while performing downstream image processing on image data from the image sensor 202B. In one embodiment, the number of auto-focus circuits 350 can correspond to the number of image sensors 202. In other words, each image sensor 202 can have a corresponding auto-focus circuit dedicated to auto-focus for the image sensor 202. The device 100 can perform auto-focus for different image sensors 202 even if one or more image sensors 202 are not in active use. This allows for seamless transitions between two image sensors 202 when the device 100 switches from one image sensor 202 to another. For example, in one embodiment, the device 100 can include a wide-angle camera and a telephoto camera as a dual-rear camera system for photos and image processing. The device 100 can display an image captured by one of the dual cameras and can switch between the two cameras from time to time. The displayed image can seamlessly transition from image data captured by one image sensor 202 to image data captured by another image sensor 202 without having to wait for the second image sensor 202 to adjust its focal length because two or more auto-focus circuits 350 can continuously provide auto-focus data to the image sensor system 201.

[0051] Raw image data captured by different image sensors 202 can also be transmitted to the sensor interface 302. The sensor interface 302 receives raw image data from the image sensors 202 and processes the raw image data into image data that can be processed by other stages in the pipeline. The sensor interface 302 can perform various pre-processing operations such as image cropping, merging, or scaling to reduce image data size. In some embodiments, pixels are sent from the image sensors 202 to the sensor interface 302 in a raster order (e.g., horizontally, row by row). Subsequent processing in the pipeline can also be performed in a raster order, and results can also be output in a raster order. Although FIG. 3 Only a single image sensor 201 and a single sensor interface 302 are shown in FIG. 3, but when more than one image sensor is provided in the device 100, a corresponding number of sensor interfaces can be provided in the ISP 206 to process raw image data from each image sensor.

[0052] The front-end pipeline stage 330 processes image data in raw or full color. The front-end pipeline stage 330 can include, but is not limited to, the raw processing stage 306 and the resampling processing stage 308. For example, the raw image data can be in a Bayer raw image format. In the Bayer raw image format, pixel data for a value specific to a particular color (rather than all colors) is provided in each pixel. In image capture sensors, image data is typically provided in a Bayer pattern. The raw processing stage 306 is capable of processing image data in the Bayer raw image format.

[0053] Operations performed by the raw processing stage 306 include, but are not limited to, sensor linearization, black level compensation, fixed pattern noise reduction, defective pixel correction, raw noise filtering, lens shading correction, white balance gain, highlight recovery, and color difference recovery (or correction). Sensor linearization refers to mapping non-linear image data to a linear space for other processing. Black level compensation refers to providing a digital gain, offset, and clipping independently for each color component (e.g., Gr, R, B, Gb) of the image data. Fixed pattern noise reduction refers to removing offset fixed pattern noise and obtaining fixed pattern noise by subtracting a dark frame from an input image and multiplying different gains to pixels. Defective pixel correction refers to detecting defective pixels and then replacing defective pixel values. Raw noise filtering refers to reducing noise of image data by averaging neighboring pixels of similar brightness. Highlight recovery refers to estimating pixel values for those pixels that are clipped (or close to clipping) from other channels. Lens shading correction refers to applying a gain per pixel to compensate for an intensity drop that is roughly proportional to the distance from the optical center of the lens. White balance gain refers to providing a digital gain, offset, and clipping for white balance independently for all color components (e.g., Gr, R, B, Gb of a Bayer pattern).

[0054] The components of ISP 206 can convert raw image data to image data in a full color gamut, and thus, in addition to or instead of raw image data, raw processing stage 306 can process image data in a full color gamut.

[0055] Resampling processing stage 308 performs various operations to convert, resample, or scale image data received from raw processing stage 306. The operations performed by resampling processing stage 308 can include, but are not limited to, demosaicing operations, per-pixel color correction operations, gamma mapping operations, color space conversion, and downscaling or subband segmentation. Demosaicing operations refer to converting or interpolating color missing samples from raw image data (e.g., in a Bayer pattern) into output image data into a full color gamut. Demosaicing operations can include low-pass directional filtering of interpolated samples to obtain full color pixels. Per-pixel color correction operations refer to a process that performs color correction on a per-pixel basis using information about the relative noise standard deviation of each color channel to correct color without amplifying noise in the image data. Gamma mapping refers to converting image data from input image data values to output data values to perform gamma correction. For purposes of gamma mapping, a lookup table (or other structure that indexes a pixel value to another value) for different color components or channels of each pixel can be used (e.g., separate lookup tables for R, G, and B color components). Color space conversion refers to converting a color space of input image data to a different format. In one embodiment, resampling processing stage 308 converts RGB format to YCbCr format for further processing. In another embodiment, resampling processing stage 308 converts RBD format to RGB format for further processing.

[0056] Central control module 320 can control and coordinate the overall operation of the other components in ISP 206. Central control module 320 performs various operations including, but not limited to, monitoring various operational parameters (e.g., recording clock cycles, memory latency, quality of service, and status information), updating or managing control parameters of the other components of ISP 206, and interfacing with sensor interface 302 to control the start and stop of the other components of ISP 206. For example, while the other components in ISP 206 are in an idle state, central control module 320 can update programmable parameters of the other components. After updating the programmable parameters, central control module 320 can place these components of ISP 206 in a running state to perform one or more operations or tasks. Central control module 320 can also instruct the other components of ISP 206 to store image data (e.g., by writing to memory 310) before, during, or after resampling processing stage 308. In one embodiment, central control module 320 can be implemented as a state machine that is configured to control the operation of the other components of ISP 206. FIG. 2The system memory 230 in the memory 218 (e.g., the system memory 230 in the memory 230). In this way, in addition to or instead of processing image data output from the resample processing stage 308 through the back-end pipeline stage 340, full resolution image data in raw or full color gamut format can be stored.

[0057] The image statistics module 304 performs various operations to collect statistical information associated with the image data. The operations to collect statistical information can include, but are not limited to, sensor linearization, replacement of patterned defective pixels, sub-sampling raw image data, detection and replacement of non-patterned defective pixels, black level compensation, lens shading correction, and inverse black level compensation. After performing one or more such operations, statistical information such as 3A statistics (auto white balance (AWB), auto exposure (AE)), histograms (e.g., 2D color or component), and any other image data information can be collected or tracked. In some embodiments, when a previous operation identifies pixels that are clipped, the values of certain pixels or regions of pixel values can be excluded from the collection of certain statistical data. Although FIG. 3 Although only a single statistics module 304 is shown in the ISP 206, multiple image statistics modules can be included in the ISP 206. For example, each image sensor 202 can correspond to a separate image statistics module 304. In such embodiments, each statistics module can be programmed by the central control module 320 to collect different information for the same or different image data.

[0058] The vision module 322 performs various operations to facilitate computer vision operations at the CPU 208, such as face detection in image data. The vision module 322 can perform various operations, including pre-processing, global tone mapping and gamma correction, vision noise filtering, resizing, keypoint detection, generation of histograms of oriented gradients (HOG) and normalized cross correlation (NCC). If the input image data is not in YCrCb format, the pre-processing can include subsampling or binning operations and luminance computation. Global mapping and gamma correction can be performed on the pre-processed data on the luminance image. Vision noise filtering is performed to remove pixel defects and reduce noise present in the image data, thereby improving the quality and performance of subsequent computer vision algorithms. Such vision noise filtering can include detection and fixing of defective or outlier pixels, and performing bilateral filtering by averaging neighboring pixels of similar luminance to reduce noise. Various vision algorithms use images of different sizes and scales. For example, resizing of images is performed by binning or linear interpolation operations. Key points are locations within an image that are surrounded by image patches that are well suited to matching other images of the same scene or object. Such key points are useful in image alignment, computing camera pose, and object tracking. Keypoint detection refers to the process of identifying such key points in an image. HOG provides a description of image patches used for tasks in image analysis and computer vision. For example, HOG can be generated by (i) computing horizontal and vertical gradients using simple difference filters, (ii) computing gradient directions and magnitudes from the horizontal and vertical gradients, and (iii) binning the gradient directions. NCC is a process of computing the spatial cross correlation between an image patch and a kernel.

[0059] The back-end interface 342 receives image data from other image sources outside of the image sensor 202 and forwards the image data to other components of the ISP 206 for processing. For example, image data can be received over a network connection and stored in the system memory 230. The back-end interface 342 retrieves the image data stored in the system memory 230 and provides it to the back-end pipeline stage 340 for processing. One of the many operations performed by the back-end interface 342 is to convert the retrieved image data to a format that can be used by the back-end processing stage 340. For example, the back-end interface 342 can convert image data formatted in RGB, YCbCr 4:2:0, or YCbCr 4:2:2 to a YCbCr 4:4:4 color format.

[0060] The back-end pipeline stage 340 processes image data according to a particular full color format (e.g., YCbCr 4:4:4 or RGB). In some embodiments, components of the back-end pipeline stage 340 can convert image data to a particular full color format before further processing. The back-end pipeline stage 340 can include a noise processing stage 310 and other stages such as a color processing stage 312. The back-end pipeline stage 340 can include FIG. 3other stages not shown.

[0061] The noise processing stage 310 performs various operations to reduce noise in the image data. Operations performed by the noise processing stage 310 include, but are not limited to, color space conversion, gamma / gamma mapping, temporal filtering, noise filtering, luminance sharpening, and chrominance noise reduction. Color space conversion can convert image data from one color space format to another color space format (e.g., from RGB format to YCbCr format). Gamma / gamma operations convert image data from input image data values to output data values to perform gamma correction or inverse gamma correction. Temporal filtering uses previously filtered image frames to filter out noise to reduce noise. For example, pixel values of previous image frames are combined with pixel values of a current image frame. Noise filtering can include, for example, spatial noise filtering. Luminance sharpening can sharpen luminance values of pixel data, while chrominance suppression can attenuate chrominance to gray (e.g., no color). In some embodiments, luminance sharpening and chrominance suppression can be performed simultaneously with spatial noise filtering. The aggressiveness of noise filtering can be determined differently for different regions of an image. Spatial noise filtering can be included as part of a temporal loop that implements temporal filtering. For example, previous image frames can be processed by a temporal filter and a spatial noise filter before being stored as reference frames for a next image frame to be processed. In other embodiments, spatial noise filtering can not be included as part of a temporal loop for temporal filtering (e.g., a spatial noise filter can be applied to an image frame after the image frame is stored as a reference image frame, and thus the reference frame is not spatially filtered).

[0062] The color processing stage 312 can perform various operations associated with adjusting color information in the image data. Operations performed in the color processing stage 312 include, but are not limited to, local tone mapping, gain / offset / clipping, color correction, three-dimensional color lookup, gamma conversion, and color space conversion. Local tone mapping refers to spatially varying local tone curves in order to provide more control when rendering the image. For example, a two-dimensional grid of tone curves (which can be programmed by the central control module 320) can be bilinearly interpolated so that a smoothly varying tone curve is produced across the image. In some embodiments, local tone mapping can also apply spatially varying and intensity varying color correction matrices, which can be used, for example, to make the sky bluer while toning down the blue in the shadows in the image. Digital gain / offset / clipping can be provided for each color channel or component of the image data. Color correction can apply a color correction transform matrix to the image data. 3D color lookup can utilize a three-dimensional array of color component output values (e.g., R, G, B) to perform advanced tone mapping, color space conversion, and other color transformations. For example, a gamma conversion can be performed by mapping input image data values to output data values in order to perform gamma correction, tone mapping, or histogram matching. Color space conversion can be implemented to convert image data from one color space to another color space (e.g., RGB to YCbCr). Other processing techniques can also be performed as part of the color processing stage 312 to perform other special image effects, including black and white conversion, sepia tone conversion, negative conversion, or exposure conversion.

[0063] The output rescaling module 314 can resample, transform, and correct distortions on the fly as the ISP 206 processes the image data. The output rescaling module 314 can compute fractional input coordinates for each pixel and use the fractional coordinates to interpolate output pixels via a polyphase resampling filter. The fractional input coordinates can be produced from a variety of possible transformations of the output coordinates, such as resizing or cropping the image (e.g., via simple horizontal and vertical scaling transformations), rotating and shearing the image (e.g., via non-separable matrix transformations), perspective warping (e.g., via an additional depth transformation), and per-pixel perspective segmentation applied in a striped segment to account for variations in the image sensor during image data capture (e.g., due to rolling shutter), and geometric distortion correction (e.g., via computing a radial distance from the optical center to index a radial gain table for interpolation, and applying a radial perturbation to the coordinates to account for radial lens distortion).

[0064] When processing image data at the output rescale module 314, the output rescale module 314 can apply a transform to the image data. The output rescale module 314 can include a horizontal scaling component and a vertical scaling component. The vertical portion of this design can implement a series of image data line buffers to hold the "support" required by the vertical filter. Since the ISP 206 can be a streaming device, only the rows of image data in a finite length sliding window of rows can be available to the filter. Once a row is discarded to make room for a newly incoming row, that row can not be available. The output rescale module 314 can statistically monitor the computed input Y coordinate on the previous row and use it to compute a set of optimal rows to hold in the vertical support window. For each subsequent row, the output rescale module can automatically generate a guess about the center of the vertical support window. In some embodiments, the output rescale module 314 can implement a piecewise perspective transform table encoded as a digital differential analyzer (DDA) stepper to perform a per-pixel perspective transform between the input image data and the output image data in order to correct for artifacts and motion caused by sensor motion during capture of the image frame. As discussed above with respect to FIG. 1 and FIG. 2 The output rescale can provide image data to various other components of the device 100 via the output interface 316.

[0065] In various embodiments, the functions of the components 302-350 can be performed in a different order than the order implied by the order of these functional units in the image processing pipeline illustrated in FIG. 3 and can be performed by different functional components than the functional components illustrated in FIG. 3 In addition, the various components as described in FIG. 3 may be embodied in various combinations of hardware, firmware, or software.

[0066] Example Optical Compensation Circuit

[0067] FIG. 4 is a block diagram illustrating the raw processing stage 306 with an optical compensation circuit 412 for compensation of optical image stabilization drift performed prior to lens shading correction, in accordance with one embodiment. FIG. 4 The portion of the raw processing stage 306 shown in FIG. 6 can also include a pixel row locator circuit 406 coupled to the input of the optical compensation circuit 412, and a lens shading correction circuit 416 coupled to the output of the optical compensation circuit 412. The raw processing stage 306 includes additional components not shown in FIG. 4 In addition, some of the components of the raw processing stage 306 as described with respect to FIG. 4 may be embodied in various combinations of hardware, firmware, or software.

[0068] Raw image data 402 (e.g., in a Bayer raw image format or in a Quadra image format) can be passed onto raw processing stage 306, for example, from sensor interface 302. Raw image data 402 can be captured by at least one image sensor 202 that receives light passing through a corresponding lens. Raw image data 402 can be processed within raw processing stage 306 to generate a raw image 404 in the form of a two-dimensional array of pixels organized into rows of pixels. The pixels of raw image 404 can be passed into pixel locator circuit 406 in a raster order (e.g., horizontally, row-by-row, or line-by-line). Pixel row locator circuit 406 can determine coordinates 408 (e.g., row numbers) of the pixels in each row (or line) of raw image 404. Information about the coordinates 408 of the pixels in each row of raw image 404 can be passed onto optical compensation circuit 412.

[0069] Optical compensation circuit 412 can generate an adjusted grid of pixels in raw image 404 for lens shading correction circuit 416. Optical compensation circuit 412 can obtain information about a shift 410H of a lens optical center along a first direction (e.g., along a horizontal direction or along an x-axis) and a shift 410V of the lens optical center along a second direction (e.g., along a vertical direction or along a y-axis) that is orthogonal to the first direction (e.g., from image sensor 202). Shifts 410H and 410V can be updated for each image frame captured by image sensor 202. Optical compensation circuit 412 can use shifts 410H, 410V, and coordinates 408 obtained from pixel row locator circuit 406 to generate coordinates 414 of an adjusted grid for each pixel in raw image 404. Information about coordinates 414 of the adjusted grid for each pixel in raw image 404 can be passed onto lens shading correction circuit 416.

[0070] Lens shading correction circuit 416 can use information about coordinates 414 of the adjusted grid for each pixel in raw image 404 to perform lens shading correction processing on each pixel in raw image 404. Lens shading correction circuit 416 can include a gain lookup table (LUT) 418 with a list of gain values. Lens shading correction circuit 416 can determine a gain value for each pixel by interpolating a value from gain LUT 418 according to an interpolation function (e.g., a bilinear interpolation function) and coordinates 414 of the adjusted grid for each pixel obtained from optical compensation circuit 412. Lens shading correction circuit 416 can apply the gain value to at least one color component of each pixel in raw image 404 to obtain a version of raw image 420 that is corrected for lens shading. One or more additional components of raw processing stage 306 FIG. 4The version of the original image 420 can be processed to generate a final original image 422, which is passed to, for example, the resampling processing stage 308 for further processing.

[0071] FIG. 5 is a block diagram illustrating a detailed view of the optical compensation circuit 412 according to one embodiment. The optical compensation circuit 412 can include a horizontal offset LUT 502, a processor circuit 528 coupled to the horizontal offset LUT 502, a vertical offset LUT 504, a processor circuit 530 coupled to the vertical offset LUT 504, and a grid adjustment circuit 526 coupled to outputs of the processor circuits 528, 530. The optical compensation circuit 412 can include more or less components than those shown in FIG. 5 The various components of the optical compensation circuit 412 described with respect to FIG. 5 may be embodied in various combinations of hardware, firmware, or software.

[0072] The horizontal offset LUT 502 can output a horizontal offset value 506 based on a coordinate 408 (e.g., a row number) of a current row of pixels in the original image 404, where the coordinate 408 represents an input entry of the horizontal offset LUT 502. The horizontal offset LUT 502 can include a predetermined list of horizontal offset values, e.g., a list of 9 horizontal offset values. The list of horizontal offset values in the horizontal offset LUT 502 can be updated for each image frame captured by the image sensor 202. The horizontal offset value 506 output by the horizontal offset LUT 502 along with the coordinate 408 (e.g., the current row number) can be passed to the processor circuit 528.

[0073] The processor circuit 528 can determine a grid offset 522 along a first direction (e.g., along a horizontal direction or along an x-axis) for each row of pixels in the original image 404 using information about the offset 410H, the horizontal offset value 506, and the coordinate 408 (e.g., the current row number). The processor circuit 528 can include an interpolation function circuit 510 and a combination circuit 518 coupled to an output of the interpolation function circuit 510. The interpolation function circuit 510 can determine an interpolated value 514 by interpolating the horizontal offset value 506 based on the coordinate 408 (e.g., the current row number) according to an interpolation function (e.g., a bilinear interpolation function) of the interpolation function circuit 510. The interpolated value 514 determined by the interpolation function circuit 510 can be passed to the combination circuit 518. The combination circuit 518 can determine the grid offset 522 along the first direction by combining the interpolated value 514 and the offset 410H. For example, the combination circuit 518 can determine the grid offset 522 along the first direction as a sum of the interpolated value 514 and the offset 410H. The information about the grid offset 522 along the first direction determined by the processor circuit 528 can be passed to the grid adjustment circuit 526.

[0074] The vertical offset LUT 504 can output a vertical offset value 508 based on a coordinate 408 (e.g., a row number) of a current row of pixels in the original image 404, where the coordinate 408 represents an input entry of the horizontal offset LUT 504. The vertical offset LUT 504 can include a predetermined list of vertical offset values, e.g., a list of 9 vertical offset values. The list of vertical offset values in the vertical offset LUT 504 can be updated for each image frame captured by the image sensor 202. The vertical offset value 508 output by the vertical offset LUT 504 along with the coordinate 408 (e.g., the current row number) can be passed onto the processor circuit 530.

[0075] The processor circuit 530 can determine a grid offset 524 along a second direction (e.g., along a vertical direction or along a y-axis) for each row of pixels in the original image 404 using information about the offset 410v, the vertical offset value 508, and the coordinate 408 (e.g., the current row number). The processor circuit 530 can include an interpolation function circuit 512 and a combination circuit 520 coupled to an output of the interpolation function circuit 512. The interpolation function circuit 512 can determine an interpolated value 516 by interpolating the vertical offset value 508 based on the coordinate 408 (e.g., the current row number) according to an interpolation function (e.g., a bilinear interpolation function) of the interpolation function circuit 512. The interpolated value 516 determined by the interpolation function circuit 512 can be passed onto the combination circuit 520. The combination circuit 520 can determine the grid offset 524 along the second direction by combining the interpolated value 516 and the offset 410v. For example, the combination circuit 520 can determine the grid offset 524 along the second direction as a sum of the interpolated value 516 and the offset 410v. The information about the grid offset 524 along the second direction determined by the processor circuit 530 can be passed onto the grid adjustment circuit 526.

[0076] Grid adjustment circuit 526 can use information about grid offset 522 along the first direction and grid offset 524 along the second direction to determine the adjusted grid's coordinates 414 of each pixel in original image 404. Grid adjustment circuit 526 can apply grid offset 522 along the first direction and grid offset 524 along the second direction to at least one pixel or a group of pixels (e.g., each pixel) in each row of original image 404 to generate the adjusted grid's coordinates 414 of the pixels in original image 404. Grid adjustment circuit 526 can further adjust grid offset 524 along the second direction by scaling grid offset 524 by a scaling factor (e.g., different from 1) that depends on the coordinates 408 of the corresponding row of pixels along the second direction (e.g., along the vertical direction or y-axis) (e.g., the current row number). Alternatively, the scaling factor can not depend on coordinates 408 and can be set to "1" for each row of pixels. In one or more embodiments, some or all of the functions of grid adjustment circuit 526 can be part of lens shading correction circuit 416.

[0077] FIG. 6 is a conceptual diagram 600 illustrating optical image stabilization compensation performed by optical compensation circuit 412 according to one embodiment. Diagram 600 shows an example image frame 602 overlaid on a two-dimensional grid. Image frame 602 can correspond to original image 404. The top-left corner of image frame 602 (e.g., the first pixel in the first row in image frame 602) has an offset 410H along the first direction (e.g., along the horizontal direction or along the x-axis) and an offset 410V along the second direction (e.g., along the horizontal direction or along the x-axis) with respect to the center axis of image sensor 202. Offsets 410H and 410V can be updated for each image frame. Curve 604 can represent the interpolation function (e.g., a bilinear interpolation function) of interpolation function circuit 510 and / or interpolation function circuit 512 in FIG. 5 FIG. 6 It can be observed that the interpolated values (e.g., interpolated value 514 and interpolated value 516) defined by interpolation function 604 depend on the vertical coordinates (e.g., coordinates along the y-axis) of the row of pixels in image frame 602 that is currently being processed (e.g., coordinates 408 in FIG. 5 FIG. 5

[0078] FIG. 7 is a conceptual diagram 700 illustrating grid adjustment for three rows of pixels as part of optical image stabilization compensation performed by optical compensation circuit 412 according to one embodiment. Original pixels P 0O ​​​The grid position of the original pixel P 0O in pixel row 0 can be first adjusted to the grid position of the virtual pixel P 0R by applying a grid offset 410H along the first direction (e.g., along the horizontal direction or along the x-axis) and a grid offset 410V along the second direction (e.g., along the vertical direction or along the y-axis). The grid offsets 410H and 410V can be determined by adding the corresponding offsets 410H, 410V to the corresponding interpolated values (e.g., the interpolated values 514, 516 in FIG. 5 that are functions of the vertical coordinates (e.g., the coordinates 408 in FIG. 5 ) of pixel row 0. The grid offsets 410H and 410V can correspond to the grid offsets 522 and 524, respectively. The grid positions of the other pixels in pixel row 0 can be obtained based on the grid position of the real pixel P 0R .

[0079] The grid position of the original pixel P 1O in pixel row 1 can be first adjusted to the grid position of the virtual pixel P 1V by applying a grid offset 704H along the first direction (e.g., along the horizontal direction or along the x-axis) and a grid offset 704V along the second direction (e.g., along the vertical direction or along the y-axis). The grid offsets 704H and 704V can be determined by adding the corresponding offsets 410H, 410V to the corresponding interpolated values (e.g., the interpolated values 514, 516 in FIG. 5 that are functions of the vertical coordinates (e.g., the coordinates 408 in FIG. 5 ) of pixel row 1. The grid offsets 704H and 704V can correspond to the grid offsets 522 and 524, respectively. The grid position of the virtual pixel P 1V may be further adjusted to the grid position of the real pixel P 1R by scaling the grid offset 704V by a scaling factor that depends on the vertical coordinates (e.g., the coordinates 408 in FIG. 5 ) of pixel row 1. It should be noted that for the pixels in pixel row 0, the scaling factor can be equal to zero. The grid positions of the other pixels in pixel row 1 can be obtained based on the grid position of the real pixel P 1R .

[0080] The grid position of the original pixel P 2OThe grid position of the virtual pixel P 2V may be first adjusted to the grid position of the virtual pixel P FIG. 5 by applying a grid offset 706H along a first direction (e.g., along a horizontal direction or along an x-axis) and a grid offset 706V along a second direction (e.g., along a vertical direction or along a y-axis). The grid offsets 706H and 706V can be determined by adding the corresponding offsets 410H, 410V to the corresponding interpolated values (e.g., the interpolated values 514, 516 in FIG. 5 are functions of the vertical coordinate (e.g., the coordinate 408 in FIG. 7 ) of the pixel row 2. The grid offsets 706H and 706V can correspond to the grid offsets 522 and 524, respectively. The grid position of the virtual pixel P 2V may then be adjusted to the grid position of the real pixel P 2R by scaling the grid offset 706V by a scaling factor that depends on the vertical coordinate (e.g., the coordinate 408 in Example Process for Lens Shadow Correction with Optical Image Stabilization ) of the pixel row 2. Since the scaling factor for the pixel row 2 is larger than the scaling factor for the pixel row 1, the scaling process for the pixel row 2 is illustrated in as a two-step process. The grid position of the virtual pixel P 2V may be first adjusted to the grid position of the virtual pixel P 2V ’ by scaling the grid offset 706V by the scaling factor for the pixel row 1, and then the grid position of the virtual pixel P 2V ’ is adjusted to the grid position of the real pixel P 2R by further scaling by the remaining scaling factor that depends on the vertical coordinate of the pixel row 2. The grid positions of other pixels in the pixel row 2 can be obtained based on the grid position of the real pixel P 2R .

[0081] FIG. 8

[0082] FIG. 8 is a flowchart illustrating a method of optical image stabilization performed by an image processor (e.g., the ISP 206) prior to lens shading correction, according to one embodiment. The image processor can process an image having a plurality of pixels organized in a series of rows, and the image is captured by an image sensor that receives light passing through a lens. The image processor obtains 802 a first offset of an optical center of the lens along a first direction and a second offset of the optical center along a second direction that is orthogonal to the first direction.

[0083] The image processor determines 804, for each row of pixels of the image, a first grid offset in the first direction using the first offset and a coordinate of the row of pixels in the second direction. The image processor can determine the first grid offset in the first direction as a sum of the first offset and a first interpolation value that is a function of the coordinate of the row of pixels in the second direction. The image processor can determine the first interpolation value by interpolating a corresponding value from a lookup table according to an interpolation function and the coordinate of the row of pixels in the second direction. The interpolation function can be, for example, a bilinear interpolation function.

[0084] The image processor determines 806, for each row of pixels of the image, a second grid offset in the second direction using the second offset and a coordinate of the row of pixels in the second direction. The image processor can determine the second grid offset in the second direction as a sum of the second offset and a second interpolation value that is a function of the coordinate of the row of pixels in the second direction. The image processor can determine the second interpolation value by interpolating a corresponding value from a lookup table according to an interpolation function and the coordinate of the row of pixels in the second direction. The image processor can adjust the second grid offset by scaling the second grid offset by a scaling factor that depends on a number of the row of pixels in the second direction.

[0085] The image processor generates 808 an adjusted grid of pixels of the image using the first grid offset and the second grid offset. The image processor can apply the first grid offset and the second grid offset to at least one pixel in each row of pixels in the image to generate the adjusted grid of pixels of the image. The image processor can perform vignette correction processing of pixels in the image using gain interpolation as a function of the adjusted grid. The image processor can determine a gain value for a pixel in the image by interpolating a value from a gain lookup table according to an interpolation function and coordinates of the pixel updated with the first grid offset and the second grid offset. The image processor can apply the gain value to at least one color component of the pixel to obtain a version of the image for vignette correction.

[0086] Embodiments of the processes described above with reference to ​ The embodiments of the processes described above with reference to

[0087] While specific embodiments and applications have been illustrated and described, it is to be understood that the application is not limited to the precise construction and components disclosed herein and various modifications, changes and variations which will be apparent to those skilled in the art can be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope of this disclosure.

Claims

1. An image processing circuit, comprising: a first processor circuit configured to determine, for each of a plurality of rows of a plurality of pixels of an image captured by an image sensor receiving light passing through a lens including an optical center having a first offset along a first direction and a second offset along a second direction orthogonal to the first direction, a first grid offset along the first direction using the first offset and coordinates of the row of pixels along the second direction; a second processor circuit configured to determine, for each row of the plurality of rows of the image, a second grid offset along the second direction using the second offset and the coordinates of the row pixels along the second direction; as well as A grid adjustment circuit is coupled to the first processor circuit and the second processor circuit, the grid adjustment circuit being configured to generate an adjusted grid of the plurality of pixels using the first grid offset and the second grid offset.

2. The image processing circuit according to claim 1 , wherein the grid adjustment circuit is further configured to: The first grid offset and the second grid offset are applied to at least one pixel in each of the plurality of rows to generate the adjusted grid of the plurality of pixels.

3. The image processing circuit according to claim 1 , wherein the first processor circuit comprises a first combination circuit, the first combination circuit being configured to: The first grid offset along the first direction is determined as a sum of the first offset and a first interpolated value that is a function of the coordinates of the row of pixels along the second direction.

4. The image processing circuit according to claim 3, wherein the first processor circuit further comprises a first interpolation function circuit, wherein the first interpolation function circuit is configured to: The first interpolation value is determined by interpolating corresponding values ​​from a lookup table according to an interpolation function and the coordinates of the row of pixels along the second direction.

5. The image processing circuit according to claim 4, wherein: The interpolation function is a bilinear interpolation function.

6. The image processing circuit according to claim 1 , wherein the second processor circuit comprises a second combination circuit, the second combination circuit being configured to: The second grid offset along the second direction is determined as a sum of the second offset and a second interpolated value, the second interpolated value being a function of the coordinates of the row of pixels along the second direction.

7. The image processing circuit according to claim 6, wherein the second processor circuit further comprises a second interpolation function circuit, wherein the second interpolation function circuit is configured to: The second interpolation value is determined by interpolating corresponding values ​​from a lookup table according to an interpolation function and the coordinates of the row of pixels along the second direction.

8. The image processing circuit according to claim 1 , wherein the grid adjustment circuit is further configured to: The second grid offset is adjusted by scaling the second grid offset by a scaling factor that depends on the numbering of the row pixels along the second direction.

9. The image processing circuit according to claim 1, further comprising: a lens shading correction circuit coupled to the grid adjustment circuit, the lens shading correction circuit being configured to: Lens shading correction processing of the plurality of pixels is performed using gain interpolation as a function of the adjusted grid.

10. The image processing circuit according to claim 1, wherein the image processing circuit further comprises: a lens shading correction circuit coupled to the grid adjustment circuit, the lens shading correction circuit being configured to: determining a gain value for the pixel by interpolating a value from a gain lookup table according to an interpolation function and coordinates of each pixel of the plurality of pixels updated using the first grid offset and the second grid offset; as well as The gain value is applied to at least one color component of the pixel to obtain a version of the image corrected for lens shading.

11. A method for processing an image having a plurality of pixels organized in a plurality of rows, captured by an image sensor receiving light through a lens, the method comprising: Obtaining a first offset of the optical center of the lens along a first direction and a second offset of the optical center along a second direction orthogonal to the first direction; determining, for each row of the plurality of rows of the image, a first grid offset along the first direction using the first offset and coordinates of the row pixels along the second direction; determining, for each row of the plurality of rows of the image, a second grid offset along the second direction using the second offset and the coordinates of the row pixels along the second direction; as well as An adjusted grid of the plurality of pixels is generated using the first grid offset and the second grid offset.

12. The method according to claim 11, further comprising: applying the first grid offset and the second grid offset to at least one pixel in each of the plurality of rows to generate the adjusted grid of the plurality of pixels; as well as Lens shading correction processing of the plurality of pixels is performed using gain interpolation as a function of the adjusted grid.

13. The method according to claim 11, further comprising: determining a first interpolation value by interpolating corresponding values ​​from a lookup table according to an interpolation function and the coordinates of the row of pixels along the second direction; as well as The first grid offset along the first direction is determined as a sum of the first offset and the first interpolated value.

14. The method according to claim 11, further comprising: The second grid offset along the second direction is determined as a sum of the second offset and a second interpolated value, the second interpolated value being a function of the coordinates of the row of pixels along the second direction.

15. The method according to claim 14, further comprising: The second interpolation value is determined by interpolating corresponding values ​​from a lookup table according to an interpolation function and the coordinates of the row of pixels along the second direction.

16. The method according to claim 11, further comprising: The second grid offset is adjusted by scaling the second grid offset by a scaling factor that depends on the numbering of the row pixels along the second direction.

17. The method according to claim 11, further comprising: determining a gain value for the pixel by interpolating a value from a gain lookup table according to an interpolation function and coordinates of each pixel of the plurality of pixels updated using the first grid offset and the second grid offset; as well as The gain value is applied to at least one color component of the pixel to obtain a version of the image corrected for lens shading.

18. A camera device comprising: an image sensor configured to capture an image comprising a plurality of pixels organized in a plurality of rows; a lens having an optical center with a first offset along a first direction and a second offset along a second direction orthogonal to the first direction; as well as An optical compensation circuit, the optical compensation circuit being configured to: determining, for each of the plurality of rows of the image, a first grid offset along the first direction using the first offset and coordinates of the row pixels along the second direction, determining, for each row of the plurality of rows of the image, a second grid offset along the second direction using the second offset and the coordinates of the row pixels along the second direction, and An adjusted grid of the plurality of pixels is generated using the first grid offset and the second grid offset.

19. The camera device of claim 18, wherein: The optical compensation circuit is further configured to apply the first grid offset and the second grid offset to at least one pixel in each of the plurality of rows to generate the adjusted grid of the plurality of pixels, and the camera device further comprises: A lens shading correction circuit is coupled to the optical compensation circuit, the lens shading correction circuit being configured to perform lens shading correction processing for the plurality of pixels using gain interpolation as a function of the adjusted grid.

20. The camera device of claim 18, wherein the optical compensation circuit is further configured to: determining the first interpolated value by interpolating corresponding values ​​from a first lookup table according to a first interpolation function and coordinates of the row of pixels along the second direction; determining the first grid offset along the first direction as a sum of the first offset and the first interpolated value; determining a second interpolated value by interpolating corresponding values ​​from a second lookup table according to a second interpolation function and the coordinates of the row of pixels along the second direction; as well as The second grid offset along the second direction is determined as a sum of the second offset and the second interpolated value.

Images