Point of fixation downsampling of image data

Abstract

A point of fixation downsampling (FDS) circuit for downsampling pixels in an image is disclosed. The FDS circuit downsamples a first subset of pixels of the same color in an image using a first scaling factor to generate first downsampled pixels in a first downsampled version of the image. The FDS circuit further downsamples a second subset of the first downsampled pixels of the same color using a second scaling factor to generate second downsampled pixels of the same color in a second downsampled version of the image. The pixels from the first subset are arranged in a first direction and the pixels from the second subset are arranged in a second direction.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. non-provisional patent application No. 17 / 870,456, filed on July 21, 2022, which is incorporated herein by reference in its entirety. Background Art 1. Technical Field

[0003] The present disclosure relates to circuits for processing image data, and more particularly, to circuits for foveated downsampling of image data.

[0004] 2. Description of related technologies

[0005] Image data captured by an image sensor or received from other data sources is typically processed in an image processing pipeline before further processing or consumption. For example, the raw image data may be corrected, filtered, or otherwise modified before being provided to subsequent components such as a video encoder. In order to perform correction or enhancement on the captured image data, various components, unit levels, or modules may be employed.

[0006] Such image processing pipelines may be constructed so that correction or enhancement of captured image data can be performed in an advantageous manner without consuming other system resources. Although many image processing algorithms can be performed by executing software programs on a central processing unit (CPU), executing these programs on the CPU consumes a large amount of bandwidth of the CPU and other peripheral resources as well as increases power consumption. Therefore, image processing pipelines are typically implemented as hardware components separate from the CPU and dedicated to executing one or more image processing algorithms.

[0007] However, the image processing pipeline does not take into account the use of a wide-angle lens (e.g., a fisheye lens) to generate image data. When a wide-angle lens is used to generate image data, the refraction angle of light with different wavelengths changes, thereby manifesting itself on the image sensor as misaligned, shifted focus between the red, green, and blue channels. As a result, color fringing appears at the sharp and high-contrast edges of a full-color image generated from the image data. Summary of the invention

[0008] An embodiment relates to an image processor, which includes a foveated downsampling and correction circuit for correcting chromatic aberration in an image captured by one or more image sensors coupled to the image processor. The foveated downsampling and correction circuit includes a first correction circuit (e.g., a vertical foveated downsampling and correction circuit) and a second correction circuit (e.g., a horizontal correction circuit) coupled to the first correction circuit. The first correction circuit performs downsampling and interpolation on pixel values ​​of a first subset of pixels of the same color in an original image using a first downsampling scaling factor and a first interpolation coefficient to generate a first corrected pixel value of the same color pixel in a first corrected version of the original image. The pixels in the first subset are arranged in a first direction (e.g., a vertical direction), the first downsampling scaling factor gradually changes along the first direction, and the first interpolation coefficient corresponds to a first offset value. The first offset value represents a first distance from each downsampled pixel position along the first direction to a corresponding first virtual pixel in the first direction.

[0009] The second correction circuit receives the first corrected pixel value of the first corrected version and uses the second interpolation coefficient to perform interpolation of the pixel values ​​of the second subset of pixels in the first corrected version to generate second corrected pixel values ​​of pixels of the same color in the second corrected version of the original image. The pixels in the second subset are arranged in a second direction (e.g., a horizontal direction) perpendicular to the first direction, and the second interpolation coefficient corresponds to a second offset value. The second offset value represents a second distance from the second subset of pixels to the corresponding second virtual pixel in the second direction.

[0010] In some embodiments, the image processor further comprises a downsampling circuit coupled to the second correction circuit. The downsampling circuit receives a second corrected pixel value of a pixel of the same color in the second corrected version. The downsampling circuit performs downsampling on a subset of pixels of the same color of the second corrected version using a second downsampling scaling factor to generate corrected pixel values ​​of pixels of the same color in the corrected version of the original image. The pixels in the subset are arranged in a second direction, and the second downsampling scaling factor varies gradually along the second direction.

[0011] Embodiments of the present disclosure further relate to a foveated downsampling circuit in an image processor, the foveated downsampling circuit being used to perform foveated downsampling on pixels in a captured image. The foveated downsampling circuit comprises a first downsampling circuit and a second downsampling circuit coupled to the first downsampling circuit. The first downsampling circuit downsamples a first subset of pixels of the same color in an original image using a first scaling factor to generate first downsampled pixels of the same color in a first downsampled version of the image, the first subset of pixels being arranged in a first direction. The second downsampling circuit receives a second subset of the first downsampled pixels arranged in a second direction. Then, the second downsampling circuit downsamples a second subset of pixels of the same color using a second scaling factor to generate second downsampled pixels of the same color in a second downsampled version of the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a high-level diagram of an electronic device according to one embodiment.

[0013] Figure 2 is a block diagram showing components in an electronic device according to one embodiment.

[0014] Figure 3A is a first block diagram illustrating an image processing pipeline implemented using an image signal processor according to one embodiment.

[0015] Figure 3B is a second block diagram illustrating an image processing pipeline implemented using an image signal processor according to one embodiment.

[0016] Figure 3C is a second block diagram illustrating an image processing pipeline implemented using an image signal processor according to one embodiment.

[0017] Figure 4A is an example vertical foveation downsampling of an image according to one embodiment.

[0018] Figure 4B is an example horizontal foveation downsampling of an image according to one embodiment.

[0019] Figure 5A is a conceptual diagram illustrating longitudinal / axial chromatic aberration according to one embodiment.

[0020] Figure 5B is a conceptual diagram illustrating lateral / transverse chromatic aberration according to one embodiment.

[0021] Figure 5C is a conceptual diagram illustrating raw image data generated by an image sensor using a wide-angle lens according to one embodiment.

[0022] Figure 6 is a block diagram showing a detailed view of a gaze point downsampling and correction circuit according to one embodiment.

[0023] Fig. 7A is a conceptual diagram illustrating combined vertical foveation downsampling and interpolation of raw image data according to one embodiment.

[0024] Figure 7B is a conceptual diagram illustrating horizontal interpolation of raw image data according to one embodiment.

[0025] Figure 8 is a diagram illustrating a pixel neighborhood for a given pixel according to one embodiment.

[0026] Fig. 9A is a block diagram showing a detailed view of an example foveated downsampling circuit according to one embodiment.

[0027] Fig. 9B is a block diagram showing a detailed view of another example foveated downsampling circuit according to one embodiment.

[0028] Fig. 10A is a block diagram of an example raw processing stage with foveated downsampling circuitry, according to one embodiment.

[0029] Fig. 10B is a block diagram of another example raw processing stage with foveated downsampling circuitry according to one embodiment.

[0030] Fig. 10C is a block diagram of an example resampling processing stage with foveated downsampling circuitry, according to one embodiment.

[0031] Fig.11 is a flow chart illustrating a method of performing foveated downsampling and correction to reduce color fringing of raw image data according to one embodiment.

[0032] Fig.12 is a flowchart of a method for performing foveated downsampling according to one embodiment.

[0033] The drawings depict, and the detailed description describes, various non-limiting embodiments for purposes of illustration only. DETAILED DESCRIPTION

[0034] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. Many specific details are shown in the following detailed description to provide a full understanding of the various described embodiments. However, the embodiments may be implemented without these specific details. In other cases, well-known methods, processes, components, circuits, and networks are not described in detail, thereby not unnecessarily obscuring various aspects of the embodiments.

[0035] Embodiments of the present disclosure relate to a foveated downsampling and correction circuit in an image processor for correcting chromatic aberration in a captured image generated by one or more image sensors coupled to the image processor. The foveated downsampling and correction circuit includes a vertical foveated downsampling and correction circuit and a horizontal correction circuit coupled to an output of the vertical foveated downsampling and correction circuit. The vertical foveated downsampling and correction circuit performs combined foveated downsampling and chromatic aberration restoration in a vertical direction of an original image generated by one or more image sensors. The vertical foveated downsampling and correction circuit generates first corrected pixel values ​​for pixels of the same color in a first corrected version of the original image. The horizontal correction circuit receives the first corrected pixel value from the vertical foveated downsampling and correction circuit and performs chromatic aberration restoration in a horizontal direction of the first corrected version of the original image. The horizontal correction circuit generates second corrected pixel values ​​for pixels of the same color in a second corrected version of the original image, the second corrected version having reduced chromatic aberration compared to the original image.

[0036] Embodiments of the present disclosure further relate to a foveation downsampling circuit in an image processor, wherein an image may be divided into two overall portions: a foveation (or center) portion corresponding to a foveation (or center) field of view, and a peripheral portion corresponding to a peripheral field of view. Since the peripheral field of view is generally less important, the peripheral portion of the image may be downsampled to a lower resolution for simpler processing. On the other hand, the foveation (or center) portion of the image may be downsampled but with a higher resolution than the peripheral portion of the image, or may not be downsampled to retain its original resolution.

[0037] The circuit for foveated downsampling proposed herein includes a first downsampling circuit (e.g., a vertical foveated downsampling circuit or a horizontal foveated downsampling circuit) and a second downsampling circuit (e.g., a horizontal foveated downsampling circuit or a vertical foveated downsampling circuit) coupled to the first downsampling circuit. The first downsampling circuit performs foveated downsampling on a first subset of pixels of the same color in an image (e.g., an original image) in a streaming manner using a first scaling factor to generate first downsampled pixels of the same color in a first downsampled version of the image, the first subset of pixels being arranged in a first direction (e.g., a vertical direction or a horizontal direction). The second downsampling circuit receives a second subset of the first downsampled pixels arranged in a second direction (e.g., a horizontal direction or a vertical direction) in a streaming manner. Then, the second downsampling circuit performs foveated downsampling on a second subset of pixels of the same color in a streaming manner using a second scaling factor to generate second downsampled pixels of the same color in a second downsampled version of the image. According to the method, pixels passed into the foveated downsampling circuit can be processed in a single pass through the columns and rows of the image.

[0038] Exemplary Electronic Devices

[0039] Embodiments of electronic devices, user interfaces for such devices, and associated processes for using such devices are described herein. In some embodiments, the device is a portable communication device, such as a mobile phone, that also includes other functions, such as a personal digital assistant (PDA) and / or a music player function. Exemplary embodiments of portable multifunction devices include, but are not limited to, the Apple Watch from Apple Inc. (Cupertino, California). Devices, iPod Devices, Apple Equipment and Device. Other portable electronic devices, such as wearable devices, laptops, or tablet computers may alternatively be used. In some embodiments, the device is not a portable communication device, but rather a desktop computer or other computing device not designed for portable use. In some embodiments, the disclosed electronic device may include a touch-sensitive surface (e.g., a touch screen display and / or a touchpad). Figure 1 The described example electronic device (eg, device 100) may include a touch-sensitive surface for receiving user input. The electronic device may also include one or more other physical user interface devices, such as a physical keyboard, mouse, and / or joystick.

[0040] Figure 1is a high-level diagram of an electronic device 100 according to one embodiment. The device 100 may include one or more physical buttons, such as a "home" button or a menu button 104. The menu button 104 is used, for example, to navigate to any application in a set of applications executing on the device 100. In some embodiments, the menu button 104 includes a fingerprint sensor that identifies a fingerprint on the menu button 104. The fingerprint sensor can be used to determine whether the finger on the menu button 104 has a fingerprint that matches a fingerprint stored for unlocking the device 100. Alternatively, in some embodiments, the menu button 104 is implemented as a soft key in a graphical user interface (GUI) displayed on a touch screen.

[0041] In some embodiments, the device 100 includes a touch screen 150, a menu button 104, a push button 106 for turning the device on / off and for locking the device, a volume adjustment button 108, a subscriber identity module (SIM) card slot 110, an earphone jack 112, and a docking / charging external port 124. The push button 106 can be used to turn the device on and off by pressing the button and keeping the button pressed for a predefined time interval; lock the device by pressing the button and releasing the button before the predefined time interval passes; and / or unlock the device or initiate an unlocking process. In an alternative embodiment, the device 100 also accepts voice input for activating or deactivating certain functions through a microphone 113. The device 100 includes various components, including but not limited to a memory (which may include one or more computer-readable storage media), a memory controller, one or more central processing units (CPUs), a peripheral device interface, an RF circuit system, an audio circuit system, a speaker 111, a microphone 113, an input / output (I / O) subsystem, and other input or control devices. Device 100 may include one or more image sensors 164, one or more proximity sensors 166, and one or more accelerometers 168. Device 100 may include more than one type of image sensor 164. Each type may include more than one image sensor 164. For example, one type of image sensor 164 may be a camera and another type of image sensor 164 may be an infrared sensor that may be used for facial recognition. Additionally or alternatively, image sensors 164 may be associated with different lens configurations. For example, device 100 may include rear image sensors, one with a wide-angle lens and another with a telephoto lens. Device 100 may include Figure 1 Components not shown in the figure, such as an ambient light sensor, a dot projector, and a flood illuminator.

[0042] Device 100 is only one example of an electronic device, and device 100 may have more or fewer components than listed above, some of which may be combined into one component or have a different configuration or arrangement. The various components of device 100 listed above are embodied in hardware, software, firmware, or a combination thereof, including one or more signal processing and / or application specific integrated circuits (ASICs). Although Figure 1 The components in FIG. 1 are shown as being generally located on the same side as touch screen 150, but one or more components may also be located on the opposite side of device 100. For example, the front side of device 100 may include an infrared image sensor 164 for facial recognition and another image sensor 164 as a front camera of device 100. The back side of device 100 may also include an additional two image sensors 164 as rear cameras of device 100.

[0043] Figure 2 1 is a block diagram illustrating components in device 100 according to one embodiment. Device 100 can perform various operations including image processing. For this and other purposes, device 100 may include image sensor 202, system on chip (SOC) component 204, system memory 230, permanent storage (e.g., flash memory) 228, motion sensor 234, and display 216, among other components. Figure 2 The components shown in are merely illustrative. For example, the device 100 may include Figure 2 Other components not shown in the figure (such as speakers or microphones). In addition, some components (such as motion sensor 234) can be omitted from device 100.

[0044] Image sensor 202 is a component for capturing image data. Each image sensor in image sensor 202 may be embodied as, for example, a complementary metal oxide semiconductor (CMOS) active pixel sensor, a camera, a video camera, or other device. Image sensor 202 generates raw image data, which is sent to SOC component 204 for further processing. In some embodiments, the image data processed by SOC component 204 is displayed on display 216, stored in system memory 230, permanent storage device 228, or sent to a remote computing device via a network connection. The raw image data generated by image sensor 202 may be a Bayer color filter array (CFA) pattern (hereinafter also referred to as "Bayer pattern"). Image sensor 202 may also include optical components and mechanical components that assist image sensing components (e.g., pixels) in capturing images. Optical components and mechanical components may include an aperture, a lens system, and an actuator that controls the focal length of image sensor 202.

[0045] The motion sensor 234 is a component or a group of components for sensing the motion of the device 100. The motion sensor 234 may generate a sensor signal indicating the orientation and / or acceleration of the device 100. The sensor signal is sent to the SOC component 204 for various operations, such as turning on the device 100 or rotating the image displayed on the display 216.

[0046] The display 216 is a component for displaying images generated by the SOC component 204. The display 216 may include, for example, a liquid crystal display (LCD) device or an organic light emitting diode (OLED) device. Based on data received from the SOC component 204, the display 216 may display various images, such as menus, selected operating parameters, images captured by the image sensor 202 and processed by the SOC component 204, and / or other information received from the user interface of the device 100 (not shown).

[0047] System memory 230 is a component for storing instructions executed by SOC component 204 and for storing data processed by SOC component 204. System memory 230 can be embodied as any type of memory, including, for example, dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) RAMBUS DRAM (RDRAM), static RAM (SRAM), or a combination thereof. In some embodiments, system memory 230 can store pixel data or other image data or statistics in various formats.

[0048] Persistent storage 228 is a component for storing data in a non-volatile manner. Persistent storage 228 retains data even when power is unavailable. Persistent storage 228 may be embodied as a read-only memory (ROM), flash memory, or other non-volatile random access memory device.

[0049] The SOC component 204 is embodied as one or more integrated circuit (IC) chips and performs various data processing processes. The SOC component 204 may include subcomponents such as an image signal processor (ISP) 206, a central processing unit (CPU) 208, a network interface 210, a motion sensor interface 212, a display controller 214, a graphics processor unit (GPU) 220, a memory controller 222, a video encoder 224, a storage controller 226, and various other input / output (I / O) interfaces 218 and a bus 232 connecting these subcomponents. The SOC component 204 may include more than Figure 2 More or fewer subcomponents than those shown.

[0050] ISP 206 is hardware that performs various stages of the image processing pipeline. In some embodiments, ISP 206 may receive raw image data from image sensor 202 and process the raw image data into a form usable by other subcomponents of SOC component 204 or components of device 100. ISP 206 may perform various image processing operations, such as image translation operations, horizontal and vertical scaling, color space conversion, and / or image stabilization transformations, as described below with reference to Figure 3A Described in detail.

[0051] CPU 208 may be implemented using any suitable instruction set architecture, and may be configured to execute instructions defined in the instruction set architecture. CPU 208 may be a general purpose or embedded processor using any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, RISC, ARM, or MIPS ISAs, or any other suitable ISA. Figure 2 A single CPU is shown in FIG, but SOC component 204 may include multiple CPUs. In a multi-processor system, each CPU may collectively implement the same ISA, but this is not required.

[0052] GPU 220 is a graphics processing circuit system for performing operations on graphics data. For example, GPU 220 may render objects to be displayed into a frame buffer (e.g., a frame buffer including pixel data for an entire frame). GPU 220 may include one or more graphics processors that may execute graphics software to perform some or all graphics operations or hardware acceleration of certain graphics operations.

[0053] I / O interface 218 is hardware, software, firmware, or a combination thereof for interfacing with various input / output components in device 100. I / O components may include devices such as keyboards, buttons, audio devices, and sensors such as a global positioning system. I / O interface 218 processes data for sending data to such I / O components or processes data received from these I / O components.

[0054] The network interface 210 is a subcomponent that supports the exchange of data between the device 100 and other devices via one or more networks (e.g., carrier or proxy devices). For example, video or other image data may be received from other devices via the network interface 210 and stored in the system memory 230 for subsequent processing (e.g., via a processor such as described below). Figure 3A The network may include, but is not limited to, a local area network (LAN) (e.g., Ethernet or a corporate network) and a wide area network (WAN). Image data received via network interface 210 may be processed by ISP 206 for image processing.

[0055] The motion sensor interface 212 is circuitry for interfacing with the motion sensor 234. The motion sensor interface 212 receives sensor information from the motion sensor 234 and processes the sensor information to determine the orientation or movement of the device 100.

[0056] Display controller 214 is a circuit system for sending image data to be displayed on display 216. Display controller 214 receives image data from ISP 206, CPU 208, graphics processor or system memory 230, and processes the image data into a format suitable for display on display 216.

[0057] Memory controller 222 is a circuit system for communicating with system memory 230. Memory controller 222 can read data from system memory 230 for processing by ISP 206, CPU 208, GPU 220, or other subcomponents of SOC component 204. Memory controller 222 can also write data to system memory 230 received from various subcomponents of SOC component 204.

[0058] The video encoder 224 is hardware, software, firmware, or a combination thereof for encoding video data into a format suitable for storage in the permanent storage 228 or for passing the data to the network interface 210 for transmission over a network to another device.

[0059] In some embodiments, one or more subcomponents of SOC component 204 or some functions of these subcomponents may be performed by software components executing on ISP 206, CPU 208, or GPU 220. Such software components may be stored in system memory 230, permanent storage 228, or another device that communicates with device 100 via network interface 210.

[0060] Image data or video data may flow through various data paths within SOC component 204. In one example, raw image data may be generated from image sensor 202 and processed by ISP 206, and then sent to system memory 230 via bus 232 and memory controller 222. After the image data is stored in system memory 230, the image data may be accessed by video encoder 224 for encoding or by display 216 for display via bus 232.

[0061] In another example, image data is received from a source other than image sensor 202. For example, video data may be streamed, downloaded, or otherwise transferred to SOC component 204 via a wired or wireless network. Image data may be received via network interface 210 and written to system memory 230 via memory controller 222. The image data may then be obtained from system memory 230 by ISP 206 and processed through one or more image processing pipeline stages, as described below with reference to FIG. Figure 3A The image data may then be returned to system memory 230 or sent to video encoder 224 , display controller 214 (for display on display 216 ), or storage controller 226 for storage in permanent storage 228 .

[0062] Example Image Signal Processing Pipeline

[0063] Figure 3A is a block diagram illustrating an image processing pipeline implemented using ISP 206 according to one embodiment. Figure 3A In an embodiment, ISP 206 is coupled to image sensor system 201 to receive raw image data, the image sensor system including one or more image sensors 202A to 202N (hereinafter collectively referred to as "image sensor 202" or also individually referred to as "image sensor 202"). Image sensor system 201 may include one or more subsystems that individually control image sensor 202. In some cases, each image sensor 202 may operate independently, while in other cases, image sensors 202 may share some components. For example, in one embodiment, two or more image sensors 202 may share the same circuit board that controls the mechanical components of the image sensors (e.g., an actuator that changes the focal length of each image sensor). The image sensing components of image sensor 202 may include different types of image sensing components that may provide raw image data to ISP 206 in different forms. For example, in one embodiment, the image sensing components may include a plurality of focus pixels for autofocus and a plurality of image pixels for capturing images. In another embodiment, the image sensing pixels may be used for both autofocus and image capture purposes.

[0064] ISP 206 implements an image processing pipeline that may include a set of stages for processing image information from creation, capture, or reception to output. ISP 206 may include, among other components, a sensor interface 302, a central control 320, a front-end pipeline stage 330, a back-end pipeline stage 340, an image statistics module 304, a vision module 322, a back-end interface 342, an output interface 316, and auto-focus circuits 350A to 350N (hereinafter collectively referred to as "auto-focus circuits 350" or individually as "auto-focus circuits 350"). ISP 206 may include Figure 3A Other components not shown or may be omitted Figure 3A One or more components shown in .

[0065] In one or more embodiments, different components of ISP 206 process image data at different rates. Figure 3A In an embodiment of the present invention, the front-end pipeline stages 330 (e.g., the raw processing stage 306 and the resampling processing stage 308) can process the image data at the initial rate. Therefore, various different techniques, adjustments, modifications, or other processing operations are performed by these front-end pipeline stages 330 at the initial rate. For example, if the front-end pipeline stages 330 process two pixels per clock cycle, the raw processing stage 306 operations (e.g., black level compensation, highlight restoration, and defective pixel correction) can process two pixels of image data at a time. In contrast, one or more back-end pipeline stages 340 can process image data at a different rate that is less than the initial data rate. For example, in Figure 3A In an embodiment of the present invention, the back-end pipeline stages 340 (eg, the noise processing stage 310, the color processing stage 312, and the output rescaling 314) may be processed at a reduced rate (eg, one pixel per clock cycle).

[0066] The raw image data captured by image sensor 202 may be transmitted in different ways to different components of ISP 206. In one embodiment, raw image data associated with in-focus pixels may be sent to autofocus circuitry 350, while raw image data corresponding to image pixels may be sent to sensor interface 302. In another embodiment, raw image data corresponding to both types of pixels may be sent to both autofocus circuitry 350 and sensor interface 302 simultaneously.

[0067] The autofocus circuit 350 may include hardware circuitry that analyzes raw image data to determine the appropriate focal length for each image sensor 202. In one embodiment, the raw image data may include data transmitted from image sensing pixels dedicated to image focusing. In another embodiment, the raw image data from the image capture pixels may also be used for autofocus purposes. The autofocus circuit 350 may perform various image processing operations to generate data to determine the appropriate focal length. The image processing operations may include cropping, merging, image compensation, scaling to generate data for autofocus purposes. The autofocus data generated by the autofocus circuit 350 may be fed back to the image sensor system 201 to control the focal length of the image sensor 202. For example, the image sensor 202 may include a control circuit that analyzes the autofocus data to determine a command signal to be sent to an actuator associated with the lens system of the image sensor 202, thereby changing the focal length of the image sensor 202. The data generated by the autofocus circuit 350 may also be sent to other components of the ISP 206 for other image processing purposes. For example, some data may be sent to the image statistics module 304 to determine information about automatic exposure.

[0068] The auto-focus circuit 350 may be a separate circuit from other components such as the image statistics module 304, the sensor interface 302, the front end 330, and the back end 340. This allows the ISP 206 to perform auto-focus analysis independently of other image processing pipelines. For example, the ISP 206 may analyze the raw image data from the image sensor 202A to adjust the focus of the image sensor 202A using the auto-focus circuit 350A while performing downstream image processing on the image data from the image sensor 202B. In one embodiment, the number of auto-focus circuits 350 may correspond to the number of image sensors 202. In other words, each image sensor 202 may have a corresponding auto-focus circuit dedicated to auto-focus of the image sensor 202. The device 100 may perform auto-focus for different image sensors 202 even if one or more image sensors 202 are not in active use. This allows a seamless transition between two image sensors 202 when the device 100 switches from one image sensor 202 to another. For example, in one embodiment, the device 100 may include a wide-angle camera and a telephoto camera as a dual rear camera system for photos and image processing. The device 100 may display an image captured by one of the dual cameras and may switch between the two cameras from time to time. The displayed image may seamlessly transition from image data captured by one image sensor 202 to image data captured by another image sensor 202 without waiting for the second image sensor 202 to adjust its focus because the two or more autofocus circuits 350 may continuously provide autofocus data to the image sensor system 201.

[0069] Raw image data captured by the different image sensors 202 may also be transmitted to the sensor interface 302. The sensor interface 302 receives the raw image data from the image sensor 202 and processes the raw image data into image data that can be processed by other stages in the pipeline. The sensor interface 302 may perform various pre-processing operations, such as image cropping, merging, or scaling to reduce the image data size. In some embodiments, pixels are sent from the image sensor 202 to the sensor interface 302 in raster order (e.g., horizontally, row by row). Subsequent processing in the pipeline may also be performed in raster order, and the results may also be output in raster order. Although Figure 3A Only a single image sensor system 201 and a single sensor interface 302 are shown in FIG. 1 , but when more than one image sensor system is provided in the device 100 , a corresponding number of sensor interfaces may be provided in the ISP 206 to process raw image data from each image sensor system.

[0070] The front-end pipeline stage 330 processes the image data in raw or full color gamut. The front-end pipeline stage 330 may include, but is not limited to, a raw processing stage 306 and a resampling processing stage 308. For example, the raw image data may be in a Bayer raw image format. In a Bayer raw image format, pixel data with values ​​dedicated to a specific color (rather than all colors) is provided in each pixel. In image capture sensors, image data is typically provided in a Bayer pattern. The raw processing stage 306 is capable of processing image data in a Bayer raw image format.

[0071] The operations performed by the raw processing stage 306 include, but are not limited to, sensor linearization, black level compensation, fixed pattern noise reduction, defective pixel correction, raw noise filtering, lens shading correction, white balance gain, highlight restoration, and chromatic aberration restoration (or correction). Sensor linearization refers to mapping nonlinear image data to a linear space for other processing. Black level compensation refers to independently providing digital gain, offset, and clipping for each color component (e.g., Gr, R, B, Gb) of the image data. Fixed pattern noise reduction refers to removing offset fixed pattern noise and obtaining fixed pattern noise by subtracting a dark frame from the input image and multiplying different gains by the pixels. Defective pixel correction refers to detecting defective pixels and then replacing the defective pixel values. Raw noise filtering refers to reducing the noise of image data by averaging adjacent pixels of similar brightness. Highlight restoration refers to estimating the pixel values ​​of those pixels that are clipped (or close to clipping) from other channels. Lens shading correction refers to applying gain to each pixel to compensate for the intensity drop that is roughly proportional to the distance from the optical center of the lens. White balance gain refers to providing digital gain, offset and clipping for white balance independently for all color components (e.g., Gr, R, B, Gb in Bayer style).

[0072] The foveated downsampling and correction (FDS-C) circuit 307 in the raw processing stage 306 performs chromatic aberration restoration by performing foveated downsampling and aberration correction. The chromatic aberration restoration performed by the FDS-C circuit 307 refers to correcting chromatic aberration in the raw image data caused by the use of a wide angle lens in the image sensor 202 to generate the raw image. Figure 6 , FIG. 7A to FIG. 7B and Fig.11 Details are provided regarding the structure and operation of FDS-C circuit 307. Components of ISP 206 may convert raw image data to image data in a full color gamut, and thus, raw processing stage 306 may process image data in a full color gamut in addition to or in lieu of raw image data.

[0073] The resampling processing stage 308 performs various operations to convert, resample or scale the image data received from the raw processing stage 306. The operations performed by the resampling processing stage 308 may include, but are not limited to, demosaicing operations, per-pixel color correction operations, gamma mapping operations, color space conversion, and reduction or sub-band segmentation. The demosaicing operation refers to converting or interpolating color-missing samples from the raw image data (e.g., in a Bayer pattern) into output image data into a full color domain. The demosaicing operation may include low-pass directional filtering of the interpolated samples to obtain full-color pixels. The per-pixel color correction operation refers to the process of performing color correction on a per-pixel basis using information about the relative noise standard deviation of each color channel to correct the color without amplifying the noise in the image data. Gamma mapping refers to converting image data from input image data values ​​to output data values ​​to perform gamma correction. For the purpose of gamma mapping, a lookup table (or other structure that indexes a pixel value to another value) for different color components or channels of each pixel may be used (e.g., separate lookup tables for R, G, and B color components). Color space conversion refers to converting the color space of the input image data into a different format. In one embodiment, the resampling processing stage 308 converts the RGB format to the YCbCr format for further processing. In another embodiment, the resampling processing stage 308 converts the RBD format to the RGB format for further processing.

[0074] The central control module 320 can control and coordinate the overall operation of other components in the ISP 206. The central control module 320 performs various operations including, but not limited to, monitoring various operating parameters (e.g., recording clock cycles, memory latency, quality of service, and status information), updating or managing control parameters of other components of the ISP 206, and interfacing with the sensor interface 302 to control the start and stop of other components of the ISP 206. For example, when other components in the ISP 206 are in an idle state, the central control module 320 can update the programmable parameters of the other components. After updating the programmable parameters, the central control module 320 can put these components of the ISP 206 into a running state to perform one or more operations or tasks. The central control module 320 can also instruct other components of the ISP 206 to store image data before, during, or after the resampling processing stage 308 (e.g., by writing Figure 2 In this manner, full resolution image data in an original or full color gamut format may be stored in addition to or in lieu of processing image data output from the resampling processing stage 308 through the back end pipeline stage 340.

[0075] The image statistics module 304 performs various operations to collect statistical information associated with the image data. Operations to collect statistical information may include, but are not limited to, sensor linearization, replacement of patterned defective pixels, subsampling raw image data, detection and replacement of non-patterned defective pixels, black level compensation, lens shading correction, and inverted black level compensation. After performing one or more such operations, statistical information such as 3A statistics (auto white balance (AWB), auto exposure (AE)), histograms (e.g., 2D color or component), and any other image data information may be collected or tracked. In some embodiments, when a previous operation identifies pixels that are clipped, certain pixel values ​​or regions of pixel values ​​may be excluded from the collection of certain statistical data. Although Figure 3A Only a single statistics module 304 is shown in the figure, but multiple image statistics modules may be included in the ISP 206. For example, each image sensor 202 may correspond to a separate image statistics module 304. In such an embodiment, each statistics module may be programmed by the central control module 320 to collect different information of the same or different image data.

[0076] The visual module 322 performs various operations to facilitate computer vision operations at the CPU 208, such as face detection in image data. The visual module 322 may perform various operations, including preprocessing, global tone mapping and gamma correction, visual noise filtering, resizing, key point detection, generation of histogram of oriented gradients (HOG) and normalized cross correlation (NCC). If the input image data is not in YCrCb format, the preprocessing may include subsampling or merging operations and brightness calculations. Global mapping and gamma correction may be performed on the preprocessed data on the brightness image. Visual noise filtering is performed to remove pixel defects and reduce the noise present in the image data, thereby improving the quality and performance of subsequent computer vision algorithms. Such visual noise filtering may include detecting and fixing point or defective pixels, and performing bilateral filtering to reduce noise by averaging adjacent pixels of similar brightness. Various visual algorithms use images of different sizes and scales. For example, resizing of the image is performed by merging or linear interpolation operations. Key points are locations within an image that are surrounded by image blocks that are very suitable for matching other images of the same scene or object. Such key points are useful in image alignment, calculating camera poses, and object tracking. Keypoint detection refers to the process of identifying such keypoints in an image. HOG provides a description of image patches for tasks in image analysis and computer vision. For example, HOG can be generated by (i) computing horizontal and vertical gradients using a simple difference filter, (ii) computing the gradient direction and magnitude from the horizontal and vertical gradients, and (iii) merging the gradient directions. NCC is the process of computing the spatial cross-correlation between an image patch and a kernel.

[0077] The back-end interface 342 receives image data from other image sources other than the image sensor 202 and forwards the image data to other components of the ISP 206 for processing. For example, the image data may be received via a network connection and stored in the system memory 230. The back-end interface 342 retrieves the image data stored in the system memory 230 and provides it to the back-end pipeline stage 340 for processing. One of the many operations performed by the back-end interface 342 is to convert the retrieved image data into a format that can be used by the back-end processing stage 340. For example, the back-end interface 342 may convert image data formatted in RGB, YCbCr4:2:0, or YCbCr 4:2:2 to a YCbCr 4:4:4 color format.

[0078] The back-end pipeline stage 340 processes the image data according to a specific full-color format (e.g., YCbCr 4:4:4 or RGB). In some embodiments, components of the back-end pipeline stage 340 may convert the image data to a specific full-color format before further processing. The back-end pipeline stage 340 may include other stages such as the noise processing stage 310 and the color processing stage 312. The back-end pipeline stage 340 may include Figure 3AOther stages not shown.

[0079] The noise processing stage 310 performs various operations to reduce noise in the image data. The operations performed by the noise processing stage 310 include, but are not limited to, color space conversion, gamma / de-gamma mapping, temporal filtering, noise filtering, brightness sharpening, and chroma noise reduction. Color space conversion can convert image data from one color space format to another color space format (e.g., RGB format converted to YCbCr format). The gamma / de-gamma operation converts image data from input image data values ​​to output data values ​​to perform gamma correction or inverse gamma correction. Temporal filtering uses previously filtered image frames to filter out noise to reduce noise. For example, the pixel values ​​of the previous image frame are combined with the pixel values ​​of the current image frame. Noise filtering may include, for example, spatial noise filtering. Luminance sharpening may sharpen the brightness values ​​of pixel data, while chroma suppression may attenuate chroma to gray (e.g., no color). In some embodiments, luma sharpening and chroma suppression may be performed simultaneously with spatial noise filtering. The aggressiveness of noise filtering may be determined differently for different regions of the image. Spatial noise filtering may be included as part of a time loop to implement temporal filtering. For example, a previous image frame may be processed by a temporal filter and a spatial noise filter before being stored as a reference frame for the next image frame to be processed. In other embodiments, spatial noise filtering may not be included as part of the temporal loop for temporal filtering (e.g., a spatial noise filter may be applied to an image frame after the image frame is stored as a reference image frame, and thus the reference frame is not spatially filtered).

[0080] The color processing stage 312 can perform various operations associated with adjusting the color information in the image data. The operations performed in the color processing stage 312 include, but are not limited to, local tone mapping, gain / offset / limiting, color correction, three-dimensional color lookup, gamma conversion, and color space conversion. Local tone mapping refers to a spatially varying local tone curve to provide more control when rendering an image. For example, a two-dimensional grid of tone curves (which can be programmed by the central control module 320) can be bilinearly interpolated so that a smoothly varying tone curve is produced on the image. In some embodiments, local tone mapping can also apply a spatially varying and intensity-varying color correction matrix, which can be used, for example, to make the sky bluer while lowering the blue in the shadows in the image. Digital gain / offset / limiting can be provided for each color channel or component of the image data. Color correction can apply a color correction transformation matrix to the image data. 3D color lookup can perform advanced tone mapping, color space conversion, and other color transformations using a three-dimensional array of color component output values ​​(e.g., R, G, B). For example, a gamma conversion can be performed by mapping input image data values ​​to output data values ​​to perform gamma correction, tone mapping, or histogram matching. Color space conversion may be implemented to convert image data from one color space to another (e.g., RGB to YCbCr). Other processing techniques may also be performed as part of the color processing stage 312 to perform other special image effects, including black and white conversion, sepia tone conversion, negative conversion, or light exposure conversion.

[0081] As the ISP 206 processes the image data, the output rescaling module 314 may resample, transform, and correct distortion on the fly. The output rescaling module 314 may calculate fractional input coordinates for each pixel and use the fractional coordinates to interpolate the output pixels via a polyphase resampling filter. The fractional input coordinates may be generated from a variety of possible transformations of the output coordinates, such as resizing or cropping the image (e.g., via simple horizontal and vertical scaling transformations), rotating and shearing the image (e.g., via inseparable matrix transformations), perspective warping (e.g., via an additional depth transform), and per-pixel perspective segmentation applied in stripe-like segments to account for changes in the image sensor during image data capture (e.g., due to a rolling shutter), as well as geometric distortion correction (e.g., via calculating the radial distance from the optical center to index an interpolated radial gain table, and applying radial perturbations to the coordinates to account for radial lens distortion).

[0082] When processing image data at the output rescaling module 314, the output rescaling module 314 may apply a transformation to the image data. The output rescaling module 314 may include a horizontal scaling component and a vertical scaling component. The vertical portion of the design may implement a series of image data line buffers to maintain the "support" required for the vertical filter. Since the ISP 206 can be a streaming device, there may only be rows of image data in a sliding window of limited length of rows available for the filter. Once a row is discarded to make room for a new incoming row, the row may be unavailable. The output rescaling module 314 may statistically monitor the calculated input Y coordinates on previous rows and use it to calculate a set of optimal rows to be kept in the vertical support window. For each subsequent row, the output rescaling module may automatically generate a guess about the center of the vertical support window. In some embodiments, the output rescaling module 314 may implement a piecewise perspective transformation table encoded as a digital differential analyzer (DDA) stepper to perform a per-pixel perspective transformation between the input image data and the output image data to correct for artifacts and motion caused by sensor motion during the capture of the image frame. As described above with respect to Figure 1 and Figure 2 As discussed, the output rescaling may provide the image data to various other components of device 100 via output interface 316 .

[0083] In various embodiments, the functions of components 302 to 350 can be similar to Figure 3A The order in which these functional units in the image processing pipeline are shown in FIG. 1 may be different from the order implied by the order in which these functional units are executed or may be different from the order in which the functional units are executed. Figure 3A The functional components shown in FIG. are implemented by different functional components. Figure 3A The various components described in the specification may be embodied in various combinations of hardware, firmware or software.

[0084] Figure 3B is another block diagram illustrating an image processing pipeline implemented using ISP 206 according to one embodiment. Figure 3B The image processing pipeline in corresponds to Figure 3A The image processing pipeline in . Figure 3B The image processing pipeline and Figure 3A The image processing pipelines of FIG. 30 are substantially the same, except that the raw processing stage 306 includes a foveated downsampling (FDS) circuit 309 for performing foveated downsampling on pixels of the raw image data. Figure 3C is another block diagram illustrating an image processing pipeline implemented using ISP 206 according to one embodiment. Figure 3C The image processing pipeline in Figure 3BThe image processing pipelines in FIG. 3 are substantially the same as those in FIG. 3 , except that the FDS circuit 309 is not a part of the raw processing stage 306, but is integrated into the resampling processing stage 308. In this case, the FDS circuit 309 may perform foveated downsampling on different color channels (e.g., R, G, B color channels) of a pixel of image data having a color format (e.g., RGB format). FIG. 9A to FIG. 9B , FIG. 10A to FIG. 10C and Fig.12 Details regarding the structure and operation of the FDS circuit 309 are provided.

[0085] Example foveated downsampling

[0086] Figure 4A 400 is an example vertical gaze point downsampling of image 400 according to one embodiment. Image 400 may be processed by downsampling pixels in image 400 along a first direction (e.g., a vertical direction). Image 400 may be an original image obtained by one or more image sensors 202. Image 400 may be divided into zoom segments 402. 1 , 402 2 , 402 3 , …, 402 m , 402 m+1 , …, 402 m+n-2 , 402 m+n-1 , 402 m+n , where m and n are integers. In one embodiment, the image 400 is divided into uniformly scaled segments 402 1 to 402 m+n In this case, all scaling segments (eg, all vertical downsampling steps) 402 1 to 402 m+n In another embodiment, the image 400 is divided into non-uniformly scaled segments 402 1 to 402 m+n In this case, one or more scaling segments (eg, one or more downsampling steps) in the scaling segment 402 1 to 402 m+n Different from other zoomed segments in image 400. In some embodiments, zoomed segment 402 1 to 402 m+n Two or more zoom segments in may be grouped into a single zoom segment associated with the same zoom factor (eg, the same downsampling ratio). Zoom segment 402 1 to 402 m+nSome zoom segments in the image 400 may have a size of a single pixel, where, for example, no downsampling is performed. The size of each zoom segment (eg, downsampling step) and the number of zoom segments in the image 400 (eg, the number of downsampling steps) may be configurable.

[0087] Each zoom segment 402 1 to 402 m+n The pixels in can be downsampled (or scaled) by S v1 :1, S v2 :1, S v3 :1, ..., S vm :1, S v(m+1) :1, ..., S v(m+n-2) :1, S v(m+n-1) :1, S v(m+n) :1 Downsampling along the vertical direction, where S v1 To S v(m+n) is the corresponding downsampling (or scaling) factor, and S v1 >S v2 >S v3 >…>S vm ≤S v(m+1) v(m+n-2) v(m+n-1) v(m+n) In one or more embodiments, S v1 =S v(m+n) , S v2 =S v(m+n-1) , S v3 =S v(m+n - 2) , and S vm =S v(m+1) Additionally, at least one scaling factor may be equal to 1, for example, pixels in at least one corresponding scaling segment may not be downsampled. vm is equal to 1 and / or a scaling factor S v(m+1) is equal to 1, and zoom segment 402 m and 402 m+1 Pixels in a zoom region (eg, within a defined vicinity of the central axis 404) may not be downsampled.

[0088] In one illustrative example, image 400 may be divided into a total of three zoom regions. In this case, peripheral zoom region 402 1 to 402 m-1 The zoom factor of 2 (eg, a downsampling ratio of 2:1) may be grouped into a single zoom segment, the center (eg, gaze point) zoom segment 402 m and 402 m+1 ​​​The zoom segments 402 (and one or more additional zoom segments within a defined vicinity of the central axis 404) may be grouped into a single zoom segment with a zoom factor of 1 (e.g., a 1:1 downsampling ratio), wherein no vertical downsampling is performed, and the peripheral zoom segments 402 m+2 to 402 m+n A scaling factor of 2 (eg, a downsampling ratio of 2:1) may be grouped into a single scaling zone.

[0089] In the first vertical fixation downsampling mode, the scaling factor S v1 To S v(m+n) The scaling factor S may be changed gradually, for example, at one or more rates less than a threshold rate. In the second vertical fixation downsampling mode, the scaling factor S v1 To S v(m+n) The vertical gaze point downsampling mode may be configurable. Figure 6 Details on the hardware implementation of vertical fixation downsampling combined with chromatic aberration restoration are provided below. Fig. 9A and Fig. 9B Details on the hardware implementation of independent vertical fixation downsampling are provided.

[0090] Figure 4B is an example horizontal gaze point downsampling of image 410 according to one embodiment. Image 410 may be processed by downsampling pixels along a second direction (e.g., horizontal direction). Image 410 may be in a Figure 4A The image 400 is obtained after vertical gaze point downsampling is performed on the image 400. In this case, Figure 4A After the vertical fixation point is downsampled, Figure 4B Alternatively, image 410 may be a raw image obtained by one or more image sensors 202. In this case, Figure 4A Before the vertical fixation downsampling is performed Figure 4B Horizontal fixation downsampling shown in .

[0091] Image 410 may be divided into zoom sections 412 1 ,412 2 ,412 3 , …, 412 m ,412 m+1 , …, 412 p+r-2 , 402 p+r-1 , 402 p+r , where p and r are integers. In one embodiment, the image 410 is divided into uniformly scaled segments 412 1 To 412 p+rIn this case, all scaling segments (eg, all vertical downsampling steps) 412 1 To 412 p+r In another embodiment, the image 400 is divided into non-uniformly scaled segments 412 1 To 412 m+n In this case, one or more scaling segments (eg, one or more downsampling steps) 412 1 To 412 p+r Different from other zoomed segments in image 410. In some embodiments, zoomed segment 412 1 To 412 p+r Two or more zoom segments in may be grouped into a single zoom segment associated with the same zoom factor (eg, the same downsampling ratio). 1 To 412 p+r Some zoom segments in the image 410 may have a size of a single pixel, where, for example, no downsampling is performed. The size of each zoom segment (eg, downsampling step) and the number of zoom segments in the image 410 (eg, the number of downsampling steps) may be configurable.

[0092] Each zoom segment 412 1 To 412 p+r The pixels in can be downsampled (or scaled) by S h1 :1, S h2 :1, S h3 :1, ..., S hp :1, S h(p+1) :1, ..., S h(p+r-2) :1, S h(p+r-1) :1, S h(p+r) :1 Downsampling along the horizontal direction, where S h1 To S h(p+q) is the corresponding downsampling (or scaling) factor, and S h1 >S h2 >S h3 >…>S hm ≤S h(p+1) h(p+r-2) h(p+r-1) h(p+r) In one or more embodiments, S h1 =S h(p+2) , S h2 =S h(p+r-1) , S h3 =S h(p+r-2) , and S hp =S h(p+1) ​​​Additionally, at least one scaling factor may be equal to 1, for example, pixels in at least one corresponding scaling segment may not be downsampled. hp is equal to 1 and / or a scaling factor S h(p+1) is equal to 1, and zoom segment 412 p and 402 p+1 Pixels in a zoom region (eg, within a defined vicinity of the central axis 414) may not be downsampled.

[0093] In one illustrative example, image 410 may be divided into a total of three zoom regions. In this case, peripheral zoom region 412 1 To 412 p-1 The zoom factor of 2 (eg, a downsampling ratio of 2:1) may be grouped into a single zoom segment, the center (eg, gaze point) zoom segment 412 p and 412 p+1 The zoom segments 412 (and one or more additional zoom segments within a defined vicinity of the central axis 414) may be grouped into a single zoom segment with a zoom factor of 1 (e.g., a 1:1 downsampling ratio), wherein no vertical downsampling is performed, and the peripheral zoom segments 412 m+2 To 412 m+n A scaling factor of 2 (eg, a downsampling ratio of 2:1) may be grouped into a single scaling zone.

[0094] In the first horizontal fixation downsampling mode, the scaling factor S h1 To S h(p+r) The scaling factor S may be changed gradually, for example, at one or more rates less than a threshold rate. In the second horizontal fixation downsampling mode, the scaling factor S h1 To S h(p+r) The mode of horizontal fixation downsampling may be configurable. Figure 6 Details on the hardware implementation of horizontal fixation downsampling combined with chromatic aberration restoration are provided below. Fig. 9A and Fig. 9B Details on the hardware implementation of independent horizontal fixation downsampling are provided.

[0095] Example Chromatic Aberration Restoration

[0096] Typically, chromatic aberration is caused by the inability of a lens to focus different wavelengths of light (eg, different colors of light) to the same focal point. Figure 5A An example of longitudinal (eg, axial) chromatic aberration is shown. Figure 5AAs shown, wide angle lens 502 refracts light 504 such that different wavelengths of light (eg, red, green, and blue) are focused at different distances from wide angle lens 502 (eg, different distances from focal plane 506 ) along optical axis 508 . Figure 5B 1 shows lateral (eg, transverse) chromatic aberration according to one embodiment. Figure 5B As shown, wide angle lens 510 refracts light 512 so that light of different wavelengths (e.g., red light, green light, and blue light) is focused at different locations on focal plane 514 (e.g., at different distances from optical axis 516). Figure 5A and Figure 5B The described chromatic aberration caused by the use of wide angle lenses 502, 510 manifests itself as color fringing at edges in a full color image.

[0097] Figure 5C 5 shows raw image data generated using light 504 captured by image sensor 202 using wide angle lens 502, according to one embodiment. Figure 5C As shown, the raw image data is in a Bayer pattern 518. The Bayer pattern 518 includes alternating rows of red-green pixels and green-blue pixels. Generally speaking, since the human eye is more sensitive to green light than to both red and blue light, the Bayer pattern 518 includes more green pixels than red or blue pixels.

[0098] Example gaze point downsampling and correction circuit

[0099] Figure 6 is a block diagram illustrating a detailed view of a foveated downsampling and correction (FDS-C) circuit 307 according to one embodiment. The FDS-C circuit 307 corrects chromatic aberration in a raw image 602 generated by one or more image sensors 202. Specifically, the FDS-C circuit 307 performs combined foveated downsampling and chromatic aberration restoration in a first direction (e.g., a vertical direction) of the raw image 602 to generate first corrected pixel values ​​632 of a first corrected version of the raw image. The FDS-C circuit 307 further performs chromatic aberration restoration in a second direction (e.g., a horizontal direction) of the first corrected version of the raw image to generate second corrected pixel values ​​636 of a second corrected version of the raw image with reduced chromatic aberration. In one or more embodiments, the raw image 602 is in a Bayer pattern and is generated by at least one image sensor 202 using at least one wide-angle lens, such as with respect to FIG. Figure 5C Since the original image 602 is generated using at least one wide-angle lens, the full-color image generated directly from the original image 602 will include chromatic aberration. By using the second corrected pixel values ​​636 of the second corrected version to generate the full-color image instead of the original image 602, the chromatic aberration in the full-color image is reduced.

[0100] In one embodiment, the FDS-C circuit 307 includes a pixel locator circuit 603, a downsampling scaling factor lookup table (LUT) 604, a gaze point downsampling locator circuit 608, an offset LUT 612, an offset interpolator circuit 616, a vertical phase LUT 622, a horizontal phase LUT 624, a vertical gaze point downsampling and correction circuit 630, and a horizontal correction circuit 634. Additionally, the FDS-C circuit 307 is coupled to a horizontal gaze point downsampling and scaler circuit 648. In other embodiments, the FDS-C circuit 307 may have a plurality of Figure 6 More or fewer circuits and LUTs may be shown. For example, horizontal gaze point downsampling and scaler circuit 648 may be part of FDS-C circuit 307.

[0101] The down-sampling scaling factor LUT 604 stores down-sampling scaling factors indexed by positions in a first direction (e.g., a vertical direction) of an image (e.g., an original image 602). The down-sampling scaling factor LUT 604 receives index information 605 related to positions of corresponding pixels along the first direction in the original image 602. The index information 605 of the corresponding pixels along the first direction in the original image 602 is extracted by the pixel locator circuit 603. Upon receiving the index information 605, the down-sampling scaling factor LUT 604 outputs a corresponding down-sampling scaling factor 606, which is passed to the fixation point down-sampling pixel locator circuit 608.

[0102] The foveated downsampling locator circuit 608 receives the downsampling scaling factor 606 from the downsampling scaling factor LUT 604 and calculates a downsampling pixel position 610 (e.g., a downsampling landing point) along a first direction of the original image 602. Information about the downsampling pixel position 610 calculated by the foveated downsampling locator circuit 608 is provided to the offset LUT 612.

[0103] The offset LUT 612 stores a grid of pre-calculated horizontal and vertical offset values. The horizontal offset value and the vertical offset value of a particular pixel represent the horizontal distance and the vertical distance to a virtual pixel, respectively, whose pixel value corresponds to the pixel value of the particular pixel without any color difference. The grid includes a plurality of grid points with a plurality of pixel offset values. The pre-calculated offset values ​​in the grid may be associated with the optical configuration of the corresponding image sensor 202 (e.g., using a particular wide-angle lens). Therefore, the offset LUT 612 may store different sets of offset values ​​associated with different image sensors 202. In one or more embodiments, the grid is coarser than the arrangement of the pixels of the Bayer pattern 502. A particular pixel position may be associated with one or more grid points and include four pixel offset values: a horizontal pixel offset value of a red pixel, a vertical pixel offset value of a red pixel, a horizontal offset value of a blue pixel, and a vertical offset value of a blue pixel. The horizontal offset value of a green pixel and the vertical offset value of a green pixel may be set to zero.

[0104] Upon receiving information about the downsampled pixel positions 610 in a first direction of the original image 602, the offset LUT 612 may provide corresponding vertical offset values ​​614 to the offset interpolator circuit 616. Additionally, the offset LUT 612 may provide corresponding horizontal offset values ​​614 to the offset interpolator circuit 616 based on information about the positions of a subset of pixels of the original image 602 arranged in a second direction (e.g., a horizontal direction) perpendicular to the first direction.

[0105] The offset interpolator circuit 616 is coupled to the offset LUT 612 and receives pre-calculated horizontal and vertical offset values ​​614 from the offset LUT 612. In one embodiment, the offset interpolator circuit 616 calculates horizontal and vertical offset values ​​for a subset of pixels (e.g., blue and red pixels) included in the original image 602. Specifically, the offset interpolator circuit 616 calculates a first offset value 618 (e.g., a vertical offset value) for the blue or red pixel by performing interpolation on the pre-calculated vertical offset value 614. In addition, the offset interpolator circuit 616 calculates a second offset value 620 (e.g., a horizontal offset value) for the blue or red pixel by performing interpolation on the pre-calculated vertical offset values ​​of the grid points surrounding the blue or red pixel, as described below with reference to FIG. Figure 8As described. That is, for each red or blue pixel in the original image 602, the offset interpolator circuit 616 calculates the horizontal pixel offset of the red channel of the pixel, the vertical pixel offset value of the red channel of the pixel, the horizontal pixel offset of the blue channel of the pixel, and the vertical pixel offset value of the blue channel of the pixel. In one or more embodiments, the offset interpolator circuit 616 does not calculate the vertical and horizontal pixel offsets of the green channel of the pixel (for example, the vertical and horizontal pixel offsets of the green channel are zero). However, in one or more other embodiments, the offset interpolator circuit 616 can also calculate the horizontal pixel offset of the green channel of the pixel and the vertical pixel offset value of the green channel of the pixel. Generally speaking, when calculating the horizontal and vertical pixel offsets of two color channels, the horizontal and vertical pixel offsets of the remaining color channels (RGB) are not calculated.

[0106] Fig. 7A Vertical gaze point downsampling and interpolation based on vertical offset pixel correction for the red channel of a subset of pixels included in the original image 602 is shown according to one embodiment. Due to chromatic aberration in the vertical direction, the pixel value of the red pixel P2 (as part of the Bayer pattern 518) captured by the image sensor 202 is inaccurate. The pixel value of the virtual pixel 706 at the downsampled pixel position 702 is used to obtain the corrected pixel value (e.g., first corrected pixel value 632) at that position, which is obtained by vertically offsetting the downsampled pixel position 702 (if there is no horizontal shift of focus due to chromatic aberration) by a distance 704 (e.g., a negative vertical pixel offset). Therefore, the corrected pixel value is generated at the downsampled pixel position 702 and output from the vertical gaze point downsampling and correction circuit 630 as the first corrected pixel value 632. Similarly, for a positive vertical pixel offset, a corrected pixel value (e.g., first corrected pixel value 632) at the downsampled pixel position 712 is obtained using the pixel value of the virtual pixel 716 at that position, which is obtained by vertically offsetting the downsampled pixel position 712 by a distance 714 (e.g., a positive vertical pixel offset). Thus, a corrected pixel value is generated at the downsampled pixel position 712 and output from the vertical fixation downsampling and correction circuit 630 as the first corrected pixel value 632.

[0107] As will be further described below, the first offset value 704 (or the first offset value 714) is used as a parameter to obtain a phase value for bilinear or bicubic interpolation (e.g., equal to the distance from the position of the virtual pixel 706 to the position of the red pixel P2, and similarly equal to the distance from the position of the virtual pixel 716 to the position of the red pixel P2). The phase value is used to obtain an interpolation coefficient for bilinear or bicubic interpolation of the pixel values ​​of the adjacent red pixels P0, P1, P2, and P3 in the vertical direction to calculate the pixel value of the virtual pixel 706 (or the virtual pixel 716). The calculated pixel value of the virtual pixel 706 (or the calculated pixel value of the virtual pixel 716) then becomes the first corrected pixel value 632 output from the vertical fixation downsampling and correction circuit 630 at the downsampled pixel position 702 (or at the downsampled pixel position 712). Such correction of pixel values ​​and pixel positions is performed for all red pixels to account for vertical chromatic aberration and / or vertical fixation downsampling. The blue channel of the pixel is also corrected with respect to the vertical fixation downsampling. Fig. 7A The red channels of the pixels shown are corrected for their vertical offset in a similar manner.

[0108] Figure 7B Horizontal interpolation based on horizontal offset pixel correction for the red channel of a subset of pixels is shown according to one embodiment. Figure 7B The red pixels in have pixel values ​​corrected using the vertical offset, as described above in reference Fig. 7A As explained. The pixel value of the red pixel P6 for vertical color difference correction does not take into account horizontal color difference. In order to take into account horizontal color difference, the pixel value of pixel P6 is replaced with the pixel value of a virtual pixel 726 (or a virtual pixel 736), which is horizontally offset from the position 722 of pixel P6 by a distance 724 (or a second offset value) for a negative horizontal pixel offset or by a horizontal offset distance 734 (or a second offset value) for a positive horizontal pixel offset. As will be further described below, the second offset value 724 (or second offset value 734) is used as a parameter for interpolating the pixel values ​​of adjacent pixels P4, P5, P6 and P7 in the horizontal direction. Such replacement is performed across all red pixels to correct horizontal color difference. The blue channel of the pixel is also horizontally offset by a distance 724 (or a second offset value) from the position 722 of pixel P6. Figure 7B The red channels of the pixels shown are corrected for their horizontal offset in a similar manner.

[0109] Figure 8Grid points GP0 to GP3 surrounding a given pixel 802 are shown according to one embodiment. As described above, each of the grid points GP0 to GP3 has associated vertical and horizontal offset values ​​for red and blue pixels stored in the offset LUT 612. If the pixel 802 is a red pixel, the offset interpolator circuit 616 performs bilinear interpolation or bicubic interpolation on the four vertical offset values ​​of the four grid points GP0 to GP3 for the red pixel and generates an interpolated vertical offset value 618 for the red pixel. The offset interpolator circuit 616 also performs bilinear interpolation or bicubic interpolation on the four horizontal offset values ​​of the four grid points GP0 to GP3 for the red pixel and generates an interpolated horizontal offset value 620 for the red pixel. If pixel 802 is a blue pixel, the offset interpolator circuit 616 performs bilinear interpolation or bicubic interpolation on the four vertical offset values ​​of the four grid points GP0 to GP3 of the blue pixel, and generates an interpolated vertical offset value (or first offset value) 618 for the blue-red pixel, and performs bilinear interpolation or bicubic interpolation on the four horizontal offset values ​​of the four grid points GP0 to GP3 of the blue pixel, and generates an interpolated horizontal offset value (or second offset value) 620 for the blue pixel.

[0110] Return to reference Figure 6 , the offset interpolator circuit 616 provides first offset values ​​618 (e.g., vertical pixel offset values) for the red and blue channels of each pixel in the original image 602 to the vertical phase LUT 622 based on the downsampled pixel position 610 and the pre-calculated vertical offset value 614. The offset interpolator circuit 616 further provides second offset values ​​620 (e.g., horizontal pixel offset values) for the red and blue channels to the horizontal phase LUT 624 based on the pre-calculated horizontal offset value 614. In one embodiment, the vertical phase LUT 622 stores a table of interpolation coefficients (e.g., bicubic or bilinear interpolation coefficients) for a plurality of phases in a first (e.g., vertical) direction, wherein each phase has a set of coefficients (e.g., interpolation coefficients C 0 , C 1 , C 2 and C 3 Similarly, the horizontal phase LUT 624 stores a table of interpolation coefficients (e.g., bicubic or bilinear interpolation coefficients) for a plurality of phases in a second (e.g., horizontal) direction, where each phase has a set of coefficients (e.g., interpolation coefficients C 4 , C 5 , C 6 and C 7 ). Each table of interpolation coefficients is pre-calculated and associated with the same wide-angle lens associated with the offset LUT 612.

[0111] The vertical phase LUT 622 uses the first offset value 618 (e.g., vertical pixel offset) calculated for the red and blue channels of each pixel to define the phase of the bilinear or bicubic interpolation in the first (e.g., vertical) direction. Similarly, the horizontal phase LUT 624 uses the second offset value 620 (e.g., horizontal pixel offset) calculated for the red and blue channels of each pixel to define the phase of the bilinear or bicubic interpolation in the second (e.g., horizontal) direction. The phase in each of the first (e.g., vertical) and second (e.g., horizontal) directions is used as an index to its corresponding set of coefficients in the corresponding vertical phase LUT 622 and horizontal phase LUT 624.

[0112] The vertical phase LUT 622 identifies a first interpolation coefficient 626 associated with the first offset value 618 for the particular color channel, and provides the first interpolation coefficient 626 to the vertical fixation downsampling and correction circuit 630. Similarly, the horizontal phase LUT 624 identifies a second interpolation coefficient 628 associated with the second offset value 620 for the particular color channel, and provides the second interpolation coefficient 628 to the horizontal correction circuit 634.

[0113] The vertical foveation downsampling and correction circuit 630 performs combined foveation downsampling and color difference restoration in a first (e.g., vertical) direction of the original image 602. The vertical foveation downsampling and correction circuit 630 calculates corrected pixel values ​​632 having color differences corrected in the first direction relative to the original image 602. In one embodiment, the vertical foveation downsampling and correction circuit 630 uses interpolation to calculate a vertically downsampled and corrected version (P v ),Right now,

[0114] P v =C 0 P 0 +C 1 P 1 +C 2 P 2 +C 3 P 3 , (1)

[0115] Where P 0 To P 3 represents the pixel values ​​of four pixels located in the same column of the original image 602 and closest to the virtual pixel corresponding to the pixel whose value is corrected to account for vertical chromatic aberration and / or vertical gaze point downsampling, and C 0 To C 3 is the first interpolation coefficient 626.

[0116] To calculate the vertically corrected pixel value 632 for the pixel of the specific color, the vertical fixation downsampling and correction circuit 630 obtains the first interpolation coefficient 626 from the vertical phase LUT 622, i.e., retrieves the first interpolation coefficient 626 (e.g., a set of interpolation coefficients C ) corresponding to the first offset value 618 from the vertical phase LUT 622. 0 , C 1 , C 2 and C 3 ). The first offset value 618 represents a first distance (e.g., distance 704) from each downsampled pixel position 610 (or downsampled pixel position 702) to a corresponding virtual pixel (e.g., virtual pixel 706) in the first direction. Using the first offset value 618 and the first interpolation coefficient 626, the vertical gaze point downsampling and correction circuit 630 uses equation (1) to calculate a corrected pixel value 632 for a specific color channel of a pixel closest to the virtual pixel. The corrected pixel value 632 replaces the original pixel value of the specific color channel at the downsampled pixel position 610 in the first direction.

[0117] The horizontal correction circuit 634 calculates pixel values ​​636 for a particular color channel that are corrected in a second (eg, horizontal) direction with respect to the color difference of the original image 602. In one embodiment, the horizontal correction circuit 634 uses interpolation to calculate a horizontally corrected version (P h ),Right now,

[0118] P h =C 4 P 4 +C 5 P 5 +C 6 P 6 +C 7 P 7 , (2)

[0119] Where P 4 To P 7 represents the pixel values ​​of four pixels located in the same row and closest to the virtual pixel corresponding to the pixel whose value is corrected to take into account the horizontal color difference, and C 4 To C 7 is the second interpolation coefficient 628.

[0120] To calculate the horizontally corrected pixel value 636 for the pixel of a specific color, the horizontal correction circuit 634 obtains the second interpolation coefficient 628 from the horizontal phase LUT 624, i.e., retrieves the second interpolation coefficient 628 (e.g., a set of coefficients C ) corresponding to the second offset value 620 from the horizontal phase LUT 624. 4 , C 5 , C 6 and C 7). The second offset value 620 represents a second distance (e.g., distance 724) from a pixel position (e.g., the position of pixel P6) to a corresponding virtual pixel (e.g., virtual pixel 726) in a second direction. Using the second offset value 620 and the second interpolation coefficient 628, the horizontal correction circuit 634 uses equation (2) to calculate a corrected pixel value 636 for a specific color channel of a pixel closest to the virtual pixel. Since the horizontal correction circuit 634 does not perform downsampling, the corrected pixel value 636 replaces the corresponding vertical corrected pixel value 632 at the same pixel position of the vertical corrected pixel value 632.

[0121] Corrected pixel values ​​636 for pixels from the raw image 602 represent a second corrected raw image 636 vertically downsampled in both vertical and horizontal directions with reduced color difference. The image signal processor 206 may use the second corrected raw image 636 to generate a full-color image with reduced color difference.

[0122] The horizontal gaze point downsampling and scaler circuit 648 is coupled to the output terminal of the FDS-C circuit 307. The horizontal gaze point downsampling and scaler circuit 648 receives the pixel values ​​of the second corrected raw image 636 and performs horizontal gaze point downsampling and scaling on the pixel values ​​of the second corrected raw image 636.

[0123] The down-sampling scaling factor LUT 640 stores a second down-sampling scaling factor indexed by a position in a second direction (e.g., horizontal direction) of an image (e.g., a corrected raw image 636). The down-sampling scaling factor LUT 640 receives index information 638 related to the position of a corresponding pixel along the second direction in the second corrected raw image 636. The index information 638 of the corresponding pixel along the second direction in the second corrected raw image 636 is extracted by the pixel locator circuit 637. Upon receiving the index information 638, the down-sampling scaling factor LUT 640 outputs a corresponding second down-sampling scaling factor 642, which is passed to the gaze point down-sampling locator circuit 644.

[0124] The foveated downsampling locator circuit 644 receives the downsampling scaling factor 642 from the downsampling scaling factor LUT 640 and calculates a downsampling pixel position 646 (e.g., a downsampling landing point) along the second direction of the second corrected raw image 636. Information about the downsampling pixel position 646 calculated by the foveated downsampling locator circuit 644 is provided to the horizontal foveated downsampling and scaler circuit 648.

[0125] The horizontal foveated downsampling and scaler circuit 648 performs downsampling on a subset of pixels of the same color of the second corrected raw image 636 arranged in the second direction using a second downsampling scaling factor 642 that gradually varies along the second direction to generate corrected pixel values ​​for the pixels of the same color in the corrected raw image 650. The corrected pixel values ​​of the corrected raw image 650 replace the pixel value 636 of the specific color channel at the downsampled pixel position 646.

[0126] Example foveated downsampling circuit

[0127] Fig. 9A is a block diagram showing a detailed view of a first example of FDS circuit 309 according to one embodiment. FDS circuit 309 may perform gaze point downsampling on pixels in an image (e.g., a raw image generated by one or more image sensors 202, or an image in RGB format). Specifically, Fig. 9A The FDS circuit 309 in the image may perform a gaze point downsampling on an input pixel 902 in a vertical direction to generate a first downsampled pixel 922 in a first downsampled version of the image. The FDS circuit 309 may further perform a gaze point downsampling on the first downsampled pixel 922 in a horizontal direction to generate a second downsampled pixel 942 of a second downsampled version of the image. The FDS circuit 309 may include a vertical pixel locator circuit 904, a vertical scaling factor storage circuit 908, a vertical scaling factor calculator circuit 912, a vertical gaze point downsampling locator circuit 916, a vertical gaze point downsampling circuit 920 with a line buffer 903, a horizontal pixel locator circuit 924, a horizontal scaling factor storage circuit 928, a horizontal scaling factor calculator circuit 932, a horizontal gaze point downsampling locator circuit 936, and a horizontal gaze point downsampling circuit 940. The FDS circuit 309 may include a Fig. 9A Circuits more or less than those shown in .

[0128] The vertical pixel locator circuit 904 is a circuit that extracts an index 906 of each pixel 902 using information about the corresponding position of each pixel 902 along the vertical direction of the image (e.g., along the corresponding column). The vertical pixel locator circuit 904 may pass the extracted index 906 to the vertical scaling factor storage circuit 908. The vertical scaling factor storage circuit 908 may store scaling factors indexed by the position in the vertical direction of the image. The vertical scaling factor storage circuit 908 may be embodied as a LUT having a list of scaling factors indexed by the position in the vertical direction of the image. The vertical scaling factor storage circuit 908 may receive the index 906 related to the corresponding position of each pixel 902 along the vertical direction in the image. Upon receiving the index 906, the vertical scaling factor storage circuit 908 may output a corresponding scaling factor 910, which is passed to the vertical scaling factor calculator circuit 912.

[0129] The scaling factor 910 stored in the vertical scaling factor storage circuit 908 may be configured according to the configuration information 907. The configuration information 907 may configure (e.g., program) the vertical scaling factor storage circuit 908 to have a list of scaling factors 910 that vary along the vertical direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, a linear continuous distribution, some other distribution, or a combination thereof. For a piecewise fixed distribution, the vertical direction (or equivalently, the horizontal direction) may be divided into different regions (e.g., up to five different regions), and each region may be associated with a corresponding value of the scaling factor 910. For a curvature continuous distribution, the scaling factor 910 may vary continuously (e.g., along the vertical direction) at a nonlinear rate of change (e.g., defined by a corresponding curvature function). For a linear continuous distribution, the scaling factor 910 may vary continuously (e.g., along the vertical direction) at a nonlinear rate of change (e.g., defined by a slope of a corresponding linear function). The configuration information 907 may configure the rate change in the vertical direction and the number of rate changes (e.g., up to five rate changes across up to five different regions). In one or more embodiments, at least one region of rate change of scaling factor 910 in the vertical direction corresponds to zero rate change (eg, a region with constant scaling factor 910 in the vertical direction). Configuration information 907 may be received, for example, from software executed by CPU 208 .

[0130] The vertical scaling factor calculator circuit 912 is a circuit that calculates each scaling factor 914 based on one or more scaling factors 910 received from the vertical scaling factor storage circuit 908 (e.g., based on the configuration mode signal 911). In one embodiment, the configuration mode signal 911 configures the vertical scaling factor calculator circuit 912 to calculate each scaling factor 914 so that each scaling factor gradually changes along the vertical direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, a linear continuous distribution, some other distribution, or a combination thereof. In another embodiment, the configuration mode signal 911 configures the vertical scaling factor calculator circuit 912 to calculate each scaling factor 914 so that each scaling factor changes along the vertical direction at one or more change rates along the vertical direction. For example, starting at the beginning of the column (e.g., the upper edge of the column) and ending at a first vicinity of the center of the column, the first change rate of the scaling factor 914 generated by the vertical scaling factor calculator circuit 912 may be negative. Starting at a second vicinity of the center of the column and ending at the end of the column (e.g., the lower edge of the column), the second change rate of the scaling factor 914 generated by the vertical scaling factor calculator circuit 912 may be positive. In another embodiment, the configuration mode signal 911 disables the vertical scaling factor calculator circuit 912 , and the scaling factor 910 retrieved from the vertical scaling factor storage circuit 908 may be passed to the vertical gaze point downsampling locator circuit 916 as the scaling factor 914 .

[0131] The vertical gaze point downsampling locator circuit 916 is a circuit that calculates a downsampled pixel position 918 (e.g., a downsampled landing point) along the vertical direction (e.g., along a corresponding column of the image) according to the corresponding scaling factor 914. Information about the downsampled pixel position 918 may be provided to the vertical gaze point downsampling circuit 920.

[0132] The vertical foveation downsampling circuit 920 may include a line buffer 903 that buffers a set of pixels 902 belonging to a defined number of horizontal lines (e.g., rows) in an image corresponding to a scan of the image (e.g., during capture by one or more sensors 202). In one or more embodiments, the line buffer 903 is a separate component from the vertical foveation downsampling circuit 920. The vertical foveation downsampling circuit 920 may perform downsampling on a first subset of pixels 902 of the same color arranged in a vertical direction (e.g., along corresponding columns of the image) (e.g., a subset of pixels 902 stored in the line buffer 903) in a streaming manner using information about downsampled pixel locations 918 (e.g., downsampled landing points) to generate corresponding first downsampled pixels 922 of the same color in a first downsampled version of the image. The vertical gaze point downsampling circuit 920 may perform downsampling by performing interpolation (e.g., bilinear interpolation, bicubic interpolation, some other interpolation, or a combination thereof) on pixel values ​​in a first subset of pixels 902 of the same color to generate pixel values ​​of the same color for first downsampled pixels 922.

[0133] Since downsampling is performed in a streaming manner at the vertical foveation downsampling circuit 920, only a portion of the image may be received as pixels 902 from, for example, a buffer (not shown) or the system memory 230. When a sufficient number of pixels 902 of the image for generating a first downsampled pixel 922 (e.g., via interpolation) are received at the vertical foveation downsampling circuit 920 (e.g., in the line buffer 903) and are available, vertical downsampling may begin. As additional incoming pixels 902 are received, the vertical downsampling operation may continue at the vertical foveation downsampling circuit 920 to generate the first downsampled pixel 922 without waiting for the entire image. The vertical foveation downsampling circuit 920 may perform downsampling in the vertical direction in a single pass through the columns of pixels 902 (without using separate passes for different scaling factors) to reduce time and resources associated with the downsampling operation. The first downsampled pixel 922 generated by the vertical foveation downsampling circuit 920 may represent a portion of the image downsampled along the vertical direction. The first downsampled pixel 922 may be passed to a horizontal pixel locator circuit 924 and a horizontal foveation downsampling circuit 940 for further downsampling along the horizontal direction.

[0134] The horizontal pixel locator circuit 924 is a circuit that extracts an index 926 of each pixel 922 using information about the corresponding position of each pixel 922 along the horizontal direction of the image (e.g., along the corresponding row). The horizontal pixel locator circuit 924 may pass the extracted index 926 to the horizontal scaling factor storage circuit 928. The horizontal scaling factor storage circuit 928 may store scaling factors indexed by positions in the horizontal direction of the image. The horizontal scaling factor storage circuit 928 may be embodied as a LUT having a list of scaling factors indexed by positions in the horizontal direction of the image. The horizontal scaling factor storage circuit 928 may receive the index 926 related to the corresponding position of each pixel 922 along the horizontal direction in the image. Upon receiving the index 926, the horizontal scaling factor storage circuit 928 may output a corresponding scaling factor 930, which is passed to the horizontal scaling factor calculator circuit 912. The scaling factor stored in the horizontal scaling factor storage circuit 928 may be configured according to the configuration information 927. Configuration information 927 may configure (e.g., program) horizontal scaling factor storage circuit 928 to have a list of scaling factors 930 that vary along the horizontal direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, a linear continuous distribution, some other distribution, or a combination thereof. Configuration information 927 may configure the rate change in the horizontal direction and the number of rate changes (e.g., up to five rate changes across up to five different regions). In one or more embodiments, at least one region of rate change of scaling factor 930 in the horizontal direction corresponds to a zero rate change (e.g., a region with a constant scaling factor 930 in the horizontal direction). Configuration information 927 may be received, for example, from software executed by CPU 208.

[0135] The horizontal scaling factor calculator circuit 932 is a circuit that calculates each scaling factor 934 based on one or more scaling factors 930 received from the horizontal scaling factor storage circuit 928 (e.g., based on the configuration mode signal 931). In one embodiment, the configuration mode signal 931 configures the horizontal scaling factor calculator circuit 932 to calculate each scaling factor 934 so that each scaling factor gradually changes along the horizontal direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, a linear continuous distribution, some other distribution, or a combination thereof. In another embodiment, the configuration mode signal 931 configures the horizontal scaling factor calculator circuit 932 to calculate each scaling factor 934 so that each scaling factor changes along the horizontal direction at one or more change rates along the horizontal direction. For example, starting at the beginning of a row (e.g., the left edge of the row) and ending at a first vicinity of the center of the row, the first change rate of the scaling factor 934 generated by the horizontal scaling factor calculator circuit 932 may be negative. Starting at a second vicinity of the center of the row and ending at the end of the row (e.g., the right edge of the row), the second change rate of the scaling factor 934 generated by the horizontal scaling factor calculator circuit 932 may be positive. In another embodiment, the configuration mode signal 931 disables the horizontal scaling factor calculator circuit 932 , and the scaling factor 930 retrieved from the horizontal scaling factor storage circuit 928 may be passed to the horizontal fixation point downsampling locator circuit 936 as the scaling factor 934 .

[0136] The horizontal gaze point downsampling locator circuit 936 is a circuit that calculates a downsampled pixel position 938 (e.g., a downsampled landing point) along the horizontal direction (e.g., along a corresponding row of the image) according to the corresponding scaling factor 934. Information about the downsampled pixel position 938 may be provided to the horizontal gaze point downsampling circuit 940.

[0137] The horizontal foveated downsampling circuit 940 may receive a second subset of first downsampled pixels 922 arranged in a horizontal direction (e.g., along corresponding rows of the image). The horizontal foveated downsampling circuit 940 may perform downsampling on the second subset of pixels 922 of the same color in a streaming manner using information about the downsampled pixel positions 938 (e.g., downsampled landing points) to generate corresponding second downsampled pixels 942 of the same color in a second downsampled version of the image. The horizontal foveated downsampling circuit 940 may perform downsampling by performing interpolation (e.g., bilinear interpolation, bicubic interpolation, some other interpolation, or a combination thereof) on the pixel values ​​in the second subset of pixels 922 of the same color to generate pixel values ​​of the same color for the second downsampled pixels 942.

[0138] After a sufficient number of pixels 922 are received at the horizontal foveation downsampling circuit 940 for generating the second downsampled pixels 942 and these pixels are available, the horizontal foveation downsampling circuit 940 may begin to perform horizontal downsampling (e.g., via interpolation). As additional pixels 922 are received from the vertical foveation downsampling circuit 920, downsampling in the horizontal direction is performed at the horizontal foveation downsampling circuit 940 to continue generating subsequent downsampled pixels 942. The horizontal downsampling at the horizontal foveation downsampling circuit 940 is also performed in a single pass through the rows of pixels 922, rather than in separate passes for different scaling factors, thereby reducing time and resources associated with the downsampling operation. The second downsampled pixels 942 generated by the horizontal foveation downsampling circuit 940 may represent a portion of a second downsampled version of the image after performing foveation downsampling in the vertical direction and then performing foveation downsampling in the horizontal direction.

[0139] Fig. 9B is a block diagram showing a detailed view of a second example of FDS circuit 309 according to one embodiment. FDS circuit 309 may perform gaze point downsampling on pixels in an image (e.g., a raw image generated by one or more image sensors 202, or an image in RGB format). Specifically, Fig. 9B The FDS circuit 309 in the image may perform a gaze point downsampling on an input pixel 952 in a horizontal direction to generate a first downsampled pixel 972 in a first downsampled version of the image. The FDS circuit 309 may further perform a gaze point downsampling on the first downsampled pixel 972 in a vertical direction to generate a second downsampled pixel 992 of a second downsampled version of the image. The FDS circuit 309 may include a horizontal pixel locator circuit 954, a horizontal scaling factor storage circuit 958, a horizontal scaling factor calculator circuit 962, a horizontal gaze point downsampling locator circuit 966, a horizontal gaze point downsampling circuit 970, a vertical pixel locator circuit 974, a vertical scaling factor storage circuit 978, a vertical scaling factor calculator circuit 982, a vertical gaze point downsampling locator circuit 986, and a vertical gaze point downsampling circuit 990 with a line buffer 953. The FDS circuit 309 may include a ratio Fig. 9B Circuits are more or less than those shown.

[0140] The horizontal pixel locator circuit 954 is a circuit that extracts an index 956 of each pixel 952 using information about the corresponding position of each pixel 952 along the horizontal direction of the image (e.g., along the corresponding row). The horizontal pixel locator circuit 954 may pass the extracted index 956 to the horizontal scaling factor storage circuit 958. The horizontal scaling factor storage circuit 958 may store scaling factors indexed by positions in the horizontal direction of the image. The horizontal scaling factor storage circuit 958 may be embodied as a LUT having a list of scaling factors indexed by positions in the horizontal direction of the image. The horizontal scaling factor storage circuit 958 may receive the index 956 related to the corresponding position of each pixel 952 along the horizontal direction in the image. Upon receiving the index 956, the horizontal scaling factor storage circuit 958 may output a corresponding scaling factor 960, which is passed to the horizontal scaling factor calculator circuit 962. The scaling factor 960 stored in the horizontal scaling factor storage circuit 958 may be configured according to the configuration information 957. Configuration information 957 may configure (e.g., program) horizontal scaling factor storage circuit 958 to have a list of scaling factors 960 that vary along the horizontal direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, a linear continuous distribution, some other distribution, or a combination thereof. Configuration information 957 may configure the rate change in the horizontal direction and the number of rate changes (e.g., up to five rate changes across up to five different regions). In one or more embodiments, at least one region of rate change of scaling factor 960 in the horizontal direction corresponds to a zero rate change (e.g., a region with a constant scaling factor 960 in the horizontal direction). Configuration information 957 may be received, for example, from software executed by CPU 208.

[0141] The horizontal scaling factor calculator circuit 962 is a circuit that calculates each scaling factor 964 based on one or more scaling factors 960 received from the horizontal scaling factor storage circuit 958 (e.g., based on the configuration mode signal 961). In one embodiment, the configuration mode signal 961 configures the horizontal scaling factor calculator circuit 962 to calculate each scaling factor 964 so that each scaling factor gradually changes along the horizontal direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, a linear continuous distribution, some other distribution, or a combination thereof. In another embodiment, the configuration mode signal 961 configures the horizontal scaling factor calculator circuit 962 to calculate each scaling factor 964 so that each scaling factor changes along the horizontal direction at one or more change rates along the horizontal direction. For example, starting at the beginning of a row (e.g., the left edge of the row) and ending at a first vicinity of the center of the row, the first change rate of the scaling factor 964 generated by the horizontal scaling factor calculator circuit 962 may be negative. Starting at a second vicinity of the center of the row and ending at the end of the row (e.g., the right edge of the row), the second change rate of the scaling factor 964 generated by the horizontal scaling factor calculator circuit 962 may be positive. In another embodiment, the configuration mode signal 961 disables the horizontal scaling factor calculator circuit 962 , and the scaling factor 960 retrieved from the horizontal scaling factor storage circuit 958 may be passed to the horizontal fixation point downsampling locator circuit 966 as the scaling factor 964 .

[0142] The horizontal gaze point downsampling locator circuit 966 is a circuit that calculates a downsampled pixel position 968 (e.g., a downsampled landing point) along the horizontal direction (e.g., along a corresponding row of the image) according to the corresponding scaling factor 964. Information about the downsampled pixel position 968 may be provided to the horizontal gaze point downsampling circuit 920.

[0143] The horizontal foveated downsampling circuit 970 may perform downsampling on a first subset of pixels 952 of the same color arranged in a horizontal direction (e.g., along corresponding rows of the image) in a streaming manner using information about the downsampled pixel positions 968 (e.g., downsampled landing points) to generate corresponding first downsampled pixels 972 of the same color in a first downsampled version of the image. The horizontal foveated downsampling circuit 970 may perform downsampling by performing interpolation (e.g., bilinear interpolation, bicubic interpolation, some other interpolation, or a combination thereof) on the pixel values ​​in the first subset of pixels 952 of the same color to generate pixel values ​​of the same color for the first downsampled pixels 972.

[0144] Since horizontal downsampling is performed in a streaming manner at the horizontal gaze point downsampling circuit 970, only a portion of the entire image may be received and stored as pixels 952 in, for example, a buffer ( Fig. 9B950 ) or in the system memory 230. That is, when a first subset of pixels 952 sufficient to perform horizontal downsampling (e.g., via interpolation) is received at the horizontal foveation downsampling circuit 970 and these pixels are available, horizontal downsampling may begin, and downsampling may continue at the horizontal foveation downsampling circuit 970 for the incoming pixels 952 to generate first downsampled pixels 972. The horizontal foveation downsampling circuit 970 may perform downsampling in the horizontal direction in a single pass through the rows of pixels 952. The first downsampled pixels 972 generated by the horizontal foveation downsampling circuit 970 may represent a portion of the image that is downsampled along the horizontal direction. The first downsampled pixels 972 may be passed to the vertical pixel locator circuit 974 and the vertical foveation downsampling circuit 980 for further downsampling along the vertical direction of the image.

[0145] The vertical pixel locator circuit 974 is a circuit that extracts an index 976 of each pixel 972 using information about the corresponding position of each pixel 972 along the vertical direction of the image (e.g., along the corresponding column). The vertical pixel locator circuit 974 may pass the extracted index 976 to the vertical scaling factor storage circuit 978. The vertical scaling factor storage circuit 978 may store scaling factors indexed by positions in the vertical direction of the image. The vertical scaling factor storage circuit 978 may be embodied as a LUT having a list of scaling factors indexed by positions in the vertical direction of the image. The vertical scaling factor storage circuit 978 may receive an index 906 related to the corresponding position of each pixel 972 along the vertical direction in the image. Upon receiving the index 976, the vertical scaling factor storage circuit 978 may output a corresponding scaling factor 980, which is passed to the vertical scaling factor calculator circuit 982. The scaling factor 980 stored in the vertical scaling factor storage circuit 978 may be configured according to the configuration information 977. Configuration information 977 may configure (e.g., program) vertical scaling factor storage circuit 978 to have a list of scaling factors 980 that vary along the vertical direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, a linear continuous distribution, some other distribution, or a combination thereof. Configuration information 977 may configure the rate change in the vertical direction and the number of rate changes (e.g., up to five rate changes across up to five different regions). In one or more embodiments, at least one region of rate change of scaling factor 980 in the vertical direction corresponds to a zero rate change (e.g., a region with a constant scaling factor 980 in the vertical direction). Configuration information 977 may be received, for example, from software executed by CPU 208.

[0146] The vertical scaling factor calculator circuit 982 is a circuit that calculates each scaling factor 984 based on one or more scaling factors 980 received from the vertical scaling factor storage circuit 978 (e.g., based on the configuration mode signal 981). In one embodiment, the configuration mode signal 981 configures the vertical scaling factor calculator circuit 982 to calculate each scaling factor 984 so that each scaling factor gradually changes along the vertical direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, a linear continuous distribution, some other distribution, or a combination thereof. In another embodiment, the configuration mode signal 981 configures the vertical scaling factor calculator circuit 982 to calculate each scaling factor 984 so that each scaling factor changes along the vertical direction at one or more change rates along the vertical direction. For example, starting at the beginning of the column (e.g., the upper edge of the column) and ending at a first vicinity of the center of the column, the first change rate of the scaling factor 984 generated by the vertical scaling factor calculator circuit 982 may be negative. Starting at a second vicinity of the center of the column and ending at the end of the column (e.g., the lower edge of the column), the second change rate of the scaling factor 984 generated by the vertical scaling factor calculator circuit 982 may be positive. In another embodiment, the configuration mode signal 981 disables the vertical scaling factor calculator circuit 982 and the scaling factor 980 retrieved from the vertical scaling factor storage circuit 978 may be passed to the vertical gaze point downsampling locator circuit 986 as the scaling factor 984 .

[0147] The vertical gaze point downsampling locator circuit 986 is a circuit that calculates a downsampled pixel position 978 (e.g., a downsampled landing point) along the vertical direction (e.g., along a corresponding column of the image) according to the corresponding scaling factor 984. Information about the downsampled pixel position 988 may be provided to the vertical gaze point downsampling circuit 990.

[0148] The vertical foveated downsampling circuit 990 may include a line buffer 953 that buffers a set of pixels 972 belonging to a defined number of horizontal lines (e.g., rows) in a first downsampled version of the image, the defined number of horizontal lines corresponding to a scan of the image (e.g., during capture by one or more sensors 202). In one or more embodiments, the line buffer 953 is a separate component from the vertical foveated downsampling circuit 990. The vertical foveated downsampling circuit 990 may perform downsampling on a second subset of pixels 972 of the same color arranged in a vertical direction (e.g., along corresponding columns of the image) (e.g., a subset of pixels 972 stored in the line buffer 953) in a streaming manner using information about downsampled pixel positions 988 to generate second downsampled pixels 992 of the same color in a second downsampled version of the image. The vertical gaze point downsampling circuit 990 may perform downsampling by performing interpolation (e.g., bilinear interpolation, bicubic interpolation, some other interpolation, or a combination thereof) on pixel values ​​in a second subset of pixels 972 of the same color to generate pixel values ​​of the same color for second downsampled pixels 992.

[0149] When a sufficient number of pixels 972 for performing interpolation in the vertical direction are received and stored in the line buffer 953, the vertical foveated downsampling circuit 990 may begin a vertical downsampling operation. The vertical foveated downsampling circuit 990 may continue to receive pixels 972 and perform downsampling in the vertical direction in a single pass through the columns of pixels 972 to generate additional second downsampled pixels 992. The second downsampled pixels 992 generated by the vertical foveated downsampling circuit 990 may represent a portion of a second downsampled version of the image after performing foveated downsampling in the horizontal direction and then performing foveated downsampling in the vertical direction.

[0150] Fig. 10A is an example block diagram of a raw processing stage 306 with an FDS circuit 309 according to one embodiment. Fig. 10A The raw processing stage 306 in FIG. 1 includes a raw scaling circuit 1004 followed by an FDS circuit 309, among other components. A version of a raw image (e.g., an image obtained by one or more image sensors 202) of pixels 1002 may be processed by other components of the raw processing stage 306 ( Fig. 10A The raw scaling circuit 1004 may be used to generate and provide a raw image to the original image (not shown) and to provide the raw image to the original image. The raw scaling circuit 1004 may perform scaling on the pixel values ​​1002 of each color component in the version of the original image (e.g., to adjust the gain of each color component) to generate pixels 902 having scaled pixel values, which are provided to the FDS circuit 309. In some embodiments, the raw scaling circuit 1004 is bypassed. Fig. 9AAs discussed in detail, the FDS circuit 309 may generate downsampled pixels 942 for each color component in the downsampled version of the original image. The downsampled pixels 942 may be provided to the resampling processing stage 308 for performing, for example, color correction, resampling, and / or scaling on the downsampled pixels 942. Alternatively, the downsampled pixels 942 may be provided to, for example, the system memory 230 or the persistent storage 228 for later processing.

[0151] Fig. 10B is another example block diagram of a raw processing stage 306 with an FDS circuit 309 according to one embodiment. Fig. 10A The raw processing stage 306 in FIG. 1 includes an FDS circuit 309, followed by a raw scaling circuit 1006, and other components. The pixel 902 of the version of the raw image (e.g., an image obtained by one or more image sensors 202) may be processed by other components of the raw processing stage 306 ( Fig. 10B ) is generated and provided to the FDS circuit 309. Fig. 9A As discussed in detail, the FDS circuit 309 may generate downsampled pixels 942 for each color component in the downsampled version of the original image. The downsampled pixels 942 may be provided to the raw scaling circuit 1006. The raw scaling circuit 1006 may perform scaling (e.g., to adjust the gain of each color component) on the pixel values ​​of each color component of the downsampled pixels 942 to generate scaled downsampled pixels 1008. The raw scaling circuit 1006 may perform substantially the same processing operations as the raw scaling circuit 1004. In some embodiments, the raw scaling circuit 1006 is bypassed. The scaled downsampled pixels 1008 may be provided to the resampling processing stage 308 for performing, for example, color correction, resampling, and / or scaling on the scaled downsampled pixels 1008. Alternatively, the scaled downsampled pixels 1008 may be provided to, for example, the system memory 230 or the persistent memory 228 for later processing.

[0152] Fig. 10C 308 with FDS circuit 309 according to one embodiment. The raw processing stage 306 may generate processed raw image data 1010, which is provided to the resampling processing stage 308. One or more components of the resampling processing stage 308 may convert the processed raw image data 1010 into image data in a color format, such as image data 1012R, 1012G, 1012B in RGB format, which is provided to the FDS circuit 309. The FDS circuit 309 may be associated with Fig. 9A and Fig. 9BThe FDS circuit 309 may operate in substantially the same manner as described above, and the FDS circuit 309 may apply foveated downsampling to different color channels of pixels in the image data 1012R, 1012G, 1012B. The FDS circuit 309 may generate corresponding downsampled versions of the image data 1014R, 1014G, 1014B in a color format (e.g., RGB format), which may be provided to one or more other components of the resampling processing stage 308. The resampling processing stage 308 may output resampled image data 1016, which may be passed to, for example, the noise processing stage 310. Alternatively, the resampled image data 1016 may be provided to, for example, the system memory 230 or the persistent storage 228 for later processing.

[0153] Example process of fixation downsampling and correction circuit

[0154] Fig.11 1 is a flow chart illustrating a method for performing foveated downsampling and correction by an image processor (e.g., ISP 206) to reduce color fringing of raw image data according to one embodiment. The image processor performs 1102 a combined vertical foveated downsampling and interpolation of pixel values ​​of a first subset of pixels of the same color in an raw image (e.g., pixels of raw image 602) using a downsampling scaling factor (e.g., first downsampling scaling factor 606) and a first interpolation coefficient (e.g., first interpolation coefficient 626) to generate first corrected pixel values ​​(e.g., corrected pixel values ​​632) of the pixels of the same color in a first corrected version of the raw image. The pixels in the first subset are arranged in a first direction (e.g., a vertical direction), the downsampling scaling factor gradually changes along the first direction, and the first interpolation coefficient corresponds to a first offset value (e.g., first offset value 618).

[0155] The first offset value represents a first distance (e.g., distance 704, 714) from each downsampled pixel position along the first direction (e.g., each downsampled pixel position 610, 702, 712) to a corresponding first virtual pixel in the first direction (e.g., virtual pixel 706, 716). The image processor generates one of the first corrected pixel values ​​of the pixel in the first corrected version (e.g., by vertical foveation downsampling and correction circuit 630) by downsampling and interpolating a plurality of pixels in the same column of the original image using a corresponding one of the downsampled scaling factors and a corresponding subset of the first interpolation coefficients.

[0156] The image processor receives 1104 a first corrected pixel value (e.g., corrected pixel value 632) of the first corrected version (e.g., via horizontal correction circuit 634). The image processor interpolates 1106 the pixel values ​​of a second subset of pixels in the first corrected version (e.g., via horizontal correction circuit 634) using second interpolation coefficients (e.g., second interpolation coefficients 628) to generate second corrected pixel values ​​(e.g., corrected pixel value 636) of the same color pixels in the second corrected version of the original image. The pixels in the second subset are arranged in a second direction (e.g., horizontal direction) perpendicular to the first direction, and the second interpolation coefficients correspond to second offset values ​​(e.g., second offset value 620).

[0157] The second offset value represents a second distance (e.g., distance 724, 734) from the second subset of pixels to a corresponding second virtual pixel (e.g., virtual pixel 726, 736) in the second direction. The image processor generates (e.g., via horizontal correction circuit 634) one of the second corrected pixel values ​​for the pixels in the second corrected version by interpolating a plurality of pixels in the same row of the first corrected version using the corresponding subset of the second interpolation coefficients.

[0158] A first subset of pixels is in the same column of an original image having a Bayer pattern, and a second subset of pixels is in the same row of a first corrected version of the original image having a Bayer pattern. The value of each down-sampling scaling factor depends on the position of a plurality of pixels in the same column of the original image along a first direction, the plurality of pixels being used for down-sampling and interpolation to generate corresponding corrected pixel values ​​of the pixels in the first corrected version. In some embodiments, the down-sampling scaling factor gradually changes along the first direction based on, for example, a piecewise fixed distribution, a curvature continuous distribution, a linear continuous distribution, some other distribution, or a combination thereof. In one or more embodiments, one or more portions of the down-sampling scaling factor are gradually scaled down at each defined down-sampling pixel position in the first direction, and one or more other portions of the down-sampling scaling factor are gradually scaled up at each defined down-sampling pixel position in the first direction.

[0159] The image processor may further perform (e.g., by horizontal foveation downsampling and scaler circuit 648) horizontal foveation downsampling and scaling on the second corrected pixel values ​​(e.g., corrected pixel values ​​636) of the same color pixels in the second corrected version. The image processor may downsample a subset of the same color pixels of the second corrected version using a second downsampling scaling factor (e.g., second downsampling scaling factor 642) to generate corrected pixel values ​​of the same color pixels in the corrected version of the original image (e.g., corrected original image 650). The pixels in the subset are arranged in a second (e.g., horizontal) direction, and the second downsampling scaling factor varies gradually along the second direction.

[0160] As mentioned above Fig.11 The embodiments of the processes described are merely illustrative. In addition, the order of the processes may be modified or omitted.

[0161] Example process of the foveated downsampling circuit

[0162] Fig.12 1202 (e.g., via vertical foveation downsampling circuit 920 or horizontal foveation downsampling circuit 970 in FDS circuit 309) of the same color in an image to generate first downsampled pixels of the same color in a first downsampled version of the image (e.g., pixels 922 or pixels 972), the first subset of pixels being arranged in a first direction (e.g., vertical direction or horizontal direction). The image processor may downsample the first subset of pixels in a single pass along the first direction. The image processor may determine each downsampled pixel position in the first direction based on a corresponding first one of the first scaling factors (e.g., via vertical foveation downsampling locator circuit 916 or horizontal foveation downsampling locator circuit 966 in FDS circuit 309).

[0163] The first subset of pixels may be in the same column of the image in raw format or color format (e.g., RGB format). Alternatively, the first subset of pixels may be in the same row of the image. The first scaling factor may gradually change along the first direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, or a linear continuous distribution. In some embodiments, the first set of first scaling factors gradually scales down at each defined downsampled pixel position in the first direction, and the second set of first scaling factors gradually scales up at each defined downsampled pixel position in the first direction. Alternatively, the first set of first scaling factors may scale down at one or more first rate changes (e.g., one or more negative rate changes) in the first direction, and the second set of first scaling factors may scale up at one or more second rate changes (e.g., one or more positive rate changes) in the first direction. The second set of first scaling factors may be different from the first set of first scaling factors, and the second set may include one or more first scaling factors of the first set. The first scaling factor may change (e.g., scale up and scale down) in the first direction according to, for example, up to five rate changes. The number of rate changes in the first direction may be configurable.

[0164] The image processor receives 1204 a second subset of first downsampled pixels (e.g., pixels 922 or pixels 972) arranged in a second direction (e.g., horizontal direction or vertical direction). The image processor downsamples 1206 the second subset of pixels of the same color using a second scaling factor (e.g., by horizontal foveation downsampling circuitry 940 or vertical foveation downsampling circuitry 990) to generate second downsampled pixels of the same color (e.g., pixels 942 or pixels 992) in a second downsampled version of the image. The image processor may perform downsampling on the second subset of pixels in a single pass along the second direction. The image processor may determine each downsampled pixel position in the second direction based on a corresponding first scaling factor in the second scaling factors (e.g., via vertical foveation downsampling locator circuitry 936 or horizontal foveation downsampling locator circuitry 986).

[0165] The second subset of pixels may be in the same row of the first downsampled version of the image, for example, in the original format or color format. Alternatively, the second subset of pixels may be in the same column of the first downsampled version of the image. The second scaling factor may gradually change along the first direction according to, for example, a piecewise fixed distribution, a curvature continuous distribution, or a linear continuous distribution. In some embodiments, the first set of second scaling factors is gradually scaled down at each defined downsampled pixel position in the second direction, and the second set of second scaling factors is gradually scaled up at each defined downsampled pixel position in the second direction. Alternatively, the first set of second scaling factors may be scaled down in the second direction with one or more third rate changes (e.g., one or more negative rate changes), and the second set of second scaling factors may be scaled up in the second direction with one or more fourth rate changes (e.g., one or more positive rate changes). The second set of second scaling factors may be different from the first set of second scaling factors, and the second set may include one or more second scaling factors of the first set. The second scaling factor may change (e.g., scale up and scale down) in the second direction according to, for example, up to five rate changes. The number of rate changes in the second direction may be configurable.

[0166] As mentioned above Fig.12 The embodiments of the processes described are merely illustrative. In addition, the order of the processes may be modified or omitted.

[0167] While specific embodiments and applications have been illustrated and described, it should be understood that the invention is not limited to the precise construction and components disclosed herein, and that various modifications, changes and variations that will be apparent to those skilled in the art may be made to the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope of the present disclosure.

Claims

1. An image processor, comprising: a first downsampling circuit configured to downsample a first subset of pixels of the same color in an image using a first scaling factor to generate first downsampled pixels of the same color in a first downsampled version of the image, the first subset of pixels being arranged in a first direction; and A second down-sampling circuit, the second down-sampling circuit is coupled to the first down-sampling circuit, and the second down-sampling circuit is configured as: receiving a second subset of the first downsampled pixels arranged in a second direction, and A second subset of the pixels of the same color is downsampled using a second scaling factor to generate second downsampled pixels of the same color in a second downsampled version of the image.

2. The image processor according to claim 1, wherein: The first downsampling circuit is further configured to downsample a first subset of the pixels in a single pass along the first direction; and The second downsampling circuit is further configured to downsample a second subset of the pixels in a single pass along the second direction. 3 . The image processor of claim 1 , wherein the first scaling factor changes gradually along the first direction, and the second scaling factor changes gradually along the second direction. The image processor according to claim 1 , wherein the first direction is a vertical direction, and the second direction is a horizontal direction.

5. The image processor of claim 1, wherein the first subset of pixels are in the same column of the image in raw format or color format and the second subset of pixels are in the same row of the first downsampled version of the image.

6. The image processor of claim 1, wherein the first subset of pixels are in the same row of the image in raw format or color format and the second subset of pixels are in the same column of the first downsampled version of the image. 7 . The image processor of claim 1 , wherein at least one of the first scaling factor and the second scaling factor gradually changes along at least one of the first direction and the second direction based on a piecewise fixed distribution, a curvature continuous distribution, or a linear continuous distribution.

8. The image processor according to claim 1, wherein: The first set of first scaling factors is gradually scaled down at each defined downsampled pixel position in the first direction; The second set of first scaling factors is gradually scaled up at each defined downsampled pixel position in the first direction; The first set of second scaling factors is gradually scaled down at each defined downsampled pixel position in the second direction; and The second set of second scaling factors is progressively scaled up at each defined downsampled pixel position in the second direction.

9. The image processor according to claim 1, wherein: The first set of first scaling factors scales down in the first direction at one or more first rates of change; and a second set of the first scaling factors scaled up in the first direction at one or more second rates of change; the first set of second scaling factors is scaled down in the second direction at one or more third rates of change; and The second set of second scaling factors is scaled up in the second direction at one or more fourth rate changes.

10. An image processor according to claim 9, wherein at least one of the one or more first rate changes is negative, at least one of the one or more second rate changes is positive, at least one of the one or more third rate changes is negative, and at least one of the one or more fourth rate changes is positive.

11. The image processor according to claim 1, further comprising: a first down-sampled locator circuit coupled to the first down-sampled circuit, the first down-sampled locator circuit configured to determine each down-sampled pixel position in the first direction based on a corresponding first one of the first scaling factors; and A second down-sampled locator circuit, coupled to the second down-sampled circuit, is configured to determine each down-sampled pixel position in the second direction based on a corresponding second one of the second scaling factors.

12. A method comprising: downsampling a first subset of pixels of the same color in an image using a first scaling factor to generate first downsampled pixels of the same color in a first downsampled version of the image, the first subset of pixels being arranged in a first direction; as well as A second subset of the first downsampled pixels of the same color is downsampled using a second scaling factor to generate second downsampled pixels of the same color in a second downsampled version of the image, the second subset of pixels being arranged in a second direction.

13. The method according to claim 12, further comprising: downsampling a first subset of the pixels in a single pass along the first direction; as well as A second subset of the pixels is downsampled in a single pass along the second direction.

14. The method of claim 12, wherein the first subset of pixels are in the same column of the image in raw format or color format and the second subset of pixels are in the same row of the first downsampled version of the image.

15. The method of claim 12, wherein: The first set of first scaling factors is gradually scaled down at each defined downsampled pixel position in the first direction; and The second set of first scaling factors is progressively scaled up at each defined downsampled pixel position in the first direction.

16. The method of claim 12, wherein: The first set of first scaling factors scales down in the first direction at a negative rate of change; and The second set of first scaling factors scales up in the first direction at a positive rate of change.

17. The method according to claim 12, further comprising: determining each downsampled pixel position in the first direction based on a corresponding first scaling factor in the first scaling factors; as well as Each down-sampled pixel position in the second direction is determined based on a corresponding second scaling factor of the second scaling factors.

18. A system, comprising: at least one image sensor configured to capture an image; and an image processor, the image processor being coupled to the at least one image sensor, the image processor comprising: a first downsampling circuit configured to downsample a first subset of pixels of the same color in the image using a first scaling factor to generate first downsampled pixels of the same color in a first downsampled version of the image, the first subset of pixels being arranged in a first direction; and A second down-sampling circuit, the second down-sampling circuit is coupled to the first down-sampling circuit, and the second down-sampling circuit is configured as: receiving a second subset of the first downsampled pixels of the first downsampled version, the second subset of the first downsampled pixels being arranged in a second direction, and A second subset of the pixels of the same color is downsampled using a second scaling factor to generate second downsampled pixels of the same color in a second downsampled version of the image.

19. The system of claim 18, wherein: The first downsampling circuit is further configured to downsample a first subset of the pixels in a single pass along the first direction; and The second downsampling circuit is further configured to downsample a second subset of the pixels in a single pass along the second direction.

20. The system of claim 18, wherein: The first set of first scaling factors is gradually scaled down at each defined downsampled pixel position in the first direction; and The second set of first scaling factors is progressively scaled up at each defined downsampled pixel position in the first direction.

Images