Method and system for camera control and image processing having a multi-frame based window for image data statistics

Abstract

Methods, systems, and articles of manufacture relate to camera control and image processing with multi-frame based windowing for image data statistics.

Description

Background Art

[0001] Digital image processing devices on multi-function devices such as smartphones or on dedicated digital cameras use automatic features to improve the quality of images, such as the preview screen on the digital camera and the recorded images and recorded videos. This includes features for setting parameters to capture images, such as the 3A features of autofocus (AF) and automatic exposure control (AEC). This also includes other features that modify image data during processing of captured images (such as the 3A Auto White Balance (AWB) feature and other non-3A features), or algorithms that use statistical information of the frame (such as local tone mapping and global tree mapping (LTM and GTM) for dynamic range conversion, and digital video stabilization (DVS)).

[0002] Digital image processing devices use AWB to provide accurate colors in pictures reproduced from captured images. AWB is the process of finding or defining the color white in the picture, known as the white point. Other colors in the picture are determined relative to the white point. AWB adjusts the gain of different color components (e.g., red, green, and blue) relative to each other to render white objects as white, despite differences in color temperature or varying sensitivities of the image scene.

[0003] Autofocus automatically adjusts the camera lens position to provide proper focus on objects in the scene being video recorded. In various implementations, a digital camera may use phase detection autofocus (PDAF) (PDAF is also known as phase autofocus or phase-based autofocus), contrast-based autofocus, or both. Phase detection autofocus separates left and right light rays passing through the camera lens to sense left and right images. The left and right images are compared, and the difference in image position on the camera sensor can be used to determine the shift of the camera lens for autofocus. Contrast autofocus measures the brightness contrast at multiple lens positions until a maximum contrast is reached. The intensity difference between adjacent pixels of the camera sensor naturally increases with proper image focus, so that the maximum contrast indicates proper focus. Other methods of AF may also be used.

[0004] Automatic exposure control is used to automatically calculate and adjust the correct exposure necessary to capture and generate good-quality images. Exposure is the amount of incident light captured by the sensor and can be adjusted by adjusting the camera's aperture size and shutter speed, as well as neutral density (ND) filter control (if present) and flash power, some of which may be electronic systems rather than mechanical devices. ND filters are sometimes used with a mechanical shutter when the mechanical shutter speed is not fast enough for the brightest lighting conditions. The AEC can also calculate analog gain and digital gain (if present), which amplify the raw image signal generated by the exposure time used. Together, the exposure parameters determine the total exposure time, referred to herein as the total exposure. Gain affects the signal level or brightness of the raw (RAW) image coming out of the camera sensor. If the total exposure is too short, the image will appear darker than the actual scene, which is called underexposure. If the signal falls below the noise floor or is quantized to zero, the image may be so underexposed that it is lost. On the other hand, if the total exposure is too long, the output image will appear brighter than the actual scene, which is called overexposure. Both situations can result in a loss of detail, leading to poor image quality.

[0005] To perform these 3A adjustments and processing modifications provided by other algorithms such as the LTM and GTM mentioned above, the camera system captures raw image data for a frame and provides the raw image data to one or more processors, such as an image signal processor (ISP). The ISP then performs calculations on the raw image data to generate 3A statistics for the captured image that essentially indicate the range and location of luma and / or chroma pixel values, or provides other statistical tables, maps, and / or graphs based on luma and / or chroma values. These 3A and other statistics are generated for a single frame of a video sequence and on a frame-by-frame basis. The statistics can then be used to generate new parameters for setting 3A and other controlled camera settings for subsequent frames.

[0006] Similarly, in conventional digital camera systems, a certain amount of time is consumed to generate 3A and other statistical information, transfer data to and from memory (such as dynamic random access memory (DRAM)) that stores image data and / or other statistical information, and calculate new 3A parameters to apply to camera settings for capturing the next image, or to modify image data for displaying the image. This amount of time often takes too long, resulting in a wait time for applying the new 3A parameters, so that instead of applying the new parameters to the next frame after analyzing the current frame, the new parameters are not applied until at least two or three frames away from the currently analyzed frame. This delay can result in lower quality images that are noticeable to the user. Some "on-the-fly" systems attempt to address this problem by reducing memory transactions and applying the new parameters to the next frame regardless of whether the analysis is complete. However, this can also result in noticeable artifacts and lower quality images. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The material described herein is illustrated by way of example and not limitation in the accompanying drawings. For simplicity and clarity of illustration, the elements shown in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Furthermore, where deemed appropriate, reference numerals may be repeated across the figures to indicate corresponding or similar elements. In the drawings:

[0008] Figure 1 is a schematic diagram of a video frame processing time diagram for conventional 3A operations;

[0009] Figure 2 is a schematic diagram of a video frame processing time diagram for 3A operation according to at least one of the implementations herein;

[0010] Figure 3 is a flow chart of a method of camera control and image processing with multi-frame based windowing for image data statistics according to at least one of the implementations herein;

[0011] Figures 4A-4B is a schematic timing diagram illustrating 3A operations for camera control and image processing when using a rolling shutter according to at least one of the implementations herein;

[0012] Figure 5 is a detailed flow chart of a method of camera control and image processing with multi-frame based windowing for image data statistics according to at least one of the implementations herein.

[0013] Figure 6 is a schematic diagram of a process of 3A operation in a non-operational mode according to at least one of the implementations herein;

[0014] Figure 7 is another schematic diagram of a process of 3A operation in a partially operational mode according to at least one of the implementations herein;

[0015] Figure 8 is another schematic diagram of a process of 3A operation in a fully operational mode according to at least one of the implementations herein;

[0016] Figure 9 A method of camera control and image processing with multi-frame based windowing for image data statistics in operation by an image processing device according to at least one of the implementations herein;

[0017] Figure 10 is a diagram of an example image processing system;

[0018] Figure 11 is a diagram of an example system; and

[0019] Figure 12 is an illustration of an example system that is arranged entirely in accordance with at least some implementations of the present disclosure. DETAILED DESCRIPTION

[0020] One or more implementations will now be described with reference to the accompanying drawings. Although specific configurations and arrangements are discussed, it should be understood that this is done for illustrative purposes only. Those skilled in the relevant art will recognize that other configurations and arrangements may be employed without departing from the spirit and scope of this specification. It will be apparent to those skilled in the relevant art that the various techniques and / or arrangements described herein may also be employed in a variety of other systems and applications beyond those described herein.

[0021] Although the following description sets forth various implementations that may be displayed in an architecture such as, for example, a system-on-a-chip (SoC) architecture, implementations of the various techniques and / or arrangements described herein are not limited to a particular architecture and / or computing system and may be implemented by any architecture and / or computing system used for similar purposes. For example, the techniques and / or arrangements described herein may be implemented using, for example, a variety of architectures using multiple integrated circuit (IC) chips and / or packages, and / or various computing devices and / or consumer electronics (CE) devices such as set-top boxes, smartphones, cameras, laptop computers, tablet computers, and the like. Furthermore, although the following description may set forth numerous specific details such as logic implementation, types and interrelationships of system components, logic partitioning / integration options, etc., the claimed subject matter may be implemented without such specific details. In other instances, certain material such as, for example, control structures and complete software instruction sequences may not be shown in detail to avoid obscuring the disclosure herein.

[0022] The content disclosed herein may be implemented in hardware, firmware, software, or any combination thereof. The content disclosed herein may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any medium and / or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include a read-only memory (ROM); a random access memory (RAM); a magnetic disk storage medium; an optical storage medium; a flash memory device; an electrical, optical, acoustic, or other form of propagated signal (e.g., a carrier wave, an infrared signal, a digital signal, etc.), and the like. In another form, a non-transient article such as a non-transient computer-readable medium may be used in conjunction with any of the above examples or other examples, except that it does not essentially include transient signals. It essentially includes, in addition to signals, those elements that can temporarily store data in a "transient" manner such as RAM.

[0023] References in the specification to "one implementation," "implementation," "example implementation," etc., indicate that the described implementation may include a particular feature, structure, or characteristic, but not every implementation may include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an implementation, it is considered within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in conjunction with other implementations, whether or not explicitly described herein.

[0024] Systems, articles of manufacture, and methods for camera control and image processing with multi-frame based windows for image data statistics

[0025] As mentioned above, many digital cameras and multi-purpose devices having one or more cameras provide automatic control functions, such as the 3A features including autofocus (AF) to set the lens position on the camera, automatic exposure control (AEC) to set the exposure time, and automatic white balance (AWB) to optimize the colors displayed on the resulting image. As also mentioned, each of these controls uses statistical information in order to generate new parameters or settings to perform these functions. The generation of these 3A and other statistics is limited to a frame-by-frame basis. These other statistics are used by LTM or GTM (local or global tone mapping), which are also frame-by-frame and are used to calculate frame processing adjustments, to point out one possible example of other non-3A statistics used.

[0026] More specifically, by way of example regarding autofocus (AF), a camera can use phase detection autofocus (PDAF) (PDAF is also known as phase autofocus or phase-based autofocus), contrast-based autofocus, or both. Phase detection autofocus separates left and right light rays passing through the camera lens to sense left and right images. The left and right images are compared, and the difference in image position on the camera sensor can be used to determine the shift of the camera lens for autofocus. Phase detection autofocus can allow a digital camera to determine the correct direction and amount of lens position movement based on a single frame of captured information.

[0027] Alternatively, autofocus can be implemented using contrast measurements. Contrast is typically calculated as the sum of image gradients in the x (horizontal) direction (alternatively the sum of gradients in the x and y directions) in chrominance or luminance pixel values (or in other words, across image pixels in the Bayer or YUV domain). In contrast autofocus techniques, luminance contrast is measured at multiple lens positions until a maximum contrast is reached. The intensity difference between adjacent pixels of a camera's sensor naturally increases with correct image focus, so that maximum contrast indicates correct focus. To determine compensating lens shift when using this technique, 3A statistics include contrast or gradient maps and their sums, as well as sums of frames for contrast autofocus and / or maps of the intensity or color difference of pixel pairs for phase detection autofocus, for left and right images.

[0028] Another autofocus value that may be used is a measure of sharpness. Sharpness herein refers to a measure of the blurriness at the boundaries between regions of an image of different hues and / or colors. Typically, the sharpness or focus value may be calculated based on the convolution of a high pass filter with the image data calculated line by line. The modulation transfer function (MTF) and spatial frequency response (SFR) are some of the techniques that may be used to measure sharpness. It will be understood that sharpness herein may be a relative term measured on a possible scale that may be used for an image and is referred to without units.

[0029] Thus, the 3A statistics used for autofocus may include a contrast or gradient map (and / or a sum thereof), the difference between the left and right phase images, the difference between the upper and lower phase images, a sharpness value and / or a contrast value. A low resolution representation of the image may also be used to represent the intensity level of the image signal. The statistical information data can be extracted from multiple locations within the image reconstruction pipeline / image signal processor (ISP). For example, the data can be extracted after applying black level correction, linearization and shading correction. In addition to the image statistics, the algorithm can also use metadata (such as effective exposure parameters) and results from other camera control algorithms as additional inputs to the algorithm calculations. These autofocus statistics can then be used by the AF control or logic component to process the data in the AF algorithm and make decisions about where to move the lens next.

[0030] With respect to automatic exposure control (AEC), the statistical information may include a luminance histogram or other intensity measurement graph, a low-resolution representation of the original image, and, as an example, a low-resolution local or global luminance histogram in the form of a frame's worth of data to be used to calculate the frame's average luminance, to then generate digital and analog gain, and thus exposure time. These statistical information may also be used by other 3A controls.

[0031] With respect to automatic white balance (AWB), methods such as colorization by correlation, gamut mapping, grayscale edge, and / or gray world AWB can be used to determine the white point and RGB gains. The 3A statistics used can include the color average (a low-resolution representation of the original image) for each color component.

[0032] Other general statistics that may be generated, whether for 3A operation or other functions, may include luma and / or chroma averages, luma and / or chroma high frequency and texture content, motion content from frame to frame, any other color content values, picture statistics regarding deblocking control (e.g., information that controls deblocking and / or non-deblocking), RGBS grids, filter response grids, and RGB histograms, to name a few. One specific type of general statistics is local or global tone mapping (LTM or GTM) techniques, which are used, for example, to convert between high dynamic range color and low dynamic range, or may be used for other algorithms. Other statistics may be used for digital video stabilization, also known as electronic image stabilization (EIS), which is used in some cameras to shift the electronic image from video frame to video frame and is sufficient to counteract camera motion (such as jitter from a user's hand). This technique may use pixels outside the boundaries of the visible frame to provide a buffer for motion and reduce disturbing vibrations from the video by smoothing the transition from one frame to another. The technique uses pixel data and statistical information including local motion vectors (ie, a map of motion from one frame to another), and it includes data from outside the mentioned boundaries.

[0033] Other general statistics can be used with the multi-frame statistics window described herein, and the description herein is not limited to any one type of statistics. In general, all collected statistics can be used by any of the camera control algorithms. That is, one type of statistic is not limited to one type of camera control algorithm.

[0034] Most statistics in the field of imaging and camera algorithms are traditionally calculated, collected, and processed at the frame level. Therefore, a short window can be set in which new 3A parameters are requested, 3A statistics are generated, and transmitted to the 3A components for use. If the window closes before the transmission is complete, the new settings are not used in the next image or frame from the sensor, but rather in the next available frame, which may be two or three frames away from the current frame being analyzed. These delays or latencies can occur due to bandwidth limitations at the control processing unit (CPU) (whether in the camera sensor control, image signal processor (ISP), or other shared processors), which can cause the camera control module or processor to be busy with other processing tasks. Otherwise, delays can simply be caused by bandwidth limitations in the data transfer paths between any of the system components, including memory transactions to and from memory such as DRAM that stores image data and / or statistics. These delays increase the risk of noticeably low image quality due to blurred image portions (for AF), sudden brightness changes (for AEC), and incorrect displayed colors (for AWB). Other impacts or delayed functionality may also occur due to delays in other general statistics mentioned above.

[0035] For example, reference Figure 1 , video processing sequence 100 illustrates these shortcomings, the captured frames of the video sequence are shown as including image data input from frame FN-1 (102) to frame FN+3 (110). In this example, the input data is obtained and 3A and other statistical information is generated for frame FN (104) during a 3A statistics calculation (stat.calc.) period 111 and by using, for example, an ISP pipeline. The statistical information is then stored along with the image data in a frame buffer such as DRAM or other memory (unless an on-the-fly operation is being performed) and used during a 3A library processing unit 112 to generate new parameters. Depending on the type of statistical information and the amount of computation required, this analysis of the statistical information can occur on some processor during the capture and reception of the input for frame FN+1 (106) during Vblank or after. Depending on the latency of generating the new parameters and whether the new parameters are generated before the deadline for each frame, the new parameters are applied to the next available frames FN+1, FN+2 and / or FN+3. As shown, the application module 114 can be a focus module or other 3A application module, or a non-3A application module such as for dynamic range conversion using LTM and / or GTM, DVS, etc. The application module 114 receives autofocus parameters and can have sufficient time to apply those parameters during the input of frame FN+1 (106) to improve lens focus for at least some of the integral or pixel rows (or rows) on frame FN+1 and set for subsequent frames such as frame FN+2, etc. Auto-exposure or other parameters can be applied by the sensor, and auto-white balance parameters can be applied to the image data after the parameters are generated by the sensor / imaging pipeline 116. In this example, the waiting time shown causes the parameters to be applied to frame FN+2 or, alternatively, frame FN+3 (as shown in dashed lines). In short, due to the time consumed for frame buffering, and thus more memory bandwidth and power consumption, as well as the other reasons mentioned above, when these waiting times do not allow all or other parameters of the new 3A based on the FN (104) data to be applied to improve a more substantial part of the immediately next frame FN+1, a noticeably lower quality image may result.

[0036] As also mentioned, this can be particularly problematic for some algorithms if the imaging pipeline and processing is performed "on the fly" (OTF) without storing image data in DRAM for application of 3A or other parameters on the immediately following frame. In these cases, whatever portion of the statistics cannot be generated in time, or whatever portion of the new parameter calculation cannot be performed in time, is simply skipped. This can alter all or part of the next frame by partially applying new parameters generated so far or up to the frame deadline, which makes it into the deadline window for parameter application to the next frame, or simply skipping some 3A improvements. If "on the fly" techniques are used to apply new parameters to the next frame FN+1 regardless of the conditions of the new parameters, there often is not enough time to generate acceptable new parameters and apply them in time before capture of the next frame begins (as application in the middle of a frame is typically unacceptable and can result in undesirable artifacts).

[0037] Another difficulty arises regarding the time consumed by memory transactions involving 3A statistics and many other general statistics used to improve image quality. As an example mentioned above, such general statistics can include local or global tone mapping (LTM and GTM) for dynamic range conversion. Conventional data collection, calculation, and application of 3A and other general statistics is optimal if a frame buffer is used to store the collected data, along with the image data, used to generate the statistics and stored in memory such as RAM (DRAM as an example). This provides time to analyze frame FN, for example, for LTM parameters, and still affect the display of the same frame FN by applying the LTM parameters to the image data of frame N before displaying the frame FN. However, if a frame buffer is unavailable, for example, intentionally to reduce power consumption, features that use statistics may not operate at optimal or high-quality levels. As mentioned, this type of on-the-fly operation will omit the generation of statistics when there is insufficient time to generate those statistics in a timely manner to improve the current frame being displayed. The image processing system then skips the application of features that improve image quality but require the statistics to operate. This applies to many features that use other general statistics or even use 3A statistics for other purposes. In the best case, statistics calculations that are ready at the end of FN can be applied to frame FN+2. This latency can cause artifacts or performance degradation, especially in frames with changing light conditions or with motion.

[0038] To address these issues, statistics for a single frame representing statistics are obtained from two frames that effectively provide statistics in an earlier frame, or are obtained earlier during frame capture (or ISP processing) than in conventional systems. Specifically, the methods and systems disclosed herein calculate 3A and other statistics based on the application time required to generate and use the statistics to provide new parameters using a multi-frame statistics collection window for the application rather than a purely frame-based application system. The method makes 3A and other statistics available for analysis earlier and achieves smaller frame delays for "on the fly" mode. This is achieved by relying on an electronic rolling shutter used with, for example, many complementary metal oxide semiconductor (CMOS) based shutters, and performing continuous exposure over almost the entire video sequence and from frame to frame. Specifically, a global shutter exposes all rows in a frame simultaneously and then reads them row by row, thereby creating very clear and distinct breaks in data in time between frames. In contrast, in a rolling shutter, all rows of a frame are exposed starting at different points in time and one after the other (as discussed below). Figures 4A-4B ). With this arrangement, the exposure of the later rows of the first frame (frame FN-1) may be completed close in time to, or even overlap with, the exposure of the earlier rows of the next frame (FN). Thus, the separation of data from frame to frame by using a rolling shutter is significantly less significant in time than with a global shutter.

[0039] This method exploits this concept by combining statistics from two consecutive frames, such as a later row (or integral or pixel row) of frame FN-1 and an earlier row of frame FN. This results in earlier completion of the 3A and other statistical information analysis of at least some of the data for frame FN, and despite using data from frame FN-1, it is based on a complete single frame of statistics. This, in turn, allows the resulting parameters to be applied one frame earlier than conventional processing, although the actual frame that receives the influence or effect of the new parameters depends on the speed and capacity of the image signal processor (ISP), the camera registers that the algorithm affects to implement 3A or other features, the 3A implementation mechanism (e.g., lens motor), the amount of memory transactions and the system memory transaction bandwidth, and / or the computation time of the algorithm, as explained in detail below.

[0040] For example, reference Figure 2, the video processing sequence 200 has captured frames of a video sequence including image data input from frame FN-1 (202) to frame FN+2 (208) and is shown in an on-the-fly mode. On-the-fly (OTF) processing mode refers to processing that is completed during the capture of the frame, as long as the frame enters the ISP without entering off-chip or on-chip memory (such as RAM) for a full or partial frame delay. In this example, the system collects data to form a complete single frame worth of statistics by collecting data from both the first frame FN-1 (202) and the second or current (or next) frame FN (204) during a 3Astat calc. period 210 that overlaps the input and processing time of both frames FN-1 and FN. The traditional ISP idle period and vblank period between the two frames may or may not be used to generate statistics. The process shown in the sequence 200 uses an on-the-fly (OTF) mode, where statistical information data is collected from two consecutive frames and used to calculate new capture or processing parameters before the next frame. As shown, the statistical information can be processed during the 3A library processing period 212, when the new parameters are generated. The new AF parameters can then be applied by the focus module 214 to at least the portion of the same current frame FN that was also used to initially generate the later portion of the statistics. Otherwise, the statistics can be applied during the sensor / imaging pipeline period 216 and applied to frame FN+1, which is only one frame prior to frame FN. By generating the statistical information from image data of the same frame to which the new parameters are applied, or from image data of a frame just prior to the frame to which the new parameters are applied, image quality can be significantly improved. As explained below, this arrangement can also reduce memory, bandwidth, and associated power consumption.

[0041] refer to Figure 3 , an example process 300 for camera control and image processing with multi-frame-based windowing for image data statistics described herein is arranged in accordance with at least some implementations of the present disclosure. In the illustrated implementation, process 300 may include one or more operations, functions, or actions illustrated by one or more of even-numbered operations 302 through 306. As a non-limiting example, process 300 will be described herein with reference to any of the exemplary image processing systems described herein and related thereto.

[0042] Process 300 may include "receiving captured image data of frames of a video sequence" 302. In one example, a camera or image sensor may provide image data in the form of captured frames and the image data may be placed in a bitstream input to the system and provided to one or more processors such as an image signal processor (ISP). This operation may include any pre-processing of the image data to prepare the data for use in generating statistical information. Details are explained below.

[0043] Process 300 may also include "generating a representation of a single frame of image data statistics, including providing an option to use image data from one, two, or more frames to generate the statistics" 304. Specifically, image data may be obtained and used to calculate statistics collected within a multi-frame statistics collection window that temporally extends across two or more frames and overlaps the period of processing image data for at least two frames (and, in one form, two consecutive frames). The obtained image data (such as chrominance and / or luminance pixel values) represents pixels forming a complete single frame, and spatially represents a single frame, as an example. For example, this may include image data for pixels in the lower (or bottom half) rows of the first or previous frame FN-1, and image data for pixels in the upper (or top half) rows of the current frame FN. The window may begin and end depending on a cutoff time affecting a particular frame, such as frame FN or FN+1. By way of yet another example, a particular type of statistics may be specifically used for an identified region of interest (ROI) across the frame. In this case, the start and end of the window can be set to capture image data of the ROI. Thus, by these forms, the duration of the window can depend at least in part on the type of statistics being generated, the location of the ROI (when the ROI is being targeted) and / or how much time is required to generate the statistics. Details are provided below. By one form, the window is set so that a single-frame representation of the space includes image data of exactly the same number and location of pixels as would be obtained from a single frame. By one form, each of the two frames contributes half or nearly half (such as within a row) of the image data for the representation of the single frame of statistics. By an alternative approach, the window includes generating statistics using data that comes more from row lines of pixels from one of the two contributing frames than from the other of the two contributing frames. By an alternative form, the window has the same fixed start and end positions on each first and second frame, respectively, but by another alternative, the start and / or end positions of the window can change frame by frame, depending at least in part on when the window has enough image data to start generating statistics and when it does not overlap with the previous window. By yet another alternative, the frame representation may not be strictly spatial (or may be only global spatial rather than local spatial). In other words, for example, the image data used within a multi-frame window may not be continuous row-by-row image data. In this case, the image data of a single frame used to represent the statistics is located only between the spatial start and end of the window, but otherwise may be in any pattern, form, or order, including random selection, as long as the statistical information values for a single frame are being represented.

[0044] In some cases, statistics from one or less frames of a multi-frame window are used instead of more frames (and even if a multi-frame window is being used) because some criteria have not yet or cannot be met, such as frame deadlines, so that the multi-frame window cannot provide any benefit, or in cases where the parameters between frames have changed significantly (such as with a scene break, for example), and the break itself affects the statistical calculations between frames. In this case, the system can revert to traditional single-frame-based statistics collection. Other criteria may exist, such as speed (bitrate) or quality-based thresholds. In other forms, when the captured parameters change between frames (e.g., from FN to FN+1), the calculated statistics can be used to form a scaling calculation to match the statistics of one frame with the statistics of another frame, as if the statistics were performed on the same frame with the same parameters. In this case, effectively, statistics from a single frame (from within the representation of the single frame of statistics from the multi-frame window) are applied to the data of another frame, thereby using a single frame or less of statistics from the multi-frame window.

[0045] By yet another alternative, a multi-frame window may include statistical information values for more than one frame to form a representation. In this approach, the algorithm may be able to use the bottom of FN and all of FN+1, and process the differences in statistics of overlapping pixel positions or regions (such as, for example, pixel rows) or other relevant regions of the frame, if the statistics vary due to, for example, lighting conditions or motion and further adjustment of the newly calculated parameters is required, and result in modified parameters that are then applied and can reduce or eliminate the effects of changes in lighting or motion. This can be used with LTM or AEC calculations, to name a few. In this case, statistical information values for 11 / 2 frames are being used. It will then be understood that reference to one, two or more frames refers to up to one, two or more frames of statistical information data actually being used from the statistical information data collected by the multi-frame window. The derivation of statistical information can effectively form a representation of the statistical information values of a single frame, which is treated as a single frame of statistics to calculate the parameters.

[0046] Additionally, the types of statistics generated during a window may be statistics used to update camera parameters for capturing subsequent images, such as 3A statistics (AWB, AEC, and / or AF), or statistics used to modify pixel data to improve the displayed image, such as with digital video stabilization or dynamic range conversion statistics (using LTM and / or GTM as described above), to name a few. Any of the examples of statistics mentioned above may be used, or any statistics generated on a pixel, row, block, or other basis that may be generated in units that are smaller than an entire actual frame, or that are otherwise not limited to a single identical capture of an image. Total statistics (such as average intensity or color per frame) may or may not be generated during the window, but instead may be generated after the window.

[0047] Process 300 may continue with “allowing access to a representation of a single frame of image data statistics to set image capture parameters for capturing the next available frame of a video sequence or to modify image data for processing the next available frame of a video sequence, or both” 306. In other words, the collected image data representing the single frame may be used to generate 3A statistics for autofocus, automatic exposure control, and / or auto white balance. Otherwise, the collected image data representing the single frame may be used to generate general statistics that are used to perform other non-3A functions. This may include functions such as digital video stabilization, other methods of local or global tone mapping or low to high dynamic color conversion, or other functions, to name a few possible functions. Many functions using a single frame of image data statistics should be able to be performed using the methods described herein.

[0048] In one form, the generated statistics are stored with the image data in a memory such as RAM or DRAM so that the ISP can retrieve the statistics from the memory to use the statistics to calculate new 3A or other image capture parameters, or can use the statistics to modify the image data to display a frame. The next available frame refers to a frame that has a deadline, and when the statistics are generated in time to meet the frame deadline to affect the capture or display of that next available frame. Additionally, this may refer to features that affect the entire frame or only a portion of the frame. In this form, the second or current frame (or frame FN ( Figure 2 )) and small parts of the current frame FN may be updated by other 3A parameters or display modifications, but are not applied until the next available frame FN+1 (which is one frame after the current frame FN).

[0049] This application may only be applied to a portion of frame FN+1. While this may be a significant improvement over statistics collection that is limited to a frame-by-frame basis, where only frames subsequent to FN+2 or FN+3 can be updated, as mentioned, this can still be improved by an on-the-fly (OTF) mode. As described above, an OTF mode may be used where instead of placing the image data (and the statistics when stored in RAM) in RAM, the image data is used as generated by saving the image data in a more local memory, and as an example, saving the image data in the ISP registers, and immediately using the statistics of the image data to calculate new parameters or modify the image data in order to reduce delays caused by memory transactions. As an example, image data (and optionally statistics) generated from the image data of both frames contributing to the window are saved in the ISP registers for the full OTF mode. This enables new parameters to be applied earlier to the same (or second) current frame contributing to the window (e.g., FN in sequence 200), and can include both AEC and AF parameters as well as the time to modify at least a portion of the image data for AWB and other features for at least a portion of the current frame FN and for the full frame FN+1. A partial OTF mode can also be provided, in which the image data (and optionally statistics generated therefrom) for one (but not both) of the frames contributing to the window are stored in ISP registers rather than in RAM. This is faster than non-OTF mode, allowing for more current frame updates, but still slower than full OTF mode, as explained below.

[0050] It will be understood that a representation of a single frame of image data statistics may also be referred to as image data statistics values for a single frame, which is treated as if it had been generated from a single frame, even if actually formed from image data of multiple frames. Furthermore, a representation of a single frame of image data statistics may be used alone or in conjunction with other representations of a single frame of image data statistics or one or more actual single frames of image data statistics, and need not necessarily be used alone. For example, statistics representing a single frame may be used together with other statistics of other actual frames to determine an average intensity of image data values over multiple frames.

[0051] Reference Figures 4A-4B, an image processing system 400 is shown in a timeline diagram with time running from left to right, and includes at least a camera or image sensor 402, a camera data stream protocol input system 404 (such as a Mobile Industry Processor Interface (MIPI) or other input protocol), and the operation of one or more image signal processors (ISPs) or other processors 406. One or more of the ISPs or processors 406 can operate an automatic statistics calculation unit (or statistics unit) 408, which uses image data from multiple frames and generates a representation of a single frame of 3A and other statistics by using a multi-frame statistics collection window 453, as described in detail below. The statistics unit 408 can also generate some different statistics on a frame-by-frame basis and deliver the 3A and other statistics to other components. An automatic or 3A control logic component or unit (or just automatic control) 412 provides the 3A and other statistics for use by the 3A automatic adjustment control. The system 400 or in one form an automatic or 3A control unit 412 may include the operation of or be communicatively coupled to the following components: an auto focus (AF) component 414, an automatic exposure control (AEC) component 416, and an automatic white balance (AWB) component 418, any combination of these components, etc. These are provided here as examples, and it will be understood that other applications using these or other general statistics described herein may also be illustrated here, such as applications using the LTM and GTM described above, digital video stabilization, etc.

[0052] Now in more detail, the illustrated image sensor 402 can be controlled to operate a rolling shutter or electronic focal plane shutter process, in which pixels are reset for photocharge in a row-by-row manner and read out in a row-sequential manner for the entire frame. In this type of system, the exposure times from row to row can overlap and do not need to be performed simultaneously. The time between the row reset (rt) 419 and the row readout (rd) time 420 is the integration time or exposure time (exposure) 422, which is the time for accumulating photocharge in one or more rows of pixels on a frame (or picture or image) or even just a portion of the row. The length of the integration 422 can vary depending on how many rows or pixels the integration covers. The row readout measures the amount of photocharge in the row and the amount of photocharge since the last reset of the integration time was started.

[0053] Reset 419 is performed one row at a time down the frame, row by row, so that each diagonal column 424 of exposure or integration row 422 corresponds to frame 426 or 427. Frame 426, and each frame, is displayed using a number of exposure or integration rows 422a through 422n, typically used for video display screens. Because reset 419 is temporally offset, when exposure times 422 are the same or similar throughout a single frame, the diagonal columns are formed with uniform steps. Regardless of the length of integration 422 and the position of reset 419, readout 420 is timed so that readout is performed one row at a time down frame 426, forming the diagonal lines of readout 420 with uniform or nearly uniform steps as shown. Here, the exposure times or integrations for two frames, FN-1 426 and FN 427, are shown. The time period used to capture the brightness of a frame (FN-1 exposure) is measured from readout to readout and overlaps with the FN exposure of the next frame. However, it will be understood that the length of the integral 422 may vary for a frame, depending on the number or portion of rows covered by the integral. In this case, the positions of the resets may not form a perfect diagonal line and may be more staggered, while the timing of the readout is maintained so that the readout can be performed sequentially and one at a time as shown. In addition, the length of the exposure time may be limited by the frame length so that the exposure time cannot be longer than the frame length. This is marked by a vertical (frame) blanking (vblank 441), which will be placed at the bottom of the diagonal column 426 (shown on the input raw data bit stream 432) to divide the frame. The exposure between frames is almost continuous, interrupted only by the vblank period, which is typically less than 5% of the frame time. However, if necessary, the frame length can be increased by making the frame blanking (vertical blanking) longer.

[0054] Readout 420 can be separated from the next row reset 419 in the same row by a horizontal blanking space or row blanking space 428 (shown only on the first row 422a and shown as 440 on the bitstream 432) and an adjustment or hold time block 430 (also shown only on the first row 422a for simplicity). Here, instead of a horizontal blanking space placed at the end of each pixel row in the frame as for a global shutter, a horizontal blanking space can be provided at the end of each integration row (which may or may not be aligned with the end of each pixel row in the frame) for a rolling shutter. The area actually sensed in the total row length (horizontally) and the number of rows in the frame (vertically) may be larger than the visible space on the device or the space to be analyzed. Therefore, horizontal blanking or row blanking is the scan time covering the pixel positions along the row and is an invisible non-transmission period. The same is true for vertical blanking, described below. The blanking space is used for synchronization. Horizontal blanking or row blanking can be used to provide components (such as an ISP) with sufficient time to process the incoming data. For example, if the row blanking is too short compared to the pixel clock or the rate at which the sensor is sending data, the ISP process may fail (such as with an input buffer overflow). Vertical blanking or frame blanking is used to control the frame rate, where the longer the vertical blanking, the slower the frame rate, the longer the maximum possible exposure time, and the more time to perform calculations before starting to read out the next frame. A hold time block 430 can also be used after the horizontal blanking 428 and before the next reset 420 for the same row to limit or prevent any overlap of the readout periods.

[0055] For example, by one approach, input data or a bitstream 432 having data from a sensor(s) 402 may have a start of field (SOF) 434, a main transmission portion Tx 436 of raw image data for multiple overlapping rows in the integration columns 426 associated with a single frame 426. In the illustrated form, the data for the readout 420 of all integration rows 422a-422n is placed in the main transmission portion Tx 436. The main transmission portion 436 ends at an end of field (EOF) 438, followed by a horizontal blanking time 440, and then a longer vertical blanking period 441 may be provided when the end of one frame is reached and data for the next frame begins. For example, the horizontal blanking time 440 in the bitstream 432 increases the time it takes to read out a full frame to allow the receiving side (such as the ISP 406) to clear its input buffer before the next row is sent. By another alternative, horizontal blanking is not used, where the horizontal blanking is zero or non-zero, and dummy pixels may be transmitted or not transmitted during the row blanking period.

[0056] In addition to the chrominance and luminance data in the raw image data, other data can be stored in memory, such as exposure data including exposure time and analog and digital gain. This data is then provided to the ISP as needed to generate statistics and / or new 3A or other parameters.

[0057] Reference Figure 5 , most of the remaining description of the operation of the system 400 may be described in conjunction with the process 500 of camera control and image processing with multi-frame based windowing for image data statistics. The process 500 described herein is arranged in accordance with at least some implementations of the present disclosure. In the illustrated implementation, the process 500 may include one or more operations, functions, or actions as illustrated by one or more of the even-numbered operations 502 through 544. As a non-limiting example, reference will be made herein to Figures 4A-4B and Figure 10 An example image processing system and Figure 6-8 Process 500 is described with reference to the related processes in operation shown in FIG.

[0058] Process 500 may include "receiving image data for frame FN-1 from a camera sensor" 502. Frame FN-1 refers to the first (or earlier or previous) frame of two frames that contribute image data to form a single-frame representation of image data statistics as used herein. The second or current frame FN is described below. In one form, the two frames are sequential, and only two frames are used. In another form, as described above, more than two frames of image data may be used, or even less than one frame of image data may be used.

[0059] In one example, once the ISP 408 receives the raw image data, the process 500 may include applying pre-processing to the raw data 504. This may include noise reduction, pixel linearization, and shading compensation. It may also include resolution reduction, Bayer demosaicing, and / or vignette removal, among others.

[0060] These operations are illustrated on timeline 400 by the operation of ISP / processing unit 406 at time period 444, which retrieves and analyzes raw image data, including providing pre-processing, accompanied by an idle period 442 between the reception of adjacent frames and corresponding to horizontal blanking period 440 (corresponding to horizontal blanking (hblank) period 428) and vertical blanking period 441. ISP 406 may have ISP registers 407 to store data or context being processed. The resulting image data is provided at image output time period 446, where the image data is stored in a memory such as DRAM, cache, or other memory as needed. It will be understood that image analysis period 444 and image output period 446 can extend well beyond horizontal blanking period 440 and into vertical blanking period 441 to complete image data analysis and generation.

[0061] Turning now to the operation of the statistics unit 408, some general statistics can optionally still be generated on a frame-by-frame basis (in addition to using a multi-frame window), as shown by the frame-based statistics output window 448 divided by the period 450 between frames. Thus, once pre-processing has been performed, general image statistics 506 can be calculated. This can include luma / chroma value combinations such as average values, luma / chroma high frequency and texture content, motion content from frame to frame, any other color content values, picture statistics related to deblocking control (e.g., information controlling deblocking / non-deblocking), RGBS grids, filter response grids, and RGB histograms, to name a few. Whether used for 3A features or other non-3A features, several of these statistics can be generated by alternatively using a multi-frame statistics collection window, as discussed below. These frame-by-frame statistics can be provided on a macroblock or coding unit (CU) basis (e.g., per 16×16 or 8×8 or other sized pixel block), or can be provided on a per-pixel, per-frame, or other unit basis as needed depending on compatibility parameters for certain standard coding schemes (such as H.264 / Advanced Video Coding (AVC) or High Efficiency Video Coding (HEVC), JPEG, or other image processing and coding standards). These standards can be used at the end of post-processing when YUV data is available and the ISP has reconstructed the image. These general statistics can be stored in double data rate (DDR) DRAM or other sufficient memory and can then be provided for further analysis during the time period image output 446 and frame-based statistics output period 448.

[0062] The process 500 may also include "setting the statistics collection window position to overlap frames FN-1 and FN" 508. As an example, the multi-frame statistics collection window 453 extends over the divided time period 451 between both frames FN-1 (426) and FN (427) and the frame-statistics time periods labeled 3AstatsClct-1 (454a) and (454b). The window 453 extending over two frames refers to extending over the ISP processing period of the two frames, but may also refer to extending over the exposure time corresponding to the frames or over the exposure time of the frames. In this example, the exposure time of FN-1 ends at the beginning of the time window 453, but the window still extends over both the ISP processing period 444 of the two frames FN-1 and FN. The next statistics collection time period 3AstatsClct.-2 (455) can be used to form the next multi-frame statistics collection window (not shown) for the lower portion of the current frame FN and the upper portion of the next frame FN+1. Image data is acquired during two frame-statistics time periods (also referred to as first and second window portions) 454a and 454b within window 453, and 3A or other statistical information is calculated using the image data. During divided time 451, statistics unit 408 can continue to calculate statistics even if image data is no longer available from the previous frame or first frame FN-1. This allows for substantially or essentially continuous statistical information generation throughout multi-frame statistics collection window 453, during vertical blanking 441, and during image data processing idle time 442 (or effectively eliminates image data processing idle time 442). Idle time 442 can still be reserved by the ISP to apply new 3A parameters and implement 3A modifications. Image data retrieval and statistics generation then resumes at the second or current frame FN in the second frame-statistics time period (or second window portion) 454b.

[0063] By one approach, the end of window 453, and thus the start of window 453, can be determined by a deadline for applying the new parameters to a certain next frame, whether that next frame is the same second frame FN that forms the window, the following frame FN+1, or any subsequent target frame. The start and end of the window depends on multiple factors. For example, the new parameters based on the statistics should be sent before the next frame is captured or processed (depending on the algorithm and parameters). Furthermore, to set the start time for statistics collection across two frames, the system should consider the propagation time of the new parameters (e.g., I2C command), the worst-case calculation time by the CPU (or ISP), and the frame time. The total time can then be converted to a frame time (position) at which statistics collection begins. If any of these deadlines are not met for any reason, the new parameters can be applied to the next frame as a fallback. The deadline is calculated from the latest time the parameters must be sent, taking into account the time it takes to send the new parameters (via I2C, register write, or other means), counting back to how much time the CPU uses to calculate the new parameters (depending on the algorithm). From this, the time for frame statistics collection can be calculated and the start time can be set. By way of a specific example of AEC, a multi-frame statistics collection window can be set so that the deadline for ISP 3A or other parameter updates (such as camera sensor register write deadlines) can be met. This can refer to meeting the deadline for frame FN+1, but can even refer to the deadline for at least part of frame FN, depending on whether an on-the-fly mode is being used and the type of on-the-fly mode, as explained in detail below. Also for AEC, if an algorithm using statistics determines the variation in brightness differences from frame to frame, the exposure differences between frames FN-1 and the FN portion, and determined by analyzing the statistics from those portions, should be normalized. Normalization can include setting a lower overall exposure gain to match a higher overall exposure. However, due to the temporal stability traditionally applied in exposure control, typically successive frames will not have significantly different exposures.

[0064] Thus, in one form, the length of the window is set as mentioned above by determining the length of time that should be required to collect statistics, calculate new parameters, and propagate all new parameters to meet the frame deadline.

[0065] By alternative form, the start and end of the window can be fixed for each pair of frames. In one case, each of two consecutive frames FN-1 and FN provides image data and statistics for exactly half of the pixel rows of a single frame, with the first frame (FN-1) providing image data for the lower half of the frame and the second frame (FN) providing image data and statistics for the upper half of the frame. When an odd number of pixel rows (or integration rows) are provided in a frame, one of the two consecutive frames can provide data and statistics for the center row, or the center row can be provided by two consecutive frames by splitting the center frame along its length. By way of other example, particularly when the statistics include the calculation of local averages or other combined pixel values, there can be some overlap so that the image data value of a single pixel position or block position can be used as a spatially adjacent contributor to the average of pixel values for, for example, two target pixel positions, each of which contributes to a different half of the statistics generated in window 453.

[0066] In other forms, one of the two frames contributing to a multi-frame statistics collection window may contribute more points or pixel rows of image data than the other of the two frames. In this form, the window can begin when a sufficient amount of image data has been generated and available to form certain statistics. The window length can then be set to process the statistics values for a single frame, so that the window extends into the next frame or two frames to complete this operation. The next window then begins where the previous window ended, so that, as an example, there is no overlap in the use of image data to form the statistics. In this option, the window length is still set based on the image data values of the retrieved frame and the time it should take to generate statistics based on the image data values of that frame. In this form, the window has a clear predetermined length for generating statistical values for the frame. In one form, this may be some average window length for the type of statistics being generated, or it may be based on the type of statistics that takes the longest amount of time to calculate.

[0067] Note that statistics collection for each statistic and algorithm is typically done per ROI (Region of Interest). Thus, multi-frame statistics collection is done to spatially accumulate all necessary statistics so that the entire frame view will be captured. However, this can be affected by the ROI being smaller than the full frame, and thus can affect the start and end points in each frame where statistics are collected or used to correspond to the location of the ROI across one or more frames.

[0068] By yet another option, instead of limiting the window length to accommodate a predetermined definition of the statistical information values for a frame, the window for subsequent frames can be set based on the actual time it takes to generate the statistical information for the previous frame. Thus, once the statistics are completed, the next window is started for the current frame and subsequent frames. In this case, there are no fixed positions for the start and end of the window relative to the retrieval of image data for certain pixel rows of the frame.

[0069] Process 500 may then include "Generating Statistics from a Portion of Frame FN-1 Image Data" 510. Here, the ISP has already pre-processed the image data, the image data is being retrieved, and is being used to generate statistics. As an example, this occurs during the multi-frame statistics collection window 453, and specifically during the frame-statistics time period 454a for the first or previous frame FN-1. This may include generating statistics based on image data generally from the second half (or lower half, bottom half, or lower integral or pixel row) of frame FN-1, or alternatively, other portions of the frame.

[0070] As described above, examples of the types of statistical information content may include 3A statistical information and non-3A statistical information. During this window 454a, as image data is acquired and once sufficient image data is obtained, statistical data is being collected or generated. This can be performed on a pixel basis, a block basis, an integrated row basis, or other units as needed. Details of the types of 3A statistical information for autofocus (AF), automatic exposure control (AEC), and automatic white balance (AWB) have been described above.

[0071] As also mentioned above, general statistics that can be used with the multi-frame statistics collection window 453 during the window 454a for the first frame FN-1 can include local or global tone mapping (e.g., LTM or GTM) for dynamic range conversion and / or other applications, or motion content from frame to frame and outer boundary pixel tracking for digital image stabilization, as well as other general statistics that can be generated for 3A operation or other functions, and can include luma and / or chroma averages, luma and / or chroma high frequency and texture content, any other color content value combination, picture statistics for deblocking control (e.g., information for controlling deblocking and / or non-deblocking), RGBS grids, filter response grids, and RGB histograms, to name a few. Other general statistics can also be used with the multi-frame statistics window described herein, but the description herein is not limited to any one type of statistics.

[0072] refer to Figure 6, process 500 may include "storing or saving image data and / or statistics" 511, and in particular, either storing the image data (and optionally also the statistics) in an external or "off-board" or "off-chip" memory (such as RAM (or RAM on the chip)), or saving the statistics in ISP registers or other close or local memory (at least more local than RAM) for immediate processing of 3A parameters with the statistics. Specifically, the statistics can be stored in, for example, a buffer or 3A bank 410 (Figure 4), and the buffer or 3A bank 410 is formed by "off-board" memory such as RAM, specifically DRAM or double data rate (DDR) DRAM. System 600 illustrates this process, where a sensor 602 captures an image of a video sequence, and then the input (or initial) processing 604 on frame FN-1 includes, for example, the generation of 3A statistics during an early or first window 454a. The statistics data is then stored in a memory such as DRAM 606 along with the image data. This is repeated in the second window 454b for frame FN at the processing stage 1 unit 608, where the previously stored statistics and image from the first window 454a may need to be retrieved when those statistics are the basis (level 1 statistics) for further statistics to be calculated in the second window 454b (level 2 statistics explained below), and then the remainder of the 3A statistics are generated and stored in DRAM 606 (this operation of generating statistics on frame FN using the second window 454b is covered by operations 512 to 520 described below in process 500). As mentioned above, the end of the second window 454b, and therefore the end of the multi-frame statistics collection window 453, can be set at the application deadline to provide sufficient time to provide new parameters to affect frame FN, FN+1, or other subsequent target frames. The 3A statistics are then retrieved from the DRAM and provided to the processing stage 2 unit 610 to generate 3A parameters or otherwise use the statistics. This is also described in detail in process 500 below. It will be appreciated that this process 600 is also applicable to non-Triple A statistical information.

[0073] Although statistics can be prepared for analysis earlier by using a multi-frame statistics collection window, it will be understood that each transfer of image data (and statistics) to and from DRAM may also incur latency and use memory bandwidth, and therefore require more memory capacity and power consumption than if such transactions and memory usage could be eliminated or reduced. Accordingly, a partial or full on-the-fly mode can be used that uses ISP registers to hold the generated image data while the ISP uses the statistics to run 3A operations and generate new 3A parameters. In this case, the image data (and statistics) are not downloaded to DRAM or other off-chip memory or external memory. This allows for smaller memory and lower power consumption, thereby improving the computer or device performing the processing. It also reduces latency so that the effects of new parameters based on statistics can occur earlier in the frame than if the image data (and statistics) were stored in RAM. The result can be an improvement in the quality of the captured and displayed images, as well as an improvement in the quality of the viewing experience and a reduction in the required memory capacity and power consumption.

[0074] refer to Figure 7As an example, a portion of the ongoing processing is described on system 700. The units and operations are largely the same as those of system 600, except that instead of the image data (and statistics) being stored in DRAM, here, after input processing 704 generates the statistics in the first window 454a of the multi-frame statistics collection window 453, the image data and statistics are saved in ISP registers (such as register 407 (Figure 4)) to be used immediately when generated and to determine whether new parameters should be generated and then generate those new parameters. The statistics data can be considered part of the 3A library, regardless of which memory the statistics are stored in once generated. The image data (and statistics) are saved in registers rather than being placed in RAM. In one form, this refers to all statistics for a particular use or type of statistics, such as all statistics related to the average color component of window 454a or all statistics for a set of statistics for a particular application, such as all statistics for autofocus. Other variations can be used, such as saving less than all of such statistics in registers rather than using RAM for storage. Thus, system 700 illustrates the next immediate operation of generating statistics in processing stage 1 708, wherein the remainder of the statistics, such as 3A or other general statistics, are generated during the second window 454b, while simultaneously retrieving image data for the upper or earlier portion (or integral or pixel rows, and at least the top pixel rows) of frame FN. Since this operation is a partially on-the-fly operation, the image data and statistics for the second window 454b of frame FN remain in DRAM 706, while the image data and statistics for the first window 454a of frame FN-1 are stored in ISP registers 407. This operation can be used when insufficient capacity exists in the ISP registers to store the image data (and statistics) for both the first and second windows. The reverse process can also be performed, wherein the image data and statistics from the first window 454a and the input processing unit 704 for frame FN-1 are stored in DRAM, while the image data and statistics from the second window 454b of frame FN and the processing stage 1 unit 708 are stored in the ISP registers.

[0075] With the partially operational system 700, since the image data and statistics for the first window 454a are stored in the ISP registers, the system still has sufficient time to apply new parameters based on the statistics earlier. For example, the processing stage 2 unit 710 can use the generated statistics from both window 454a from frame FN-1 and window 454b from frame FN to calculate new parameters and apply them to frame FN. This provides more time to analyze the statistics and apply the new parameters in time for the cutoff to affect frame FN+1 or even towards the end of frame FN itself. Similarly, for applications that take longer to implement, this can be one frame earlier than in conventional systems, affecting frame FN+1 first rather than FN+2 (or FN+2 rather than FN+3). In other words, it can often affect the quality of image capture and display one frame earlier. This can make a significant difference in 3A implementations and other common statistics implementations, such as motion artifacts in LTM, to name one example.

[0076] It will be understood that the CPU and / or other processor(s) may perform such processing instead of or in addition to the ISP. It will also be appreciated that the registers may be registers of more than one ISP, and / or registers of another type of processor (CPU, etc.). The ISP may be part of a system on a chip (SoC) or other architecture, many of which are mentioned elsewhere herein. In addition to registers, the image data and statistics may be stored on other memory that is at least more local than off-chip RAM, DRAM, or other types of RAM, whether the ISP or other processor is performing the statistics generation or analysis. This may include on-chip memory that is located on the same chip or board as the processor.

[0077] refer to Figure 8 , a fully operational process is described on system 800. The units and operations are primarily the same as those of systems 600 and 700, except that instead of saving the image data and statistical information generated from only one of the statistics processing units (804 or 808) in the ISP registers, the image data and statistical information from both processing units 804 and 808, and in turn from both statistics windows 454a and 454b from both frame FN-1 and frame FN, respectively, are saved in local memory (such as the ISP registers) while the statistics are analyzed to generate new parameters, rather than storing the image data and statistical information in RAM.

[0078] For full on-the-fly processing, the statistics generation occurring for frame FN-1 and frame FN can end even earlier than for partial on-the-fly processing, such that the statistics collection ends at the middle (or other configurable point) of frame FN, where all the spatial information exists and allows the algorithm to process it, and the new parameters are released, if needed, for an even earlier application deadline that occurs during the retrieval and processing of frame FN. In this case, when possible, the new parameters may be applied to frame FN even earlier, thereby providing improvements to a larger portion of frame FN than for partial OTF, and the application may occur to affect the entire frame FN+1, depending on the application.

[0079] However, in one form, partial or full OTF may have a reduced feature set (e.g., no image-based DVS, no or limited geometric distortion correction, reduced resolution, etc.) so that there is more capacity at the ISP registers or memory for holding image data and statistics and better avoiding further delays.

[0080] Continuing now with process 500, process 500 may include "receiving raw image data for the next frame FN from the camera sensor" 512, and now performing retrieval of the image data for frame FN, and process 500 may include "applying pre-processing of the raw data" 514 to prepare the data for statistical information calculation, as in operations 502 and 504 described above for frame FN-1.

[0081] Process 500 may then optionally include "Compute General Statistics About Frame-Based Data" 516, also as described above for frame FN-1 (operation 506), where other frame-to-frame statistics may still be generated.

[0082] The process 500 may include "generating statistics from a portion of the next frame FN" 518, where the ISP has analyzed the image data, the image data is being retrieved, and is being used to generate statistics, as mentioned for frame FN-1. This is now occurring during the latter portion of the multi-frame statistics collection window 453 (specifically, the frame statistics time period 454b of the second or next frame FN). This may include generating statistics based on image data from the first half (or upper half, top half, or upper pixel row) of the frame FN, or alternatively, other portions of the frame including the top or uppermost pixel row of the frame FN.

[0083] It will also be understood that the processing during the second window 454b (or during the second processing stage 1 unit 608, 708, or 808) can include multiple levels (or operations) of calculating statistics. For example, the first-level statistics can refer to a color map or color histogram of the image data, or the gradient of the image data, where a single or low number of calculations are being performed. The second-level statistics can then use the first-level statistics for further calculations, such as the sum of gradients for certain units (such as pixel rows or blocks), or an average for the frame using those first-level statistics. The third level can use those second-level statistics, and so on, until the statistics are in some final form that can be used by applications such as 3A components or other general video quality applications. The second window can be used to perform second-level or higher-level statistical calculations on the first-level statistics calculated in the first window 454a. Of course, depending on the type of statistics, such as row-based gradient sums, some of the second-level or other levels of statistics can also be calculated during the first window 454a when sufficient data is available.

[0084] Finally, process 500 may include "calculating statistics using two portions from two frames" 520. In one form, when statistics for the entire frame (or a single value representing the entire frame) should be based on the frame's data values provided by using both window 454a and window 454 for frame FN-1 and frame FN, respectively, such values, as well as the average color or brightness for the entire frame, may be calculated during the second window 454b when all data is available from a particular level of statistics. Otherwise, such final frame-level statistics may be calculated after the second window 454b in a time period 456 (FIG. 4) for, for example, an ISP to perform such calculations. This time period 456 may also be used to calculate other 3A or general statistics. These further 3A statistics may or may not also be considered to be performed by the automatic control 412 (FIG. 4).

[0085] Process 500 may include "Store or Save Statistics" 521, and the operation is explained above to store the generated statistics from the second window or any statistics (such as the higher level or frame level statistics described) in DRAM, or to save the statistics in the processor or ISP registers for OTF mode as described above.

[0086] Once the statistics are calculated, the statistics (if not already delivered) can be delivered to the automation control 412 during time period T_3A_stat_delivery 457. The automation control or 3A control 412 can then use the statistics, whether stored in RAM or in ISP registers as described above, to determine whether new 3A parameters are needed, and then calculate the new parameters if needed. The application of the new parameters can then be applied. These operations using the statistics can generally be applied during a 3A implementation time period 458 (Figure 4). It will be understood that the implementation time period 458 can be much longer than shown, extending over multiple frames, if necessary. It will be understood that other general statistics applied for display modification can have similar timing.

[0087] As mentioned above, the automatic controls 412 can be considered to include an AEC component 414, an auto-exposure component 416, and an auto-white balance component 418, and can also be considered to include the automatic statistics calculation unit 408. The following is just one possible example of using the controls 412 to break down the time periods 458 for the various 3A components to perform 3A operations, and many variations are possible.

[0088] Process 500 may include "Run AF" 522. Specifically, once 3A statistics are available, 3A control and adjustments may also be performed, which may be performed one after another, or in parallel, in one case where each control feature may have one or more dedicated processors. In the former case, AF control 414 has AF logic or components 460 that receive 3A statistics during a time period (T_3A_stat_delivery) 457. Once the 3A statistics are received, the AF logic unit or component 460 runs and calculates focus adjustments during a time period T_AF_execution 466. In one form, a lens driver 468 with an I2C protocol then forms a command for a new focus setting 524 during a time period T_AF_I2C 470 following the AF execution period 466, and then sends 530 the new AF setting to the camera module or sensor 532 so that the lens hardware 474 on the camera can then move the lens during a time period T_AF_Lens 472. The lens hardware may or may not be considered part of the AF control 414 .

[0089] In one form, the AF control 414 can still use statistics from a frame whose integration time overlaps with the time T_AF_lens 472 used for lens movement. For that frame, here for example frame FN+1, the command can meet the deadline for frame FN+1 due to the use of the multi-frame window 453 and OTF mode as explained above. Otherwise, the system herein can be used with a fixed focus camera module without AF, allowing the methods herein to be applied to other types of statistics and algorithms, such as AE, DVS, LTM, etc.

[0090] Process 500 may then include "Run AEC" 526. With respect to AEC, when one or more processors are being shared between 3A controllers, a first latency period (T_3A_latency_1) may delay execution of auto-exposure control 416 until AF control 414 completes AF adjustment calculations 466. AF logic or component 462 of AF control 416 may execute exposure adjustment calculations during a time period (T_AE_execution) 476 to form a new or updated exposure setting for the camera sensor. Alternatively, in a system with dedicated components, auto-exposure execution 478 may occur while AF control 412 is executing AF adjustment 466. Sensor I2C component 480 may then provide 528 the new exposure setting and send 530 the new exposure setting to the sensor during a time period T_AE_I2C 482.

[0091] Note that in both the I2C transmissions for the AE and AF commands, an uncertainty word is declared to indicate that a potential delay may occur at that point due to time periods 470 and / or 482 taking longer than indicated. It should be noted that the time at which the additional digital gain is provided to the ISP for processing does not directly depend on the timing of the AEC, AF, and AWB operations 466, 476, 486, or the I2C transmissions 470, 482. Furthermore, if some other part of the system is competing for the same I2C or CPU resources, this may result in variations in the AF execution time (T_AF_execution) and the AE I2C execution time (T_AE_I2C), or any other part requiring those resources. This becomes even more important because it is beneficial to provide a flexible time envelope for the AEC operation or analysis for each frame, allowing for a more thorough analysis to be performed on each frame, or at least on certain frames that require more in-depth analysis.

[0092] The automatic exposure control (AEC) related to this article uses an algorithm to adjust exposure parameters to capture an image and provide adjustments to brightness parameters for display of the image, whether on a live preview screen of a digital camera or other recorded display, as well as providing storage or encoding of the image or video for later viewing. When the parameters are being updated at a sufficient rate, the brightness in the live preview screen or in the recorded video will appear to change smoothly from frame to frame without noticeable flicker or sudden changes in brightness, so that a smooth convergence of brightness (or apparent stable brightness) is obtained. This provides the image with a carefully controlled, high-quality appearance. In order to achieve such smooth and sufficiently fast updates of the exposure parameters, the updated parameters are provided to the camera sensor, for example, at a rate of at least about 30 Hz. Delays may occur when the processor performing the AEC calculations receives the latest exposure parameters from the camera sensor late.

[0093] A dedicated processor can be provided to run the 3A algorithm to ensure that no other processing will delay the AEC processing. Otherwise, a dedicated I2C controller can be added to ensure that no other traffic will delay the 3A traffic. Alternatively, the system can strictly enforce the processing time limit allocated to the AEC algorithm. However, this solution may still be insufficient and may require additional hardware anyway. Such limits are very difficult to achieve, and the resulting target brightness set by the AEC may not be optimal in all image capture situations, as not all necessary analysis can be guaranteed to occur within the given strict processing time limit for all frames.

[0094] Next, the camera module 532 may capture further images using the new settings and provide new raw image data 534 to restart the cycle. Figure 5 To restart the cycle or loop the process, frame FN is now frame FN-1 and the new frame is now the current FN frame (536) and the process loops back to operation 502 where raw image data is received for processing.

[0095] Regarding AWB, process 500 may continue with "Run AWB" 538, referring to running AWB control 418 after the second wait period (T_3A_latency_2) 484 from the receipt of 3A statistics. In this case, processing may not begin until the auto-exposure execution period 476 completes. AWB control then performs adjustment calculations to provide new white balance (WB) gains 539 during a period (T_AWBetc_execution) 486 following the second wait time. As mentioned above, AWB algorithms such as color correlation, gamut mapping, grayscale edge, and / or gray world AWB methods may be used to determine the white point and RGB gains. For example, with the gray world method, an average is calculated for all color components, and then the appropriate gain is applied to each color component to make the averages equal. This calculation results in initial or normal WB gains for the image. Normal WB gains may refer to three gains for the primary colors (e.g., R, G, B). This operation may also establish an initial white point. Alternatively, if a dedicated processor is provided, AWB operation may occur during period 488 while AF and AF control are performing their adjustment calculations.

[0096] ISP parameters used to process an image (e.g., for display, storage, and / or encoding) may be updated with data 190 or 192 including 3A adjustments or settings for the next available frame and during an idle period 442 of a frame not yet processed by the ISP during Tx time 436. Figures 4A-4B As shown in , as an example, 3A processing has been delayed and is too late to be processed during the illustrated idle 442 and, therefore, the deadline for frame FN, such that frames processed during the ISP processing period 436 immediately following the idle 442 will not receive their appropriate adjustments until frame FN+1. Alternatively, if 3A processing is for the next idle (not shown) following frame FN and can result in a deadline for frame FN+1, then updated data 192 can be received to update the next ISP Tx 436 period because the next idle has not yet occurred.

[0097] Thereafter, the process 500 may continue with applying 540 post-processing. Here, after the WB gain is applied to the image data, the image data may continue to the post-processing 540 of the image data. The post-processing may include CFA (color filter array) interpolation, color space conversion (e.g., such as raw RGB to unprocessed sRGB), gamma correction, RGB to YUV conversion, image sharpening, etc. The post-processing may be performed by a processor such as the ISP 408 mentioned herein or other processors for performing these processes through software and / or the hardware pipeline of the ISP.

[0098] Next, the frame or image is displayed for final or preview, such as storage for later use and / or encoding, or any combination thereof 542. The processed image can be displayed 542, whether as a viewfinder or preview on a digital camera or phone screen, or as a final display on another device, or can be stored for later viewing, and can be a video or a still image. Alternatively or additionally, the image data can be provided to an encoder for compression and transmission to another display or storage device. When recording or image capture is stopped, the camera can be turned off 544, otherwise the system can continue to capture new images.

[0099] refer to Figure 9 , process 900 illustrates basic operation of a sample image processing system 1000 with multi-frame window-based camera control and image processing for image data statistics in accordance with at least some implementations of the present disclosure. In more detail and in illustrative form, process 900 may include one or more operations, functions, or actions as illustrated in one or more of the even-numbered actions 902 through 922. As a non-limiting example, process 900 will be referred to herein. Figure 10 Specifically, the system 1000 includes a statistical information unit 1010 and an application 1015 that uses statistical information. The operation of the system may be performed as follows.

[0100] In one example, process 900 may include "Receive Image Data" 902, and as explained above, captured by integration using a rolling shutter, where the raw image data is substantially pre-processed for statistics generation and collection, as described above.

[0101] Process 900 may include "Set Statistics Window Start and End" 904, also as described above, to set the start and end of a multi-frame statistics collection window, during which a portion of the window is over a first frame (such as frame FN-1) and another portion of the window extends over a second frame (such as the next or second frame FN). The end of the window can be set so that statistics generation can be completed within a deadline that allows the use of the statistics to generate new parameters when permitted and apply those parameters so that a specific frame (such as frame FN, frame FN+1, or another subsequent target frame) is affected by the application. As mentioned, a number of different factors may affect the start and end of the window, including, for example, the statistics provided for a particular ROI.

[0102] Process 900 may include “Generating first level statistics using received image data” 906 and involves using image data to generate statistics that can be calculated as the image data is being retrieved (such as, for example, gradients) and that do not require data values for an entire frame to calculate during a window or require other statistics to calculate another level of statistics.

[0103] Process 900 may include "if needed, generating additional level statistics as first level statistics become available" 908, and thus performing second or further calculations, such as, for example, gradient sums in some units such as for integration rows or pixel rows or blocks.

[0104] Process 900 may include "generate statistical totals if needed" 910, where data values from the window and from the entire frame of two frames (FN-1 and FN) are required for calculation. As an example, this may be for the average color component or intensity (brightness) level of a frame (or indeed a virtual frame) formed using image data for the window over the two frames FN-1 and FN.

[0105] Process 900 may include "Providing access to statistical information" 912, and retrieving the statistical information from RAM (or DRAM) or, when OTF mode is being used, from a memory more local than RAM (such as an ISP register), which analyzes the statistical information as it is being generated, by way of example. The OTF mode may be a partial or full OTF mode as described above with systems 700 or 800, respectively.

[0106] Process 900 may then include operations such as determining whether new parameters are needed and, if so, generating those new parameters to "Update AF Parameters" 914, "Update AEC Parameters" 916, and / or "Modify Image Data to Correct AWB" 918, as described above.

[0107] When the multi-frame statistics collection window is used to collect other general statistics to provide parameters or modify display data, process 900 may include "modifying image data based at least in part on the statistics to provide other functionality" 920. This may include statistics for DVS or dynamic range conversion, such as LTM or GTM, to name a few example features, and as mentioned above.

[0108] Process 900 may include "Apply New Settings to Next Available Frame" 922, and as also described above, whether 3A or other features are being used. When the multi-frame statistics collection window overlaps frame FN-1 and frame FN, the settings may affect the capture and display of at least a portion of frame FN itself using AF, and possibly AEC and AWB when partial or full OTF mode is being used. Otherwise, at least a portion of frame FN+1 may be improved, and the entire frame FN+1 may be improved when OTF mode is being used.

[0109] Additionally, the system may be performed in response to instructions provided by one or more computer program products. Figure 1-9Any one or more of the operations of the processes in . Such a program product may include a signal-bearing medium that provides instructions that, when executed, for example, by a processor, may provide the functionality described herein. The computer program product can be provided in any form of one or more machine-readable media. Thus, for example, a processor comprising one or more processor cores may perform one or more of the operations of the example processes herein in response to program code and / or instructions or instruction sets delivered to the processor by one or more machine-readable media. In general, a machine-readable medium may deliver software in the form of program code and / or instructions or instruction sets that enable any of the devices and / or systems to perform as described herein. A machine or computer readable medium may be a non-transient article or medium such as a non-transient computer-readable medium and may be used with any of the examples mentioned above, except that it does not include the transient signal itself. It essentially includes those elements that can temporarily store data in a "transient" manner such as RAM, in addition to the signal itself.

[0110] As used in any implementation described herein, the term "module" refers to any combination of software logic and / or firmware logic configured to provide the functionality described herein. The software may be embodied as a software package, code and / or instruction set, and / or firmware that stores instructions executed by programmable circuitry. Modules may be embodied, in whole or individually, as implementations for use as part of a larger system, such as an integrated circuit (IC), a system on a chip (SoC), or the like.

[0111] As used in any implementation scheme described herein, the term "logic unit" refers to any combination of firmware logic and / or hardware logic configured to provide the functions described herein. As used in any embodiment described herein, "hardware" can include, for example, hard-wired circuits, programmable circuits, state machine circuits and / or firmware that stores instructions executed by programmable circuits, alone or in any combination. Logic units can be embodied as a whole or individually to form parts of larger systems, such as integrated circuits (ICs), systems on chips (SoCs), etc. For example, logic units can be embodied in logic circuits for implementing firmware or hardware of the systems discussed herein. Further, those of ordinary skill in the art will appreciate that the operations performed by hardware and / or firmware can also utilize parts of software to implement the functions of logic units.

[0112] As used in any implementation described herein, the terms "engine" and / or "component" may refer to a module or logical unit, as such terms are described above. Thus, the terms "engine" and / or "component" may refer to any combination of software logic, firmware logic, and / or hardware logic configured to provide the functionality described herein. For example, one of ordinary skill in the art will appreciate that operations performed by hardware and / or firmware may alternatively be implemented via software modules that may be embodied as software packages, codes, and / or instruction sets, and that logical units may also utilize portions of software to implement their functionality.

[0113] refer to Figure 10 , an example image processing system 1000 is arranged in accordance with at least some implementations of the present disclosure. In various implementations, the example image processing system 1000 may have an imaging device 1002 to form or receive captured image data. This may be implemented in various ways. Thus, in one form, the imaging processing system 1000 may be a digital camera or other image capture device, and in such a case the imaging device 1002 may be camera hardware and camera sensor software, modules, or components 1008. In other examples, the imaging processing system 1000 may have an imaging device 1002 that includes or may be a camera, and the logic module 1004 may be in remote communication with or otherwise communicatively coupled to the imaging device 1002 for further processing of the image data.

[0114] In either case, such technology may include a camera such as a digital camera system, a dedicated camera device, or an imaging phone, whether a still picture or video camera or some combination of the two. Thus, in one form, the imaging device 1002 may include camera hardware and optics including one or more sensors and autofocus, zoom, aperture, ND filter, auto exposure, flash, and actuator controls. These controls may be part of a sensor module or component 1006 for operating the sensors. The sensor component 1006 may be part of the imaging device 1002, or may be part of the logic module 1004, or both. Such a sensor component may be used to generate an image for a viewfinder and to capture still pictures or video. The imaging device 1002 may also have a lens, an image sensor with an RGB Bayer color filter, an analog amplifier, an A / D converter, other components that convert incident light into a digital signal, and / or combinations thereof. The digital signal may also be referred to herein as raw image data.

[0115] Other forms include camera sensor type imaging devices (e.g., webcam or webcam sensor or other complementary metal oxide semiconductor image sensor (CMOS)) instead of using a red-green-blue (RGB) depth camera and / or microphone array to locate who is speaking. The camera sensor can also support other types of electronic shutters, such as a global shutter in addition to or instead of a rolling shutter, as well as many other shutter types, as long as a multi-frame statistics collection window can be used. In other examples, an RGB depth camera and / or microphone array can be used in addition to or instead of a camera sensor. In some examples, the imaging device 1002 can be provided with an eye tracking camera.

[0116] In the illustrated example, the logic module 1004 may include a statistics unit 1010 having a window setting unit 1011 that sets the start and end of a multi-frame statistics collection window, a 3A statistics data collection unit 1012 that performs statistics calculations, and other statistics collection units 1013, and a final statistics generation unit 1014 that calculates upper-level statistics or frame-level statistics, as described above. The logic unit 1004 may also have applications 1015 that use the statistics, such as an automatic white balance control or component 1016, an autofocus (AF) component 1017, and an automatic exposure control (AEC) module or component 1018. The statistics unit 1010 (as unit 1024) and / or the applications 1015 (as unit 1028) may be provided as part of the logic module and / or an onboard processor 1020, such as an ISP 1022. This may include instructions for operating the unit, statistics being analyzed by the unit and stored in registers as described above, or both. The logic module 1004 may be communicatively coupled to the imaging device 1002 to receive the raw image data and embedded data described herein. Otherwise, a memory storage(s) 1034 may be provided to store the statistical information data in a buffer 1036 formed of RAM such as DRAM.

[0117] Image processing system 1000 may have one or more of the following: a processor 1020 (which may include a dedicated image signal processor (ISP) 1022, such as an Intel Atom), memory storage 1024, one or more displays 1026, an encoder 1038, and an antenna 1030. In one example implementation, image processing system 100 may have a display 1026, at least one processor 1020 communicatively coupled to the display, and at least one memory 1024 communicatively coupled to the processor and having a buffer 1036, for example, for storing 3A and other statistical information data. Encoder 1028 and antenna 1030 may be provided for compressing modified image data for transmission to other devices that may display or store the image. It will be understood that image processing system 1000 may also include a decoder (or encoder 1028 may include a decoder) to receive and decode image data for processing by system 1000. Otherwise, processed image 1032 may be displayed on display 1026 or stored in memory 1024. As shown, any of these components may be capable of communicating with each other and / or with portions of the logic module 1004 and / or the imaging device 1002. Thus, the processor 1020 may be communicatively coupled to both the imaging device 1002 and the logic module 1004 for operating these components. Figure 10 The illustrated image processing system 1000 may include a specific set of blocks or actions associated with specific components or modules, but these blocks or actions may be associated with components or modules different from the specific components or modules illustrated herein.

[0118] refer to Figure 11 , according to one or more aspects of the image processing system described herein, the example system 1000 of the present disclosure operates. It will be understood from the characteristics of the system components described below that these components can be associated with or used to operate a specific part or parts of the above-mentioned image processing system. In various implementations, the system 1100 can be a media system, although the system 1100 is not limited to this context. For example, the system 1100 can be incorporated into a digital still camera, a digital video camera, a mobile device with camera or video capabilities such as an imaging phone, a webcam, a personal computer (PC), a laptop computer, an ultra-laptop computer, a tablet computer, a touchpad, a portable computer, a handheld computer, a palmtop computer, a personal digital assistant (PDA), a cellular phone, a combination of a cellular phone / PDA, a television, a smart device (e.g., a smart phone, a smart tablet computer or a smart TV), a mobile Internet device (MID), a messaging device, a data communication device, and the like.

[0119] In various implementations, system 1100 may include a platform 1102 coupled to a display 1120. Platform 1102 may receive content from content devices such as content services device(s) 1130 or content delivery device(s) 1140 or other similar content sources. A navigation controller 1150 including one or more navigation features may be used to interact with, for example, platform 1102 and / or display 1120. Each of these components is described in greater detail below.

[0120] In various implementations, the platform 1102 may include any combination of a chipset 1105, a processor 1110, memory 1112, storage 1114, a graphics subsystem 1115, applications 1116, and / or a radio 1118. The chipset 1105 may provide intercommunication between the processor 1110, memory 1112, storage 1114, graphics subsystem 1115, applications 1116, and / or radio 1118. For example, the chipset 1105 may include a storage adapter (not depicted) capable of providing intercommunication with the storage 1114.

[0121] The processor 1110 may be implemented as a complex instruction set computer (CISC) or reduced instruction set computer (RISC) processor, an x86 instruction set compatible processor, a multi-core or any other microprocessor or central processing unit (CPU). In various implementations, the processor 1110 may be a dual-core processor(s), a dual-core mobile processor(s), etc.

[0122] The memory 1112 may be implemented as a volatile memory device such as, but not limited to, a random access memory (RAM), a dynamic random access memory (DRAM), or a static RAM (SRAM).

[0123] The storage device 1114 may be implemented as a non-volatile storage device such as, but not limited to, a magnetic disk drive, an optical disk drive, a tape drive, an internal storage device, an attached storage device, flash memory, battery-backed SDRAM (synchronous DRAM), and / or a network-accessible storage device. In various implementations, the storage device 1114 may include technology for increasing the protection of valuable digital media with enhanced storage performance, such as when multiple hard drives are included.

[0124] Graphics subsystem 1115 may perform processing of images, such as still images or video, for display. Graphics subsystem 1115 may be, for example, a graphics processing unit (GPU) or a visual processing unit (VPU). An analog or digital interface may be used to communicatively couple graphics subsystem 1115 and display 1120. For example, the interface may be any of a High-Definition Multimedia Interface, DisplayPort, wireless HDMI, and / or wireless HD-compatible technologies. Graphics subsystem 1115 may be integrated into processor 1110 or chipset 1105. In some implementations, graphics subsystem 1115 may be a standalone card communicatively coupled to chipset 1105.

[0125] The graphics and / or video processing techniques described herein may be implemented in a variety of hardware architectures. For example, graphics and / or video functionality may be integrated into a chipset. Alternatively, a discrete graphics and / or video processor may be used. As yet another implementation, graphics and / or video functionality may be provided by a general-purpose processor (including a multi-core processor). In further embodiments, these functions may be implemented in consumer electronic devices.

[0126] Radio 1118 may include one or more radios capable of sending and receiving signals using various suitable wireless communication technologies. Such technologies may involve communication across one or more wireless networks. Example wireless networks include, but are not limited to, wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area network (WMANs), cellular networks, and satellite networks. When communicating across such networks, radio 818 may operate in accordance with one or more applicable standards in any version.

[0127] In various implementations, the display 1120 may include any television-type monitor or display. The display 1120 may include, for example, a computer display screen, a touchscreen display, a video monitor, a television-type device, and / or a television. The display 1120 may be digital and / or analog. In various implementations, the display 1120 may be a holographic display. Similarly, the display 1120 may be a transparent surface that can receive visual projections. Such projections can convey various forms of information, images, and / or objects. For example, such projections can be visual overlays for mobile augmented reality (MAR) applications. Under the control of one or more software applications 1116, the platform 1102 may display a user interface 1122 on the display 1120.

[0128] In various implementations, content services device(s) 1130 may be hosted by any national, international, and / or independent service and, in turn, accessible to platform 1102 via the Internet, for example. Content services device(s) 1130 may be coupled to platform 1102 and / or display 1120. Platform 1102 and / or content services device(s) 1130 may be coupled to network 1160 to communicate (e.g., send and / or receive) media information to and from network 1160. Content delivery device(s) 1140 may also be coupled to platform 1102 and / or display 1120.

[0129] In various implementations, content services device(s) 1130 may comprise a cable box, a personal computer, a network, a telephone, an Internet-enabled device or appliance capable of delivering digital information and / or content, and any other similar device capable of communicating content unidirectionally or bidirectionally between content providers and platform 1102 and / or display device 1120, either directly or via network 1160. It will be appreciated that content may be communicated unidirectionally and / or bidirectionally to and from any one of the components in system 1100 and content providers via network 1160. Examples of content may include any media information, including, for example, video, music, medical and gaming information, and the like.

[0130] (One or more) content service device 1130 can receive content such as cable television programs including media information, digital information and / or other content. Examples of content providers may include any cable or satellite television or radio or Internet content providers. The examples provided are not intended to limit implementations according to the present disclosure in any way.

[0131] In various implementations, platform 1102 may receive control signals from a navigation controller 1150 having one or more navigation features. The navigation features of controller 1150 may be used to interact with, for example, user interface 1122. In various embodiments, navigation controller 1150 may be a pointing device, which may be a computer hardware component (particularly a human interface device) that allows a user to input spatial (e.g., continuous and multi-dimensional) data into a computer. Many systems, such as graphical user interfaces (GUIs), as well as televisions and monitors, allow a user to control data using physical gestures and provide data to a computer or television.

[0132] Movements of the navigation features of controller 1150 may be replicated on a display device (e.g., display 1120) through movements of a pointer, cursor, focus ring, or other visual indicators displayed on the display device. For example, under the control of software applications 1116, the navigation features located on navigation controller 1150 may be mapped to virtual navigation features displayed, for example, on user interface 1122. In embodiments, controller 1150 may not be a separate component but may be integrated into platform 1102 and / or display 1120. The present disclosure, however, is not intended to be limited to the elements or in the context shown or described herein.

[0133] In various implementations, drivers (not shown) may include technology that enables a user to instantly turn platform 1102 on and off, for example, after initial power-up, like turning a television on and off with the touch of a button, when enabled. Even when the platform is "off," program logic may allow platform 1102 to stream content to a media adapter or other content service device(s) 1130 or content delivery device(s) 1140. Additionally, for example, chipset 1105 may include hardware and / or software support for 8.1 surround audio and / or high-definition (7.1) surround audio. Drivers may include a graphics driver for integrated graphics platforms. In various embodiments, the graphics driver may include a peripheral component interconnect (PCI) Express graphics card.

[0134] In various implementations, any one or more of the components shown in system 1100 may be integrated. For example, platform 1102 and content service device(s) 1130 may be integrated, or platform 1102 and content delivery device(s) 1140 may be integrated, or, for example, platform 1102, content service device(s) 1130, and content delivery device(s) 1140 may be integrated. In various embodiments, platform 1102 and display 1120 may be an integrated unit. For example, display 1120 and content service device(s) 1130 may be integrated, or display 1120 and content delivery device(s) 1140 may be integrated. These examples are not intended to limit the present disclosure.

[0135] In various embodiments, system 1100 may be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, system 1100 may include components and interfaces suitable for communicating over a wireless shared medium, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, and the like. Examples of wireless shared media may include portions of the wireless spectrum, such as the RF spectrum. When implemented as a wired system, system 1100 may include components and interfaces suitable for communicating over a wired communication medium, such as an input / output (I / O) adapter, a physical connector for connecting the I / O adapter to a corresponding wired communication medium, a network interface card (NIC), a disk controller, a video controller, an audio controller, and the like. Examples of wired communication media may include wires, cables, metal traces, printed circuit boards (PCBs), backplanes, switch fabrics, semiconductor materials, twisted-pair wiring, coaxial cables, optical fibers, and the like.

[0136] Platform 1102 may establish one or more logical or physical channels to communicate information. The information may include media information and control information. Media information may refer to any data representing content intended for a user. For example, examples of content may include data such as from a voice conversation, a video conference, a streaming video, an electronic mail ("email") message, a voicemail message, alphanumeric symbols, graphics, images, video, text, and the like. Data from a voice conversation may be, for example, speech information, periods of silence, background noise, comfort noise, tones, and the like. Control information may refer to any data representing commands, instructions, or control words for an automated system. For example, control information may be used to route media information through a system, or to instruct a node to process the media information in a predefined manner. However, implementations are not limited to Figure 11 Elements in the context shown or described.

[0137] refer to Figure 12 , small form factor device 1200 is one example of a different physical style or form factor in which system 1000 or 1100 may be embodied. In this way, device 1200 may be implemented as a mobile computing device with wireless capabilities. A mobile computing device may refer to any device having a processing system and a mobile power source or supply, such as, for example, one or more batteries.

[0138] As described above, examples of mobile computing devices may include a digital still camera, a digital video camera, a mobile device with camera or video capabilities such as an imaging phone, a webcam, a personal computer (PC), a laptop computer, an ultralaptop computer, a tablet computer, a touchpad, a portable computer, a handheld computer, a PDA, a personal digital assistant (PDA), a cellular phone, a combination cellular phone / PDA, a television, a smart device (e.g., a smart phone, smart tablet, or smart television), a mobile Internet device (MID), a messaging device, a data communication device, and the like.

[0139] Examples of mobile computing devices also include computers that are arranged to be worn by a person, such as a wrist computer, finger computer, ring computer, eyeglass computer, belt-clip computer, arm-band computer, shoe computers, clothing computers, and other wearable computers. In various embodiments, for example, the mobile computing device may be implemented as a smartphone capable of executing computer applications as well as voice communications and / or data communications. Although some embodiments have been described with the mobile computing device implemented as a smartphone by way of example, it will be appreciated that other embodiments may also be implemented using other wireless mobile computing devices. The implementation is not limited in this context.

[0140] like Figure 12 As shown, device 1200 may include a housing having a front portion 1201 and a rear portion 1202. Device 1200 includes a display 1204, an input / output (I / O) device 1206, and an integrated antenna 1208. Device 1200 may also include navigation features 1212. I / O device 1206 may include any suitable I / O device for inputting information into the mobile computing device. Examples of I / O device 1206 may include an alphanumeric keyboard, a numeric keypad, a touchpad, input keys, buttons, switches, a microphone, a speaker, voice recognition equipment and software, etc. Information may also be entered into device 1200 via microphone 1214 or may be digitized via a voice recognition device. As shown, device 1200 may include a camera 1205 (e.g., including at least one lens, aperture, and imaging sensor) integrated into rear portion 1202 (or elsewhere) of device 1200, and a flash 1210. This implementation is not limited in this context.

[0141] The various forms of the equipment and process described herein can be realized using hardware elements, software elements or a combination of the two.The example of hardware elements may include a processor, a microprocessor, a circuit, a circuit element (for example, a transistor, a resistor, a capacitor, an inductor, etc.), an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device (PLD), a digital signal processor (DSP), a field programmable gate array (FPGA), a logic gate, a register, a semiconductor device, a chip, a microchip, a chipset, etc. The example of software may include a software component, a program, an application, a computer program, an application program, a system program, a machine program, an operating system software, middleware, firmware, a software module, a routine, a subroutine, a function, a method, a process, a software interface, an application program interface (API), an instruction set, a computing code, a computer code, a code segment, a computer code segment, a word, a value, a symbol or any combination thereof. Determine whether to use hardware elements and / or software elements to realize an embodiment and can vary according to any number of factors, such as desired computing rate, power level, thermal tolerance, processing cycle budget, input data rate, output data rate, memory resources, data bus speed and other design or performance constraints.

[0142] One or more aspects of at least one embodiment may be implemented as representative instructions stored on a machine-readable medium that represent various logic within a processor, which, when read by a machine, causes the machine to fabricate logic for performing the techniques described herein. Such representations, known as "IP cores," may be stored on a tangible, machine-readable medium and supplied to various customers or manufacturing facilities to load into fabrication machines that actually manufacture the logic or processor.

[0143] Although certain features set forth herein have been described with reference to various implementations, this description is not intended to be construed in a limiting sense. Accordingly, various modifications of the implementations described herein and other implementations that are apparent to those skilled in the art to which the present disclosure pertains are deemed to fall within the spirit and scope of the present disclosure.

[0144] The following examples involve further implementation scenarios.

[0145] As an example, a computer-implemented method of image processing includes: receiving captured image data of a frame of a video sequence; generating a representation of the single frame of image data statistics, including providing an option to generate the statistics using image data of one frame, two frames, or more frames; and allowing access to the representation of the single frame of image data statistics to set image capture parameters for capturing the next available frame of the video sequence or to modify the image data for processing the next available frame of the video sequence, or both.

[0146] As another implementation, the method includes wherein the image data statistics include 3A statistics for at least one of: auto white balance, auto focus and auto exposure control, digital video stabilization statistics, local tone mapping (LTM) statistics, global tone mapping (GTM), and dynamic range conversion statistics; wherein the next available frame is the next frame of the two frames used to represent the single frame of the image data statistics, or wherein the next available frame is the next frame immediately after the next frame of the two frames used to represent the single frame of the image data statistics; wherein the representation of the single frame of the image data statistics is a spatial representation having the same number and arrangement of pixels as the single frame and based on pixel values from both frames. The method includes: forming the representation of the single frame of the image data statistics using image data of a lower integral or pixel row of a previous frame and image data of an upper integral or pixel row of a subsequent frame; and generating the statistics during a statistics collection window, the statistics collection window starting at a first frame and ending at a second frame, wherein the first frame and the second frame are the two frames.

[0147] The method may also include at least one of the following: (1) wherein statistical information is generated during the window by using image data from the same total number of rows of pixels as in a single frame, and (2) wherein statistical information is generated during the window by using image data from different numbers of rows of pixels from two of the two frames; wherein the window extends substantially continuously from one frame to the next such that statistical information processing continues during vertical blanking times between transmissions of raw image data from the sensor; and wherein the start or end of the window, or both, depends at least in part on the time required to generate the statistical information to be used by the application, rather than corresponding solely to the start or end of the exposure time of a single frame. The method includes at least one of the following: (1) storing generated statistical information determined using image data of only one of the two frames more locally than in random access memory relative to a processor generating the statistical information so as to use the statistical information to determine camera parameters or display modifications while storing generated statistical information for the other of the two frames on random access memory; (2) scaling, which includes matching statistical information of one of the frames with statistical information of the other of the two frames to form a single frame representation of the image data statistics; and (3) processing differences in statistical information from one frame to another, including using image data of more than one frame to form a single frame representation of the image data statistics used to align specific camera or display parameters.

[0148] As yet another embodiment, a system for image processing comprises: an image capture device for capturing frames of a video sequence; at least one processor communicatively coupled to the image capture device; at least one memory communicatively coupled to the at least one processor; and a statistics unit operated by the processor and configured to operate by: receiving captured image data of frames of the video sequence; generating a representation of a single frame of image data statistics using a statistics collection time window extending over at least two frames, and including an option to generate the representation of the single frame of image data statistics using image data of one frame, two frames or more frames; and allowing access to the representation of the single frame of image data statistics to set image capture parameters so as to capture the next available frame of the video sequence or to modify image data so as to process the next available frame of the video sequence, or both.

[0149] As another example, the system includes storing the image data or statistics, or both, more locally than in random access memory (RAM) without storing the image data or statistics, or both, in RAM so as to process the image data and statistics to generate camera parameters or modify image data for displaying an image; and the system includes at least one of the following: (1) storing the image data or statistics, or both, of only one of the at least two frames more locally than in RAM so as to use the image data or statistics, or both, to generate the camera parameters or modify image data to process the image for displaying an image; (2) wherein the at least two frames include two frames, and the method includes storing the image data statistics of both of the two frames more locally than in RAM so as to use the image data statistics to generate the camera parameters or modify image data for displaying an image; and (3) the system includes storing the image data or statistical information or both in at least one image signal processor (ISP) register, rather than storing the image data or statistical information or both in random access memory, to use the image data and statistical information to form camera parameters or modify the image data for displaying an image; wherein the window start is set at the same fixed position relative to the integral or pixel line of the frame for each window start frame; wherein the window start is set at least in part based on both of the following: (1) when sufficient image data is obtained to generate statistical information; and (2) when image data that has not been used to generate statistical information for a previous window is provided; and wherein the start and end of the window are associated with a region of interest (ROI) on the frame so that image data outside the region of interest is not collected to calculate statistical information based on the ROI.

[0150] As a further example, the system includes wherein the representation of a single frame of image data statistics represents the same number and arrangement of pixels as the single frame and is based on pixel values from both of the at least two frames; and wherein the instructions cause the computing device to operate by generating statistics during a statistics collection window that begins at a previous frame and ends at a subsequent frame, wherein the previous frame and the subsequent frame are the at least two frames, and wherein the start or end or both of the window are freely located at different positions from frame to frame and relative to the integral or pixel line forming the frame.

[0151] By one approach, an article of manufacture having a computer-readable medium comprising a plurality of instructions that, in response to being executed on a computing device, cause the computing device to operate by: receiving captured image data of a frame of a video sequence; generating a representation of a single frame of image data statistics, including providing an option to generate the statistics using image data of one frame, two frames, or more frames; and enabling access to the representation of the single frame of image data statistics to use the image data statistics to set image capture parameters for capturing a next available frame of the video sequence or to modify image data for processing the next available frame of the video sequence, or both.

[0152] By another approach, the instructions cause a computing device to include wherein a representation of a single frame of image data statistics represents the same number and arrangement of pixels as the single frame and is based on pixel values from both of the at least two frames; and wherein the instructions cause the computing device to operate by generating statistics during a statistics collection window that begins at a previous frame and ends at a subsequent frame, wherein the previous frame and the subsequent frame are the at least two frames, and wherein the start or end or both of the window are freely located at different positions from frame to frame and relative to the integral or pixel line forming the frame.

[0153] In a further example, at least one machine-readable medium may include a plurality of instructions that, in response to being executed on a computing device, cause the computing device to perform the method according to any of the above examples.

[0154] In yet another example, an apparatus may include means for performing the method according to any one of the above examples.

[0155] The above examples may include specific combinations of features. However, these examples are not limited in this respect, and in various implementations, the examples may include employing only a subset of such features, employing such features in a different order, employing such features in a different combination, and / or employing additional features compared to those explicitly listed. For example, all features described with respect to any example method herein may be implemented with respect to any example apparatus, example system, and / or example article of manufacture, and vice versa.

Claims

1. A computer-implemented method for image processing, comprising: receiving captured image data of a frame of a video sequence; generating a representation of a single frame of image data statistics, including providing an option to generate the statistics using image data from two frames; as well as allowing access to a representation of a single frame of image data statistics to set image capture parameters for capturing a next available frame of a video sequence, wherein the next available frame is the next of two frames used to represent the single frame of image data statistics or the next available frame is the immediately next frame after the next of two frames used to represent the single frame of image data statistics, Wherein the representation of the single frame of image data statistics is a spatial representation having the same number and arrangement of pixels as the single frame and based on pixel values from both frames, wherein the representation of the single frame of image data statistics is formed by using a lower integral or pixel row of image data from a previous frame and an upper integral or pixel row of image data from a subsequent frame.

2. The method of claim 1, wherein the image data statistics include 3A statistics for at least one of: auto white balance, auto focus, and auto exposure control.

3. The method of claim 1, wherein the image data statistics include at least one of digital video stabilization statistics, local tone mapping (LTM) statistics, global tone mapping (GTM), and dynamic range conversion statistics.

4. The method of claim 1 , comprising generating statistics during a statistics collection window, the statistics collection window starting at a first frame and ending at a second frame, wherein the first frame and the second frame are the two frames.

5. The method of claim 4, wherein the statistical information is generated during the window by using the same total number of rows of integrals or pixels of image data as in a single frame.

6. The method of claim 4, wherein the statistical information is generated during the window by using image data of different rows of pixels from two of the two frames.

7. The method of claim 4, wherein the window extends substantially continuously from one frame to the next such that statistical information processing continues during vertical blanking times between transmissions of raw image data from the sensor.

8. The method of claim 4, wherein the start or end of the window, or both, depends at least in part on the time required to generate statistical information to be used by the application, rather than simply corresponding to the start or end of the exposure time of a single frame.

9. The method of claim 1 , comprising storing generated statistical information determined using image data of the two frames more locally than in a random access memory relative to a processor generating the statistical information so as to determine camera parameters using the statistical information.

10. The method of claim 1 including scaling, said scaling including matching statistics of one of the frames with statistics of the other of the two frames to form a representation of a single frame of image data statistics.

11. The method of claim 1 , including processing differences in statistics from one frame to another, including using image data from more than one frame to form a representation of a single frame of image data statistics that is used to align particular camera or display parameters.

12. A system for image processing, comprising: an image capture device for capturing frames of a video sequence; at least one processor communicatively coupled to the image capture device; at least one memory communicatively coupled to the at least one processor; as well as A statistics unit, operated by a processor, and configured to operate by: receiving captured image data of frames of the video sequence; generating a single frame representation of image data statistics using a statistics collection time window extending over two frames, and including an option to generate the single frame representation of image data statistics using image data from two frames; as well as allowing access to the representation of the single frame of the image data statistics to set image capture parameters for capturing a next available frame of the video sequence, wherein the next available frame is the next of the two frames representing the single frame of image data statistics or the next available frame is the immediately next frame after the next of the two frames representing the single frame of image data statistics, Wherein the representation of the single frame of image data statistics is a spatial representation having the same number and arrangement of pixels as the single frame and based on pixel values from both frames, wherein the representation of the single frame of image data statistics is formed by using a lower integral or pixel row of image data from a previous frame and an upper integral or pixel row of image data from a subsequent frame.

13. The system of claim 12 , wherein the statistics unit is configured to save the image data or the statistics or both more locally than in a random access memory (RAM) without storing the image data or the statistics or both in the RAM so as to process the image data and the statistics to generate the camera parameters.

14. The system of claim 13, wherein the statistics unit is configured to save the image data statistics of both of the two frames more locally than in RAM to generate the camera parameters using the image data statistics.

15. The system of claim 13 , wherein the statistics unit is configured to save the image data or the statistics or both in at least one image signal processor (ISP) register instead of storing the image data or the statistics or both in a random access memory to form camera parameters using the image data and the statistics.

16. The system of claim 12, wherein the window start is set at the same fixed position of each window start frame relative to an integral or pixel line of the frame.

17. A system as described in claim 12, wherein the window start is set at least in part based on both: (1) when sufficient image data is obtained to generate statistical information; and (2) when image data is provided that has not been used to generate statistical information for a previous window.

18. The system of claim 12, wherein the start and end of the window are associated with a region of interest (ROI) on the frame such that image data outside the ROI is not collected to calculate statistics based on the ROI.

19. The system of claim 12, wherein the representation of a single frame of image data statistics represents the same number and arrangement of pixels as a single frame and is based on pixel values from both of the two frames; and wherein the statistics unit is configured to operate by generating statistics during a statistics collection window that begins at a previous frame and ends at a subsequent frame, wherein the previous frame and the subsequent frame are the two frames, and wherein the start or end or both of the window are freely located at different positions from frame to frame and relative to the integral or pixel lines forming the frames.

20. An apparatus comprising means for performing the method according to any one of claims 1-11.

Images