Image Signal Processor Resource Management

Abstract

Various aspects relate to an image signal processor that processes frames at a varying frame rate. One example method includes receiving, by an image signal processor, a first sequence of image frames from an image sensor at a first frame rate; processing each image frame of the first sequence of image frames at the first frame rate; and receiving an indication from the image sensor of a change in the frame rate of the image sensor from the first frame rate to a second frame rate. The method also includes configuring one or more filters of the image signal processor to process the image frames from the image sensor in response to receiving the indication of the frame rate change from the image sensor; receiving a second sequence of image frames from the image sensor at a second frame rate; and processing each image frame of the second sequence of image frames at the second frame rate.

Description

[0001] This application claims priority to U.S. application No. 15 / 931,770, filed on May 14, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure generally relates to image or video capture devices, including resource management of an image signal processor that processes frames at varying frame rates. Background Art

[0003] Many devices include one or more image sensors that can be used to capture a sequence of image frames for video. For example, a smartphone or tablet computer includes one or more image sensors for capturing video. The device can adjust the frame rate used to capture and process image frames. For example, an image sensor may be capable of capturing image frames at multiple frame rates. The device instructs the image sensor to capture image frames at a specific frame rate, and the image sensor captures a sequence of image frames at the instructed frame rate. Each image frame is processed by an image signal processor, and the processed sequence of image frames can be encoded to generate a video. Summary of the Invention

[0004] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] Some aspects of the present disclosure relate to resource management of an image signal processor that processes frames at varying frame rates. An example method includes receiving, by an image signal processor, a first sequence of image frames from an image sensor at a first frame rate. The method also includes processing, by the image signal processor, each image frame of the first sequence of image frames at the first frame rate. The method also includes receiving, by the image signal processor, an indication of a change in frame rate of the image sensor from the first frame rate to a second frame rate. The method also includes configuring one or more filters of the image signal processor to process the image frames from the image sensor in response to receiving the indication of the change in frame rate from the image sensor. The method also includes receiving, by the image signal processor, a second sequence of image frames from the image sensor at a second frame rate. The method also includes processing, by the image signal processor, each image frame of the second sequence of image frames at the second frame rate.

[0006] In some embodiments, an indication of the frame rate change is included in metadata of image frames subsequent to the first sequence of image frames from the image sensor. The method may further include indicating, by the image signal processor, the frame rate change to the application processor once during a batch of a plurality of image frames from the second sequence of image frames. The number of image frames in the batch may be determined by the image signal processor based on the received indication of the frame rate change. In some embodiments, the number of image frames in the batch is based on a frame rate associated with the batch. In some embodiments, the image signal processor indicates the frame rate change to the application processor after configuring the one or more filters.

[0007] The method may also include processing, by the image signal processor, a first interval of image frames of the first sequence of image frames to generate a first partial preview stream. The first interval is based on a first frame rate. The method may also include processing, by the image signal processor, a second interval of image frames of the second sequence of image frames to generate a second partial preview stream. The second interval is based on a second frame rate. The method may also include generating, by the image signal processor, a video stream comprising the processed first sequence of image frames and the processed second sequence of image frames, outputting, by the image signal processor, the preview stream for display, and outputting, by the image signal processor, the video stream. In some embodiments, the preview stream uses a fixed frame rate and the video stream uses a variable frame rate.

[0008] The method may also include storing, in a memory, a lookup table that maps different frame rates to settings for configuring one or more filters. The image signal processor uses the lookup table to configure the one or more filters based on the frame rate variation. In some embodiments, the lookup table also maps different frame rates to a number of image frames per batch, for communication between the image signal processor and the application processor.

[0009] The method may further include encoding, by an encoder, the first sequence of image frames and the second sequence of image frames to generate a video stream.

[0010] An example device includes an image signal processor configured to receive a first sequence of image frames from an image sensor at a first frame rate; process each image frame in the first sequence of image frames at the first frame rate; receive an indication from the image sensor of a frame rate change of the image sensor from the first frame rate to a second frame rate; in response to receiving the indication of the frame rate change from the image sensor, configure one or more filters of the image signal processor to process the image frames from the image sensor; receive a second sequence of image frames from the image sensor at a second frame rate; and process each image frame in the second sequence of image frames at the second frame rate. In some embodiments, the indication of the frame rate change is included in metadata of image frames subsequent to the first sequence of image frames from the image sensor.

[0011] The image signal processor can be further configured to indicate to the application processor a frame rate change during a batch of a plurality of image frames from the second sequence of image frames. The number of image frames in the batch is determined by the image signal processor based on the received indication of the frame rate change. In some embodiments, the number of image frames in the batch is based on a frame rate associated with the batch.

[0012] The image signal processor may be further configured to indicate a frame rate change to the application processor after configuring the one or more filters. The image signal processor may be further configured to process a first interval of image frames of a first image frame sequence to generate a first partial preview stream (the first interval is based on the first frame rate); process a second interval of image frames of a second image frame sequence to generate a second partial preview stream (the second interval is based on the second frame rate); generate a video stream including the processed first image frame sequence and the processed second image frame sequence; output the preview stream for display; and output the video stream. In some embodiments, the preview stream uses a fixed frame rate and the video stream uses a variable frame rate.

[0013] In some embodiments, the device further includes a memory for storing a lookup table that maps different frame rates to settings for configuring one or more filters. The image signal processor is further configured to use the lookup table to configure the one or more filters based on the frame rate variation. The lookup table may also map the different frame rates to a number of image frames per batch, for communication between the image signal processor and the application processor.

[0014] In some embodiments, the device includes an encoder to encode the first sequence of image frames and the second sequence of image frames to generate a video stream. The device may include a display to display a preview stream. The device may also include an application processor to execute a camera application associated with the first sequence of image frames and the second sequence of image frames. The device may also include an image sensor configured to capture the first sequence of image frames; capture the second sequence of image frames; and indicate a frame rate change to the image signal processor.

[0015] An example non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device, cause the device to receive, by an image signal processor, a first sequence of image frames from an image sensor at a first frame rate; process, by the image signal processor, each image frame in the first sequence of image frames at the first frame rate; receive, by the image signal processor, an indication of a frame rate change from the first frame rate to a second frame rate from the image sensor; configure, in response to receiving the indication of the frame rate change from the image sensor, one or more filters of the image signal processor to process the image frames from the image sensor; receive, by the image signal processor, a second sequence of image frames from the image sensor at a second frame rate; and process, by the image signal processor, each image frame in the second sequence of image frames at the second frame rate. In some embodiments, the indication of the frame rate change is included in metadata of image frames subsequent to the first sequence of image frames from the image sensor.

[0016] Execution of the instructions may further cause the device to: indicate, by the image signal processor, to the application processor, a frame rate change during a batch of a plurality of image frames from the second sequence of image frames. The number of image frames in the batch is determined by the image signal processor based on the received indication of the frame rate change. In some embodiments, the number of image frames in the batch is based on a frame rate associated with the batch. In some embodiments, the image signal processor indicates the frame rate change to the application processor after configuring the one or more filters.

[0017] Execution of the instructions may further cause the device to process, by the image signal processor, a first interval of image frames of the first image frame sequence to generate a first partial preview stream (the first interval being based on a first frame rate); process, by the image signal processor, a second interval of image frames of the second image frame sequence to generate a second partial preview stream (the second interval being based on a second frame rate); generate, by the image signal processor, a video stream comprising the processed first image frame sequence and the processed second image frame sequence; output, by the image signal processor, the preview stream for display; and output, by the image signal processor, the video stream. In some embodiments, the preview stream uses a fixed frame rate and the video stream uses a variable frame rate.

[0018] Execution of the instructions may further cause the device to store, in memory, a lookup table that maps different frame rates to settings for configuring one or more filters. The image signal processor uses the lookup table to configure the one or more filters based on the frame rate change. The lookup table may also map the different frame rates to a number of image frames per batch for communication between the image signal processor and the application processor.

[0019] Execution of the instructions may further cause the device to encode, by an encoder, the first sequence of image frames and the second sequence of image frames to generate a video stream.

[0020] Another example device includes means for receiving a first sequence of image frames from an image sensor at a first frame rate; means for processing each image frame in the first sequence of image frames at the first frame rate; means for receiving an indication from the image sensor of a change in frame rate of the image sensor from the first frame rate to a second frame rate; means for configuring one or more filters of an image signal processor to process the image frames from the image sensor in response to receiving the indication of the frame rate change from the image sensor; means for receiving a second sequence of image frames from the image sensor at a second frame rate; and means for processing each image frame in the second sequence of image frames at the second frame rate. In some embodiments, the indication of the frame rate change is included in metadata of image frames subsequent to the first sequence of image frames from the image sensor.

[0021] The device may also include means for indicating a frame rate change to the application processor during a batch of a plurality of image frames from the second sequence of image frames. The number of image frames in the batch is determined by the image signal processor based on the received indication of the frame rate change. In some embodiments, the number of image frames in the batch is based on a frame rate associated with the batch. In some embodiments, the frame rate change is indicated to the application processor after configuring the one or more filters.

[0022] The device may also include components for processing a first interval of image frames of a first sequence of image frames to generate a first partial preview stream (the first interval being based on a first frame rate); components for processing a second interval of image frames of a second sequence of image frames to generate a second partial preview stream (the second interval being based on a second frame rate); components for generating a video stream comprising the processed first sequence of image frames and the processed second sequence of image frames; components for outputting the preview stream for display; and components for outputting the video stream. In some embodiments, the preview stream utilizes a fixed frame rate and the video stream utilizes a varying frame rate. The device may also include components for storing a lookup table that maps different frame rates to settings for configuring one or more filters. The one or more filters are configured using the lookup table based on the frame rate variation. In some embodiments, the lookup table also maps different frame rates to a number of image frames per batch for communication with an application processor.

[0023] The apparatus may further include means for encoding the first sequence of image frames and the second sequence of image frames to generate a video stream. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Various aspects of the present disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings and in which like references refer to similar elements.

[0025] Figure 1 is an example timing diagram depicting frame rate changes initiated by an image sensor.

[0026] Figure 2 is a block diagram of an example device for adjusting frame rate.

[0027] Figure 3A is a block diagram of an example image signal processor for processing frames captured at different frame rates.

[0028] Figure 3B is a block diagram of another example image signal processor for processing frames captured at different frame rates.

[0029] Figure 4 is a description of example contents of an image frame provided by an image sensor to an image signal processor.

[0030] Figure 5 is an illustrative flow chart describing example operations for image frame processing of video.

[0031] Figure 6 is an illustrative flow chart describing example operations for configuring communications between an image signal processor and a processor based on batch size.

[0032] Figure 7 is an illustrative flow chart describing example operations for generating a preview stream. DETAILED DESCRIPTION

[0033] Aspects of the present disclosure may be used in image capture and processing devices used to generate video.Some aspects include resource management of an image signal processor that processes frames at varying frame rates.

[0034] The device can be configured to adjust the frame rate for the video to be generated. For example, the application processor instructs the image signal processor (ISP) to adjust the frame rate for capturing processed image frames. The ISP instructs the image sensor to adjust the frame rate at which it captures image frames, and the ISP configures one or more image processing filters (e.g., a noise reduction filter, an edge enhancement filter, a color balance filter, etc.) to process each captured image frame. The ISP can also configure one or more other components (e.g., a component for generating a preview video) based on the frame rate.

[0035] Today, some image sensors are configured to adjust the frame rate without instructions from an image signal processor or application processor. In some embodiments, the image sensor may include or be coupled to one or more components that measure characteristics of the received light, and the frame rate may be adjusted based on these measurements. For example, an integrated circuit may measure the intensity of the light received at the image sensor. If the light intensity drops below a threshold, the image sensor may reduce the frame rate in order to increase the exposure time per frame. Other measurements may include measuring local motion, global motion, or color cast for the image sensor to determine whether to change the frame rate. The image sensor captures image frames at the new frame rate and outputs the image frames to the ISP at the new frame rate. In this way, the ISP can receive a stream of image frames from the image sensor as the frame rate changes.

[0036] The image sensor indicates the frame rate change to the ISP. For example, the image sensor can be configured to provide image frames to the ISP in the Exchangeable Image Frame (Exif) format. Each image frame in the Exif format includes an Exif header (which may include information about the image frame, such as location information from a global positioning sensor, time information from a clock, etc.). One or more image frames captured at a new frame rate can include an indication of the new frame rate in the Exif header. The ISP processes the Exif header and determines that the frame rate has been changed. In this way, the frame rate change can be initiated by the image sensor rather than the application processor.

[0037] A frame rate change may require the ISP to adjust one or more filters (or other components of the image processing pipeline) to process image frames at the new frame rate. However, the ISP is configured based on instructions from the application processor. As a result, when the image sensor changes its frame rate, the ISP may need instructions from the application processor regarding the new configuration for processing image frames.

[0038] Figure 1 is an example timing diagram 100 depicting a frame rate change initiated by image sensor 102. Image sensor 102 changes its frame rate, and image sensor 102 indicates the frame rate change to ISP 104 (108). For example, headers of image frames (captured at the new frame rate) output from image sensor 102 to ISP 104 indicate the new frame rate. In response to determining that the frame rate is changing, ISP 104 pauses frame processing (110). For example, ISP 104 processes the image frame headers to determine that the frame rate has been changed at image sensor 102. Because ISP 104 is configured based on instructions from application processor 106, ISP 104 does not process input image frames at the new frame rate.

[0039] With the processing of the image frames paused, ISP 104 indicates a frame rate change to application processor 106 (112). For example, ISP 104 may generate and output one or more software instructions to application processor 106 indicating the frame rate change. Different types of ISPs may be coupled to different types of application processors. In this manner, ISP 104 may be configured to generate instructions in a generic or middleware format that can be processed by different types of application processors, including application processor 106.

[0040] The application processor 106 can use the image sensor 102 and the ISP 104 to execute a camera application or other suitable software application for generating video. The application processor 106 (executing the application) receives an indication of a frame rate change from the ISP 104. In some embodiments, the application processor 106 can convert instructions in a general or middleware format into instructions in a format specific to the application processor 106 or the application. The application processor 106 determines from the converted instructions that the frame rate of the image sensor 102 has changed. In response to determining that the frame rate has changed, the application processor 106 determines a configuration of the ISP 104 to process image frames at the new frame rate (114). In some embodiments, the ISP configuration can include changes to one or more image processing filters (e.g., changes to a noise reduction filter, changes to an edge enhancement filter, changes to a color balance filter, changes to a shading filter, changes to a geometric correction filter, reducing or increasing the number of filters to be applied to each image frame, changing the filters to be applied to each image frame, etc.).

[0041] The device may include or be coupled to a display to provide a preview of a stream of image frames captured by image sensor 102. The preview may utilize a lower frame rate than the image frames captured by image sensor 102. For example, image sensor 102 may capture image frames at 60 frames per second (fps) or 120 fps, and the displayed preview may utilize 30 fps. ISP 104 may be configured to process intervals of captured image frames to generate a preview stream that is displayed. For example, if the preview utilizes 30 fps and the image sensor captures frames at 60 fps, ISP 104 may be configured to process every other frame into the preview stream. In some embodiments, ISP 104 generates two or more streams of image frames. For example, one stream may be a preview stream (for the preview to be displayed) and the other stream may be a video stream (for the final video to be generated). Image frames at intervals in the stream of image frames from image sensor 102 may be processed for the preview stream. All image frames (including image frames at intervals and the remaining image frames in the stream) may be processed for the video stream. In some embodiments, a first instance of an image frame is processed for a preview stream, and a second instance of the image frame is processed for a video stream. When comparing the preview stream to the final video stream, latency may be more important than image quality for the preview stream because the preview should be displayed as quickly as possible after the image frame is captured. To reduce latency, processing the image frame for the preview stream may not include applying all filters that are applied when processing the image frame for the video stream.

[0042] The intervals at which image frames are processed for generating the preview stream may be indicated by the application processor 106. In this manner, the ISP 104 may not generate a preview stream when frame processing is paused. The ISP configuration determined by the application processor 106 may also include an indication of the intervals at which image frames from the image sensor 102 are processed for generating the preview stream.

[0043] After application processor 106 determines the ISP configuration for processing image frames at the new frame rate, application processor 106 provides instructions to ISP 104 for configuring ISP 104 to process image frames from image sensor 102 at the new frame rate (116). In some embodiments, the instructions are in a generic or middleware format (e.g., in a format used by ISP 104 to indicate a change in frame rate to application processor 106). ISP 104 processes the instructions and configures ISP 104 to process image frames at the new frame rate (118). When adjusting one or more filters, ISP 104 may adjust the interval at which image frames are processed for the preview stream. ISP 104 may also adjust one or more image processing filters (e.g., a noise reduction filter, an edge enhancement filter, the number of filters, or which filter to apply) to process image frames from image sensor 102. After ISP 104 configures the one or more filters based on the instructions from application processor 106, ISP 104 may resume processing image frames from image sensor 102 (120). While ISP 104 is pausing processing, ISP 104 may discard image frames received from image sensor 102 .

[0044] For frame rates above a threshold frame rate, it may be necessary to pause image processing by the ISP 104 because the time between image frame captures may not be an adequate amount of time for (1) communication to occur between the ISP 104 and the application processor 106 and (2) configuration of one or more filters by the ISP 104 based on the communication. For example, a frame rate of 60 fps corresponds to less than 17 milliseconds (ms) between image frame captures, while a frame rate of 120 fps corresponds to approximately 8 ms between image frame captures. Communication between the ISP 104 and the application processor 106 may require greater than 8 ms (e.g., approximately 17 ms). For example, a hardware layer of the ISP 104 interacts with a software layer to communicate with the application processor 106, and delays in the software layer may cause the time required to be greater than 8 ms. In this manner, the ISP 104 cannot be configured for the next image frame from the image sensor 102 at the new frame rate without pausing processing of the image frame.

[0045] Pausing image frame processing (wherein image frames received during the pause are discarded) can negatively impact the user experience, particularly as frame rates continue to increase (e.g., image sensors configured to capture image frames at frame rates of 240 fps, 480 fps, etc.). For example, when processing is paused due to a frame rate change, the video including the frame rate change may include gaps. In another example, when processing is paused, the preview stream may be suspended.

[0046] One solution is to have the device include multiple ISPs to have a dedicated ISP for each frame rate at which the image sensor can capture image frames. For example, if the image frame can capture image frames at 30fps, 60fps, 120fps, 240fps, 480fps, and 960fps, the device includes at least six ISPs (one for each frame rate). In this way, the corresponding ISP can be used to process received image frames captured at a specific frame rate. One problem is that all ISPs are initialized or otherwise use power even when not in use. As a result, increasing the number of ISPs increases power consumption. Another problem is that increasing the number of ISPs increases the space required in the device. For example, more components (more complex components) may be required to implement a device that includes multiple ISPs for different frame rates.

[0047] In some embodiments, the ISP may be configured to process image frames captured at different frame rates. The resources of the ISP that processes frames at varying frame rates may be managed without interaction with an application processor. For example, the ISP may configure one or more filters without requiring instructions from the application processor to configure the ISP. Configuring the one or more filters may include adjusting the intervals at which image frames are processed for generating a preview stream. In this way, the ISP may not pause processing of image frames when the frame rate changes, and the video may not display interruptions (or may display reduced interruptions). The preview stream also may not be interrupted (or include reduced interruptions) when the frame rate changes because the ISP can adjust the intervals at which image frames are processed without input from the application processor. If the ISP is able to configure itself for processing image frames captured at different frame rates, an application processor may not be required for at least some operations of the ISP.

[0048] In the following description, many specific details of examples such as specific components, circuits, and processes are described to provide a thorough understanding of the present disclosure. As used herein, the term "coupled" means directly connected or connected through one or more intermediary components or circuits. Moreover, in the following description and for the purpose of explanation, specific terms are described to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that these specific details may not be necessary to practice the teachings disclosed herein. In other cases, well-known circuits and devices are shown in the form of block diagrams to avoid confusing the teachings of the present disclosure. Some parts of the following detailed description are presented with respect to programs, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, programs, logic blocks, processes, etc. are considered to be self-consistent sequences of steps or instructions that produce a desired result. The steps are steps that require physical manipulation of physical quantities. These quantities are typically in the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system, but not necessarily so.

[0049] It should be borne in mind, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise apparent from the following discussion, it should be understood that throughout this application, discussions using terms such as "access," "receive," "send," "use," "select," "determine," "normalize," "multiply," "average," "monitor," "compare," "apply," "update," "measure," "derive," "establish," "generate," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities within the computer system's registers and memories and transforms it into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.

[0050] In the figures, a single block may be described as performing one or more functions; however, in actual practice, the one or more functions performed by the block may be performed in a single component or on multiple components, and / or may be performed using hardware, using software, or using a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are described below generally in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as causing a deviation from the scope of this disclosure. In addition, the example device may include components other than those shown, including well-known components such as processors and memories.

[0051] Various aspects of the present disclosure are applicable to any suitable electronic device that includes or is coupled to one or more image sensors capable of capturing a sequence of image frames for video (e.g., a security system, a smartphone, a tablet, a laptop computer, a digital video camera, etc.). Furthermore, various aspects of the present disclosure can be implemented in devices having or coupled to image sensors having the same or different performance and characteristics (e.g., resolution, shutter speed, sensor type, etc.).

[0052] The terms "device" and "apparatus" are not limited to one or a specific number of physical objects (e.g., a smartphone, a camera controller, a processing system, etc.). As used herein, a device can be any electronic device having one or more components that can implement at least some portions of the present disclosure. Although the following description and examples use the term "device" to describe various aspects of the present disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. As used herein, an apparatus can include a device or a portion of a device for performing the described operations.

[0053] Figure 2 is a block diagram of an example device 200 for adjusting the frame rate of a video. The example device 200 includes or is coupled to an image sensor 202. The example device 200 also includes a processor 204, a memory 206 storing instructions 208, and an ISP 212. In some embodiments, the device 200 also includes or is coupled to a display 214 and includes multiple input / output (I / O) components 216. The device 200 may also include or be coupled to a power source 218 for the device 200 (e.g., a battery or a component that couples the device 200 to a power source). The device 200 may include additional features or components not shown. In one example, a wireless interface for a wireless communication device may be included, which may include multiple transceivers and a baseband processor. In another example, one or more sensors (e.g., a gyroscope or a global positioning system (GPS) receiver) may be included in or coupled to the device.

[0054] The image sensor 202 may include one or more image sensors capable of capturing a series of image frames for a video. The image sensor 202 may include one or more image sensors of a camera module of the device 200 (e.g., a single-camera module, a dual-camera module, a triple-camera module, etc., for a smartphone or other suitable device). The image sensor 202 may also include (or be coupled to) one or more lenses for focusing light, one or more apertures for receiving light, one or more shutters for blocking light outside an exposure window, one or more color filter arrays (CFAs) for filtering light outside a specific frequency range, one or more analog front ends for converting analog measurements into digital information, or other suitable components for imaging. The device 200 may also include a flash, a depth sensor, a GPS, or other suitable components for imaging.

[0055] The image sensor 202 can be configured to adjust its frame rate without intervention from the ISP 212 or the processor 204. In some embodiments, the image sensor 202 includes or is coupled to one or more components (e.g., including one or more integrated circuits or other sensors) to measure one or more characteristics of received light. For example, the intensity of the light can be measured. In another example, changes in the intensity of light on the image sensor can be measured over time to determine global motion (caused by motion of the image sensor 202) or local motion (caused by motion of one or more objects in the scene captured by the image sensor 202). The image sensor 202 can be configured to adjust the frame rate based on one or more measurements. For example, if global motion or local motion increases by more than a threshold, the image sensor 202 can increase the frame rate. In another example, the image sensor 202 can reduce the frame rate when the light intensity decreases by more than a threshold. Although some examples for adjusting the frame rate of the image sensor are provided, any suitable measurement, threshold, or other means can be performed to determine when to adjust the frame rate.

[0056] The image sensor 202 is configured to indicate a frame rate change to the ISP 212. In some embodiments, the image sensor 202 captures a first sequence of image frames at a first frame rate and provides it to the ISP 212. The image sensor 202 may change the first frame rate to a second frame rate and continue to capture image frames (e.g., a second sequence of image frames) at the second frame rate. The image sensor 202 provides a second sequence of image frames to the ISP 212 at the second frame rate. When the frame rate change is indicated by the image sensor 202 to the ISP 212, the header of one or more image frames in the second sequence may indicate the frame rate change. For example, the header may indicate the frame rate at which the image frames were captured. In another example, the header may indicate the difference between the frame rates, a specific change in the frame rate, or other suitable manner of indicating the change. Although these examples describe the use of an Exif header to indicate a frame rate change sent by the image sensor 202 to the ISP 212, any suitable format of header may be used to indicate a frame rate change. In addition to or in lieu of an image frame header, other components of the image frame may be used to indicate a frame rate change (e.g., a footer, a specific location in the body of the image frame, etc.). In addition to or in lieu of one or more components of the image frame, other means for indicating a frame rate change may be used. For example, the image sensor 202 may be configured to provide proprietary or standardized signaling external to the image frame to indicate a frame rate change.

[0057] ISP 212 may include one or more image signal processors to process captured image frames provided by image sensor 202. ISP 212 is capable of configuring itself after image sensor 202 indicates a frame rate change, without requiring intervention from processor 204. For example, ISP 212 may configure one or more filters without instructions from processor 204 in response to determining a frame rate change based on one or more Exif headers of image frames received from image sensor 202. ISP 212 may indicate the frame rate change to processor 204 simultaneously with or after configuring ISP 212. For example, ISP 212 may generate one or more general or middleware instructions (e.g., based on a bridge between processor 204 and ISP 212) and provide the instructions to processor 204 when ISP 212 next communicates with processor 204. ISP 212 does not need to wait for the next communication with processor 204 and configuration instructions from processor 204 to configure ISP 212. For example, ISP 212 may store different profiles associated with different frame rates (or frame rate variations) in a memory (e.g., a memory included in or coupled to ISP 212, memory 206, or another suitable memory). ISP 212 may be configured to identify a file to access based on the indicated frame rate or frame rate variation. ISP 212 may access the identified file, and ISP 212 may configure one or more filters based on the accessed file. The association between the profiles and the frame rates (or frame rate variations) may be stored in a lookup table in a memory included in or coupled to ISP 212.

[0058] In some embodiments, ISP 212 is further configured to control one or more aspects of image sensor 202. For example, ISP 212 may still indicate to image sensor 202 frame rate changes initiated by processor 204. ISP 212 may be configured to control other aspects of image sensor 202, including initialization, autofocus, autoexposure or other functions, flash operation, gain to be applied by the analog front end, and the like.

[0059] ISP 212 may still be able to receive configurations from processor 204 (e.g., when a frame rate change is initiated by processor 204). In this way, ISP 212 can configure itself based on an indication of a frame rate change from image sensor 202 or based on instructions from processor 204. ISP 212 may be included on a system-on-chip (SoC) along with processor 204, or ISP 212 may be separate from processor 204 in device 200. ISP 212 may also be configured to provide a stream of processed image frames for video to processor 204 (or other suitable component, such as memory 206 for storage) for encoding or further processing. In some embodiments, ISP 212 may output the processed image frames to a memory (e.g., memory 206 or a different memory accessed by ISP 212 and processor 204). For example, ISP 212 may address the memory in specific sectors (e.g., at fixed intervals associated with the size of the image frame) to store each processed image frame. In some embodiments, the ISP 212 can output a first set of processed image frames to a memory for a preview stream. The first set of processed image frames can be a subset of the image frames processed by the ISP 212, and the preview stream can be accessed by a graphics processor or other suitable component to display the preview stream on the display 214. The ISP 212 can also output a second set of processed image frames for a video stream. The second set of processed image frames can include all image frames processed by the ISP 212, and the video stream can be accessed by the processor 204 for further processing (e.g., for encoding, etc.). The first set of processed image frames can be stored in a first portion of the memory, and the second set of processed image frames can be stored in a second portion of the memory.

[0060] In some aspects, the ISP 212 may execute instructions from a memory (e.g., instructions 208 from the memory 206, instructions stored in a separate memory coupled to or included in the ISP 212, or instructions provided by the processor 204 (after being converted into ISP 212-specific instructions, as applicable). In addition to or in lieu of the ISP 212 being configured to execute software, the ISP 212 may include specific hardware (e.g., one or more integrated circuits (ICs)) to perform one or more operations described in this disclosure.

[0061] The memory 206 can be a non-transitory or non-temporary computer-readable medium that stores computer-executable instructions 208 to perform all or part of one or more operations described in this disclosure. In some embodiments, the instructions 208 include a camera application (or other suitable application) to be executed by the device 200 to generate video. The instructions 208 may also include other applications or programs executed by the device 200 (e.g., an operating system and specific applications other than those for video generation). Execution of the camera application (e.g., by the processor 204) may cause the device 200 to generate video using the image sensor 202 and the ISP 212. Although the example device 200 is described as including the memory 206, in some embodiments, the device 200 may not include the memory 206. For example, the memory may be external to the device 200, and the device 200 may be configured to access the memory.

[0062] The processor 204 may include one or more general-purpose processors capable of executing scripts or instructions (e.g., instructions 208) of one or more software programs stored in the memory 206. For example, the processor 204 may include one or more application processors configured to execute a camera application (or other suitable application for generating video) stored in the memory 206. When executing the camera application, the processor 204 may be configured to instruct the ISP 212 to perform one or more operations with reference to the image sensor 202. For example, the processor 204 may be configured to instruct the frame rate change and ISP configuration for processing image frames captured at the changed frame rate. Execution of instructions 208 external to the camera application may also cause the device 200 to perform any number of functions or operations. In some embodiments, in addition to the ability to execute software to cause the device 200 to perform many functions or operations, including the operations described herein, the processor 204 may also include an IC or other hardware.

[0063] In some embodiments, the processor 204 may include an encoder 210. The encoder 210 may be configured to encode the processed image frame stream from the ISP 212 to produce a video. The video may then be provided to the memory 206 for storage. In some embodiments, the video may be output to another suitable component, such as a display 214 or a graphics processor for display, an output component for output from the device 200, etc. In some embodiments, the encoder 210 may be implemented in software executed by the processor 204. For example, the encoder 210 may be implemented in instructions 208 stored in the memory 206. In some other embodiments, the encoder 210 may be implemented in hardware (e.g., one or more ICs) of the processor 204, or a combination of software and hardware. Although shown as being included in the processor 204, the encoder 210 may be implemented in any suitable component of the device 200 or external to the device 200. For example, the encoder 210 may be implemented in the ISP 212 or in a separate integrated circuit coupled to the ISP 212 to encode the processed image frame stream to generate a video. Although in Figure 2 The example device 200 is described as including the processor 204, however, in some implementations, the example device 200 may not include the processor 204. For example, the processor may be external to the device 200, and the device 200 may be configured to communicate with the processor.

[0064] Despite Figure 2 206, ISP 212, display 214, and / or I / O components 216 may be coupled to each other via one or more local buses (not shown for simplicity).

[0065] Display 214 can be any suitable display or screen that allows user interaction and / or presents items to the user (e.g., a preview of an image frame captured by image sensor 202). In some aspects, display 214 is a touch-sensitive display. I / O components 216 can be or include any suitable mechanism, interface, or device for receiving input (e.g., commands) from a user and providing output to the user. For example, I / O components 216 can include (but are not limited to) a graphical user interface (GUI), a keyboard, a mouse, a microphone and speakers, a squeezable bar, one or more buttons (e.g., a power button), a slider or switch, and the like.

[0066] As described above, ISP 212 may include one or more ISPs for processing image frames during the image processing pipeline. Also as described above, ISP 212 may be configured to process a subset of image frames for generating a preview stream. For example, ISP 212 may be configured to process each image frame at a specified interval from the sequence of image frames received from image sensor 202. The interval may be based on a frame rate for the preview stream compared to the frame rate used when the image frames were captured. For example, ISP 212 may also be configured to process all image frames for generating a video stream. For example, ISP 212 may be configured to process each image frame received from image sensor 202, and the processed image frames may be encoded (e.g., by encoder 210) to generate video. As described herein, ISP 212 may configure one or more of its filters (or other filters and components of the image processing pipeline) based on the frame rate variation indicated by image sensor 202.

[0067] As used herein, an interval may refer to an interval (in time or space) between objects. For example, processing an image frame at an interval of four image frames (also referred to as processing every fourth image frame) may refer to processing the first image frame of a sequence of image frames, processing the fifth image frame of the sequence of image frames (four image frames from the first image frame), processing the ninth image frame of the sequence of image frames (four image frames from the fifth image frame), and so on. In another example, accessing a memory at an interval of 16 bytes (also referred to as accessing every 16th byte) may refer to accessing byte N of the memory, accessing byte N+16 of the memory, accessing byte N+32 of the memory, and so on.

[0068] Figure 3A is a block diagram of an example ISP 300 for processing frames captured at different frame rates. The ISP 300 may be Figure 2 ISP 300 is configured to receive one or more image frame sequences captured by image sensor 302. Image sensor 302 may be Figure 2 ISP 300 can also be configured to output a first sequence of processed image frames for a preview stream and to output a second sequence of processed image frames for a video stream. In this manner, ISP 300 can be a single-input, multiple-output (SIMO) component, where the single input is a stream of image frames from image sensor 302 and the multiple outputs are a preview stream and a video stream of processed image frames. In some embodiments, the processed image frames for the preview stream are output to a first portion of the memory (e.g., Figure 2The processed image frames for the video stream are output to a second portion of the memory (memory 206 in the memory or a different memory coupled to the ISP 300 and the processor 304). The processor 304 may be Figure 2 2. In some embodiments, the processor 304 may access the memory to obtain processed image frames for the video stream, and the processor 304 may encode or further process the processed image frames. In some other embodiments, the ISP 300 may provide the processed image frames for the video stream directly to the processor 304. The first portion of the memory storing the processed image frames for the preview stream may be accessed by a graphics processor or other suitable component for displaying the preview. In some other embodiments, the ISP 300 may provide the processed image frames for the preview stream directly to the component for displaying the preview.

[0069] The ISP 300 includes a camera subsystem (CSID) 324 and an imaging front end (IFE) 326. In some embodiments, the CSID 324 includes a controller 306. The controller 306 receives a stream of image frames from the image sensor 302. For example, the controller 306 may receive a first sequence of image frames captured at a first frame rate, and the controller 306 may receive a second sequence of image frames captured at a second frame rate. Although not shown, the analog front end may convert analog image data for each image frame from the image sensor 302 into a digital format received by the CSID 324. The CSID 324 may process one or more image frame headers (or other image metadata) to determine the frame rate. If the frame rate changes, the CSID 324 may determine to configure a new frame rate for the ISP 300. In some embodiments, the CSID 324 includes the controller 306 to receive image frames and process the image frame headers (or other image metadata) to determine the frame rate. The controller 306 may also determine to configure a new frame rate for the ISP 300. Configuring the ISP 300 may include configuring one or more filters 310A-310K of the ISP 300. The filters 310A-310K may perform any suitable image processing function (e.g., color balancing, re-mosaicing or demosaicing, noise reduction, edge enhancement, color transformation, shading correction, geometry correction, etc.). Configuring the one or more filters 310A-310K may include configuring one or more parameters for the filters (e.g., thresholds, kernels, or other inputs that affect the filters) or configuring which filters are paused or active for processing image frames. Configuring the ISP 300 may also include configuring other components for processing image frames captured at the new frame rate. Although the filters 310A-310K are described as being included in the CSID 324, one or more filters 310A-310K may be included in the IFE 326 or the ISP 300 or another part of the image processing pipeline. In some embodiments, configuring the ISP 300 also includes configuring the ISP 300 to process (or queue) only a portion of the image frames for the preview stream. For example, ISP 300 may be configured to use image frames for the preview stream at predetermined intervals to compensate for the frame rate difference between the captured image frames and the preview stream.

[0070] In some embodiments, the preview frame rate can be static. For example, the preview frame rate can be maintained at 30 fps, while the frame rate of the image frame input stream can vary anywhere between 30 fps and 960 fps. If the input stream is captured at 30 fps, the preview stream can include an instance of every processed image frame (wherein the ISP 300 does not skip any image frames for the preview stream). If the input stream is captured at a rate higher than 30 fps, the preview stream may only include instances of a portion of the processed image frames (e.g., every other image frame when captured at 60 fps, every fourth image frame when captured at 120 fps, and so on). Configuring the controller 306 to configure one or more filters can include reducing the number of image frames to be processed or queued for the preview stream. In some embodiments, the controller 306 determines the interval between image frames in the stream to be processed for the preview stream. This interval can correspond to the ratio between the preview frame rate and the frame rate of the input stream. For example, if the preview frame rate is 30 fps and the frame rate of the input stream is 120 fps, the ratio is 1:4 (30 fps to 120 fps). In this manner, the controller 306 determines that the preview stream will include every fourth image frame processed by the ISP 300. The ISP 300 can therefore be configured to filter out the remaining image frames from the preview stream. In some embodiments, the ISP 300 includes (or is coupled to) a lookup table (LUT) 308. The LUT 308 can include a mapping of frame rates to intervals or ratios. In this manner, the controller 306 can obtain the intervals or ratios from the LUT 308 to configure which image frames to process or queue for the preview stream.

[0071] The ISP configuration can indicate what configuration to apply to the ISP 300 (or image processing pipeline) for a particular frame rate. For example, an ISP configuration associated with a 60 fps frame rate can indicate the configuration of filters 310A-310K, other components of the ISP 300 (or other parts of the image processing pipeline), or the interval used to process image frames for the preview stream. As described above, the ISP 300 can include (or be coupled to) a LUT 308 to store a mapping of frame rates to specific ISP configurations. For example, the image sensor 302 can be configured to capture image frames at multiple different frame rates. The LUT 308 can map at least one ISP configuration to each different frame rate. In an example embodiment, the image sensor 302 can adjust its frame rate between 30 fps, 60 fps, 120 fps, 240 fps, 480 fps, and 960 fps. The LUT 308 stores a first ISP configuration mapped to 30 fps, a second ISP configuration mapped to 60 fps, a third ISP configuration mapped to 120 fps, a fourth ISP configuration mapped to 240 fps, and so on. The controller 306 determines the frame rate from the header (or other metadata) of the received image frame, looks up the frame rate in the LUT 308, and obtains an ISP configuration that maps to the frame rate in the LUT 308. The controller 306 then configures the ISP 300 (including one or more filters 310A-310K and / or intervals for processing the preview stream) based on the obtained ISP configuration.

[0072] As described above, the image sensor 302 can be configured to adjust its frame rate without input from the ISP 300 or the processor 304, and the image sensor 302 can indicate a frame rate change to the ISP 300. In some examples, the image sensor 302 can indicate a frame rate change (e.g., indicating a new frame rate or a frame rate difference) in one or more image frame headers or other portions of the image frame metadata. The CSID 324 receives image frames from the image sensor 302. For example, the CSID 324 can receive a first sequence of image frames captured at a first frame rate, and the CSID 324 can receive a second sequence of image frames captured at a second frame rate. One or more image frames in the second sequence of image frames can indicate a frame rate change from the first frame rate to the second frame rate (e.g., a header indicating a new frame rate). In some other embodiments, the image sensor 302 can provide separate signaling to the ISP 300 to indicate a frame rate change or a new frame rate.

[0073] The IFE 326 can be configured to receive processed image frames from the CSID 324 and output the processed image frames for the preview stream and the video stream. In some embodiments, the IFE 326 can include two queues or buffers: one for processed image frames for the preview stream and one for processed image frames for the video stream. The IFE 326 can store an instance of each processed image frame in the queue for the video stream, and the IFE 326 can store instances of processed image frames in the stream of processed image frames in the queue for the preview stream at specified intervals. Configuring the ISP 300 can include configuring the IFE 326 to adjust the interval used to queue processed image frames for the preview stream. The IFE 326 can also be configured to output each processed image frame to a specified location in memory (e.g., a specified sector for starting storage of processed image frames). For example, the IFE 326 can output processed image frames for the preview stream to a first portion of memory and output processed image frames for the video stream to a second portion of memory.

[0074] In some other embodiments, the CSID 324 provides a first set of processed image frames for the preview stream and a second set of processed image frames for the video stream. The IFE 326 can be configured to output the processed image frames to an appropriate location in memory. In some other embodiments, the IFE 326 can provide the processed image frames to another appropriate component (e.g., the processor 304 for the video stream or the graphics processor for the preview stream). In some embodiments, the IFE 326 is further configured to process one or more image frames received from the CSID 324. For example, the IFE 326 can include one or more filters 310A-310K.

[0075] In some embodiments, CSID 324 and IFE 326 are included in a single ISP. In some other embodiments, ISP 300 may include multiple ISPs, with CSID 324 in one ISP and IFE in another ISP. In this manner, ISP 300 may be a single chip, multiple chips, a single system on chip (SOC), multiple SOCs, or other suitable configuration for image frame processing.

[0076] In some embodiments, the ISP 300 is configured to perform batch processing. The ISP 300 may not communicate with the processor 304 for every image frame. Batch processing may refer to processing multiple image frames between instances of communication with the processor 304. A batch may refer to a group of image frames processed between instances of communication with the processor 304. The batch size may refer to the number of image frames in the batch. The CSID 324 (e.g., the controller 306) may also configure when to communicate with the processor 304 based on the batch size. For example, when the frame rate is 30 fps, the ISP 300 and processor 304 may communicate once per frame. When the frame rate is 60 fps, the ISP 300 and processor 304 may communicate once every other frame. When the frame rate is 120 fps, the ISP 300 and processor 304 may communicate once every fourth frame. When the frame rate is 240 fps, the ISP 300 and processor 304 may communicate once every eighth frame. When the frame rate is 480 fps, the ISP 300 and processor 304 can communicate once every sixteenth frame. When the frame rate is 960 fps, the ISP 300 and processor 304 can communicate once every thirty-second frame. In this way, as the frame rate of the image sensor 302 changes, the time period during which the ISP 300 and processor 304 communicate can remain relatively static. For example, if communication occurs for every image frame at a frame rate of 30 fps, the interval between communications may be approximately 16.66 ms. Adjusting the batch size can allow the interval to remain approximately 16.66 ms as the frame rate of the image sensor 302 changes. In some embodiments, the LUT 308 maps the frame rate to the batch size. In this way, the CSID 324 can configure the ISP 300 to communicate with the processor 304 based on the batch size obtained from the LUT 308. For example, the controller 306 can configure when it communicates with the processor 304.

[0077] The interval or ratio of image frames used to process or queue for preview streams can be based on the batch size. For example, each frame can be communicated to the processor 304 based on a frame rate of 30fps, and the preview can be 30fps. In this way, the interval or ratio in the LUT 308 can be the batch size. In some other examples, the frame rate of the preview (which can be measured in frames) can be a fraction or multiple (e.g., 100 frames per second) of the communication rate between the ISP 300 and the processor 304. 1 / 4、 1 / 2, 2, 4, etc.). In this way, CSID 324 can determine the interval based on the batch size obtained from LUT 308.

[0078] The ISP 300 can also be configured to indicate one or more capture characteristics to the image sensor 302. For example, the ISP 300 can indicate to the image sensor 302 a requested frame rate change initiated by the processor 304. In some embodiments, the processor 304 can indicate the frame change (and the ISP configuration) to the controller 306. In some embodiments, the ISP configuration can be determined using the LUT 308 based on the frame rate indicated by the processor 304. The controller 306 can indicate the frame rate change to the image sensor 302 and configure one or more filters of the ISP 300 for processing image frames from the image sensor 302. As described above, configuring the one or more filters of the ISP 300 can include configuring one or more filters 310A-310K, configuring the IFE 326 to provide a subset of processed image frames for a preview stream, configuring the CSID 324 (e.g., the controller 306) to communicate with the processor 304 at regular intervals, or configuring other components of the ISP 300. Figure 3A A simplified block diagram of ISP 300 is provided to illustrate various aspects of the present disclosure. ISP 300 may include other components or different configurations of components for performing various aspects of the present disclosure, and ISP 300 and the present disclosure are not limited to the following. Figure 3A In one example, although LUT 308 is described as being included in ISP 300, the LUT may be stored outside of and coupled to ISP 300. In another example, although one CSID and one IFE are described, ISP 300 may include multiple CSIDs and / or multiple IFEs. Figure 3B An example ISP including multiple CSIDs and IFEs is described in .

[0079] Figure 3B is a block diagram of an exemplary ISP 350 for processing frames captured at different frame rates. ISP 350 may be similar to ISP 300, except that ISP 350 may include a first CSID 374A and a first IFE 376A and a second CSID 374B and a second IFE 376B. Image sensor 352 may be similar to Figure 3A The image sensor 302 in may be Figure 2 The processor 354 may be similar to the image sensor 202 of FIG. Figure 3A The processor 304 in . And can be Figure 2 An exemplary embodiment of the processor 204 in FIG.

[0080] As the frame rate increases, a single CSID (or IFE) may not have enough image frame processing rate. For example, the image sensor 352 can be configured to capture image frames at 480fps (or higher). A frame rate of 480fps corresponds to approximately 2.1ms between frames. However, the CSID and IFE may need more than 2.1ms to process the image frames. In some embodiments, the ISP 350 includes multiple CSIDs and IFEs (e.g., CSID 374A and IFE 376A and CSID 374B and IFE 376B). CSID 374A and IFE 376A can process a first portion of image frames from the image sensor 352, while CSID 374B and IFE 376B can process a second portion of image frames from the image sensor 352. For example, the first CSID 374A can process every other image frame (e.g., by applying one or more filters 360A-360K), and the second CSID 374B can process the remaining image frames (e.g., by applying one or more filters 360P-360Z). The IFE 376A provides at least some of the first portion of processed image frames for the preview stream and all of the first portion of processed image frames for the video stream. The IFE 376B provides at least some of the second portion of processed image frames for the preview stream and all of the second portion of processed image frames for the video stream. For example, each IFE 376A and 376B can be configured to address a specific sector of memory to ensure that processed frames are stored for both the preview stream and the video stream. In this way, image frames can be processed simultaneously to increase the rate at which the ISP 350 can process image frames.

[0081] In some embodiments, ISP 350 may include a buffer 364 or other suitable component for temporarily storing image frames from image sensor 352. Each of controllers 356A and 356B can then retrieve the appropriate image frame to be processed by the corresponding CSID 374A or CSID 374B. The ISP configuration stored in LUT 358 can indicate how to split the frame for processing by multiple components. In this way, controller 356A (or controller 356B) can configure buffer 364 or other components to route image frames to CSIDs 374A and 374B for processing.

[0082] CSID 374A and 374B and IFE 376A and 376B can be used in any suitable manner to process image frames at different frame rates. For example, when the frame rate is lower than 480fps (e.g., 30fps, 60fps, 120fps, or 240fps), CSID 374A and IFE 376A may be able to process all image frames. However, when the frame rate is 480fps or higher, CSID 374A and IFE 376A may have difficulty processing all image frames. In some embodiments, when the frame rate is lower than a threshold frame rate (such as lower than 480fps), ISP 350 may use only one CSID and one IFE (e.g., CSID 374A and IFE 376A). In this way, configuring ISP 350 may also include activating or deactivating CSID 374B and IFE 376B based on the frame rate. For example, controller 356A can instruct controller 356B to prepare CSID 374B and IFE 376B to process image frames. In another example, a single controller 356A can be used to control and configure two CSIDs and two IFEs. In some other embodiments, CSIDs 374A and 374B can alternate image frames for processing regardless of the frame rate. In this way, the CSID and IFE can be active for all frame rates.

[0083] ISP 350 may also be configured to indicate one or more capture characteristics to image sensor 352. For example, ISP 350 may indicate a requested frame rate change initiated by processor 354 to image sensor 352. In some embodiments, processor 354 may indicate the frame change (and ISP configuration) to ISP 350. In some embodiments, the ISP configuration may be determined using LUT 358 based on the frame rate indicated by processor 354. Although Figure 3B The processor 354 is described as being in communication with the controller 356B, but the processor 354 may provide instructions to any suitable component, such as the controller 356A. Figure 3B Image sensor 352 is depicted as receiving instructions from controller 356B, but image sensor 352 may receive instructions from any suitable component, such as controller 356A.

[0084] As in Figure 3A and Figure 3B As shown in the example description of the ISP, Figure 2 The ISP 212 in FIG. 1 may include various configurations to process image frames from the image sensor 202 and communicate with the processor 204. Figure 3A The following examples are described for various aspects of the present disclosure, but any suitable configuration of an ISP (including Figure 3B Example configuration of ISP 350 in ).

[0085] As described above, a frame rate change or new frame rate can be indicated in an image frame header or other suitable metadata. In some embodiments, when an image frame stream has a changed frame rate, image frames captured at the new frame rate can be associated with a different stream or session identifier (ID) (referred to herein as a stream ID) than image frames previously captured at the old frame rate. In this way, the ISP 300 can associate a first sequence of image frames captured at a first frame rate with a first stream ID and associate a second sequence of image frames captured at a second frame rate with a second stream ID. In this way, the ISP 300 can manage and accurately describe frame rate changes as they occur in an image frame stream.

[0086] Figure 4 4 is a description of example contents of an image frame 400 provided by an image sensor to an ISP. Image frame 400 begins with a start of frame (SOF) 402. At least a portion of SOF 402 may include a data ID field 424, a word count (WC) field 426 (including a first portion WC 1 426A and a second portion WC 1 426B), and an error correction code (ECC) 428. Data ID field 424 may include a bit sequence or code indicating the start of image frame 400. In some embodiments, WC 1 426A includes a frame count for image frame 400. WC 2 426B may include zero padding or other unused values. In some other embodiments, WC 2 426B may also include a frame count (e.g., a second instance of a frame count or combined with WC 1 426A for a single frame count). In some embodiments, the WC field 426 includes 16 bits (where subfield WC 1 426A includes bits 0-7 and WC 2 426B includes bits 8-15). Bits 0-7 may include a frame count. Bits 8-15 may contain fixed values (e.g., zero or a known bit pattern). ECC 428 may include an error correction code used to determine if there are any errors in the received image frame 400. In some embodiments, the SOF 402 may include other fields and may be used in conjunction with the SOF 402. Figure 4 Only a portion of the SOF 402 may be shown.

[0087] The image frame may also include a packet header. In the depicted image frame 400, the packet header may include a video coding (VC) field 404 and a dequantization (or decoding) table (DT) 406 (e.g., including DT 406A, DT 406B, DT 406C, and DT 406D). VC field 404 may indicate the format in which the image sensor provides the image frame to the ISP. DT 406A may be used to decode the boundary (EBD 408) of the image frame portion including image sensor pixel information 410. DT 406B may be used to decode image sensor pixel information 410. DT 406C may be used to decode gyroscope (or other sensor) data 412. DT 406D may be used to decode auto-hint marker data 414.

[0088] For portions of image frame 400 that include non-zero information (e.g., fields 404-414), these portions may include a packet footer (PF) 416 (e.g., PF 416A, PF 416B, PF 416C, and PF 416D) appended to their ends. For example, EBD 408 may include PF 416A, image sensor pixel information 410 may include PF 416B, gyroscope data 412 may include PF 416C, and autocue flag data 414 may include PF 416D. PF 416 may indicate the end of a row of fields or other detailed descriptions of fields from the remainder of image frame 400. The remainder of image frame 400 between SOF 402 and end of frame (EOF) 418 (outside of fields 404-416) may include row blanking 420. Row blanking 420 may be zero padding or other data to fill unused portions of image frame 400. As shown, there may be a gap 422 between the image frame 400 and the next image frame in the stream.

[0089] In some embodiments, the image sensor assigns a stream ID to an image frame stream. To indicate a change in frame rate, the image sensor may assign a different stream ID to image frames captured at the new frame rate. For example, a stream may include a first sequence of image frames captured at a first frame rate and a second sequence of image frames captured at a second frame rate. The first sequence of image frames may be associated with a first stream ID, and the second sequence of image frames may be associated with a second stream ID. The stream ID (e.g., a value or format) may correspond to the frame rate. For example, a portion of the stream ID may include a plurality of bits indicating the frame rate.

[0090] In some embodiments, a stream ID associated with the image frame 400 is included in a header, an EBD 408, or another suitable portion of the image frame 400. For example, the stream ID may be included in the EBD 408. The format of the stream ID in the EBD 408 (e.g., length, order of bits, information indicated in the stream ID, etc.) may be programmable. Additionally or alternatively, the location of the stream ID in the EBD 408 may be programmable. The image sensor and the ISP may be configured to use any suitable format and location of the stream ID for the image sensor to indicate the frame rate to the ISP.

[0091] In some embodiments of programmable stream IDs, the ISP may include multiple programmable registers to be used to determine the location of the stream ID in the image frame 400 and the format of the stream ID. One register may be a stream ID length register. In some embodiments, the stream ID length register is 2 bits and indicates the number of bytes in the stream ID. In this way, the stream ID length register may indicate up to four different byte lengths. For example, 00 may indicate that the stream ID is not included in the EBD 408 (a length of 0 bytes), 01 may indicate that the stream ID is 1 byte long, 10 may indicate that the stream ID is 2 bytes long, and 11 may indicate that the stream ID is 3 bytes long. In some embodiments, 11 may be reserved to indicate something other than a byte length (e.g., the stream ID is in a second position other than in the EBD).

[0092] Other registers may include a Stream ID x Offset register or a Stream ID y Offset register to indicate the location of the stream ID in the EBD 408. The EBD 408 includes one or more rows, each containing a plurality of bits. As shown, the number of bits per row may be the same as the number of bits per row of image sensor pixel information 410. However, the number of bits per row may be any suitable number. The Stream ID x Offset register may indicate the offset of the stream ID from the beginning of the row in the EBD 408. This offset may be measured in bytes. In some embodiments, the Stream ID x Offset register is 16 bits and indicates the number of bytes by which the stream ID is offset from the beginning of the row in the EBD 408. The EBD 408 may consist of two or more rows. The Stream ID y Offset register may indicate the offset of the stream ID from the first row in the EBD 408. This offset may be measured in rows. In some embodiments, the Stream ID y Offset register is 6 bits and indicates the number of rows by which the stream ID is offset from the first row in the EBD 408. Although the stream ID x-offset and y-offset registers are described as indicating the location of the stream ID in the EBD 408, in some embodiments, the stream ID x-offset and y-offset registers can be used to indicate the location of the stream ID in the image sensor pixel information 410. For example, the image sensor can place the stream ID in a location in the image sensor pixel information 410, and the ISP can use the registers to determine the location of the stream ID. In some other embodiments, the registers can be used to indicate the location of the stream ID in other parts of the image frame 400.

[0093] The stream ID may be associated with the VC ID included in the VC field 404. The VC ID may be assigned by the image sensor based on the video encoding used or other characteristics of the image frame capture (e.g., time, number of sessions, frame rate, etc.). Figure 3A , ISP 300 can decode the input image frame 400 to identify the VCID (in the VC field 404) and the stream ID (e.g., based on the number of registers indicating the location and length of the stream ID). LUT 308 can map at least a portion of the VC ID and / or the stream ID to the ISP configuration and batch size. CSID 324 can use the VC ID and / or the stream ID to look up the ISP configuration in LUT 308. For example, if the VC ID or the stream ID changes, CSID 324 can obtain the new ISP configuration from LUT 308 and configure ISP 300 based on the new ISP configuration.

[0094] In some other embodiments, the VCID can be used to determine the stream ID. For example, an image sensor can change the VC ID based on changes in frame rate. In this way, the VC ID value can be associated with the frame rate. Changes to the VC ID can be predetermined based on the frame rate, so that the stream ID can be determined from the VC ID. In some embodiments, the LUT 308 can include a mapping of the VCID to the stream ID (or frame rate) for the ISP 300 to determine the stream ID or frame rate from the VC ID.

[0095] In some embodiments, the image processing pipeline (including ISP 300) can process image frames from multiple image sensors. Input image frames from different image sensors may therefore include different VCIDs. ISP 300 can use VCIDs to manage image frames after processing and separate image frames into individual streams (e.g., storing processed image frames in separate areas of a memory for different streams). For example, ISP 300 can receive two streams with two different VCIDs, and the image processing pipeline can support processing different image frames with two different VC IDs. For example, image frames from two streams can be interleaved. IFE 326 can include two buffers or queues associated with two different VC IDs, and processed image frames can be placed in corresponding buffers. IFE 326 can then output the buffered image frames from the buffers associated with the streams to corresponding locations in the memory.

[0096] Back to reference Figure 2 ISP 212 may not communicate with processor 204 for every image frame. For example, if the frame rate is a multiple of 30 fps, the batch of image frames to be processed by ISP 212 may be at similar multiples of 30 fps between communications with processor 204. In response to when the communication occurs, processor 204 may indicate the ISP configuration to indicate the frame rate change. Figure 3A308 in ISP 200 to configure the ISP 212. However, the ISP configuration from the processor 204 may be different from the stored ISP configuration. For example, if the device 200 is in a power save mode, the processor 204 may indicate that the video will use a lower frame rate than the frame rate used by the image sensor 202 to capture the image frames. In another example, the processor 204 may indicate a difference to one or more filters in the image processing pipeline. In some embodiments, the ISP 212 compares the received ISP configuration with the stored ISP configuration to identify any differences. If one or more differences are identified, the ISP 212 may adjust one or more filters based on the identified differences. The comparison can be performed while continuing to process the image frames. In this way, the ISP 212 does not need to pause processing and does not interrupt the final video or preview.

[0097] As described herein, an ISP can be configured to manage its resources (e.g., its filters, one or more CSIDs, one or more IFEs, etc.) based on indicated frame rate changes from an image sensor. Although some examples of image frame formats, ISP configurations, and device configurations are described, various aspects of the present disclosure may be performed using any suitable device or component configurations and information formats. Reference is made to the following exemplary methods for an ISP to manage its resources. Figure 2 The example device 200 and Figure 3A An exemplary ISP 300 is described. Figure 2 and Figure 3A Utilized to teach various aspects of the present disclosure, but the operations of the following methods are not limited to being performed by a particular configuration of the device 200 or the ISP 300 .

[0098] Figure 5 FIG1 is an illustrative flow diagram describing example operations 500 for image frame processing for video. The image sensor 202 is configured to adjust its frame rate without intervention from the ISP 212 or the processor 204. The ISP 212 is configured to adjust processing of image frames from the image sensor 202 based on frame rate variations without intervention from the processor 204.

[0099] At 502, the ISP 212 receives a first sequence of image frames from the image sensor 202. The first sequence of image frames is captured at a first frame rate. At 504, the ISP 212 processes each image frame of the first sequence of image frames at the first frame rate. In some embodiments, the ISP 212 has been configured to process image frames captured at the first frame rate. For example, the ISP 212 can be pre-configured based on a frame rate previously indicated by the processor 204 or previously indicated by the image sensor 202. In this manner, the ISP 212 can process each image frame in the first sequence of image frames (504). In some embodiments, the ISP 212 processes each image frame of the first sequence of image frames for generating a video stream. The ISP 212 can also process at least a subset of the first sequence of image frames for generating a preview stream.

[0100] At 506, the ISP 212 receives an indication of a frame rate change of the image sensor 202 from the first frame rate to the second frame rate from the image sensor 202. The frame rate may be indicated in metadata (e.g., a header or EBD) of the image frame. In some embodiments, one or more image frames captured at the second frame rate may include an indication of the frame rate change to the second frame rate. For example, a stream ID or VCID in the image frame may be used to indicate the frame rate at which the image frame was captured. The ISP 212 may identify the stream ID or VCID in the image frame, and the stream ID or VCID may be used to obtain the ISP configuration. For example, the CSID 324 or the controller 306 ( Figure 3A ) can identify the stream ID or VCID and use the stream ID or VCID to obtain the ISP configuration from the LUT 308. The ISP configuration can indicate the frame rate or otherwise allow the ISP 212 to process image frames at a new frame rate. In some other embodiments, the image sensor 202 can indicate the frame rate to the ISP 212 via signaling external to the image frame.

[0101] At 508, in response to receiving an indication of a change in frame rate from the image sensor 202, the ISP 212 configures one or more filters of the ISP 212 to process image frames from the image sensor 202. In some embodiments, the ISP 212 may configure the one or more filters of the ISP 212 based on the ISP configuration. For example, the ISP 300 (e.g., the CSID 324 or the controller 306) may configure one or more filters 310A-310K. The ISP may also configure a batch size for when the ISP communicates with the processor (e.g., the ISP communicates with the processor once for each batch of image frames). The ISP may further configure an interval for processing image frames for a preview stream. The ISP may also configure other components, including the CSID, the IFE, an output module, and the like. For example, if the ISP includes a second CSID and a second IFE, the ISP may activate or deactivate the second CSID and the second IFE based on the frame rate of the received image frames.

[0102] At 510, ISP 212 receives a second sequence of image frames from image sensor 202 at a second frame rate. In some embodiments, the indication received in step 506 is included in one or more image frames included in the second sequence of image frames, and ISP 212 processes the second sequence of image frames (including the one or more image frames from which the frame rate can be determined) (512). For example, ISP 212 is configured in step 508 to process the image frames at the second frame rate. All image frames of the second sequence of image frames can be processed to generate a video stream, and at least a subset of the image frames of the second sequence of image frames can be processed to generate a preview stream.

[0103] Although not shown, the processed image frames of the video stream can be encoded to generate a video. In some embodiments, the image frames can be encoded at the ISP 212. In some other embodiments, the image frames can be provided by the ISP 212 to the processor 204 (e.g., via a memory accessible by both the ISP 212 and the processor 204) for encoding by the encoder 210. The video can include different parts at different frame rates. The processed image frames of the preview stream can be provided to the display 214, the processor 204, the graphics processor, or other suitable component (e.g., via a memory accessible by both the ISP 212 and a suitable component) for displaying a preview to the device user. The preview can use a static frame rate (with an interval for processing image frames that adjusts based on frame rate changes).

[0104] As described above, ISP 212 may not communicate with processor 204 for every image frame. For example, when the frame rate is greater than 30 fps, ISP 212 may communicate with processor 204 once during processing of a batch of image frames (batch size greater than one). In this manner, ISP 212 may process one or more image frames at a new frame rate before transmitting the new frame rate to processor 204.

[0105] Figure 6 6 is an illustrative flow chart describing example operations 600 for configuring communications between the image signal processor 212 and the processor 204 based on a batch size. At 602, the ISP 212 determines a batch size based on an indication of a frame rate change from an image sensor. The image frame may include an indication of the new frame rate, and the ISP 212 may use this indication to determine the batch size. For example, the ISP 300 (e.g., the CSID 324 or the controller 306) may determine the batch size based on an ISP configuration from the LUT 308 or based on the new frame rate indicated by the ISP configuration. In some embodiments, the ISP 212 may identify the batch size based on a VC ID or stream ID in an image frame received from the image sensor 202 (604). For example, the ISP 300 may identify the VC ID or stream ID from the image frame and determine the batch size from the ISP configuration obtained from the LUT 308 based on the VC ID or stream ID.

[0106] At 606, ISP 212 configures one or more filters based on the frame rate change. This step may be similar to Figure 5 508 in step 508. For example, the ISP 300 may obtain the ISP configuration from the LUT 308. The ISP configuration may include a batch size. The ISP 300 may then use the ISP configuration (including the batch size) to configure one or more filters (e.g., one or more filters 310A-310K or other suitable filters or components). When configuring the one or more filters, the ISP 212 may configure itself to communicate with the processor 204 once during a batch of image frames of the batch size (608). For example, the communication with the processor 204 to indicate the frame rate, frames buffered for output, any errors, or other information may be at intervals of at least 16.66 ms (e.g., corresponding to a 30 fps frame rate) or other suitable intervals. When the frame rate increases such that the time between frames is less than the interval, the determined batch size (e.g., from Figure 3AThe LUT 308 in the ISP 212 may indicate that the number of image frames to be processed by the ISP 212 at each communication with the processor 204 is greater than 1. For example, 60 fps may correspond to a batch size of 2, 120 fps may correspond to a batch size of 4, and so on. In this way, the rate at which the ISP 212 communicates with the processor 204 may be consistent. In some embodiments, the interval may correspond to the minimum time required to communicate with the processor 204 (e.g., based on software or hardware latency). The batch size may ensure that the time between communications is greater than this minimum time.

[0107] At 610, the ISP 212 communicates with the processor 204 once during each batch of image frames. In some embodiments, after configuring one or more filters of the ISP 212, the ISP 212 may communicate the frame rate change to the processor 204. In some embodiments, the communication may be an interrupt request (IRQ). However, the communication may be in any suitable format for the ISP 212 to indicate the frame rate and other information to the processor 204. The processor may respond to the IRQ with the ISP configuration information or another suitable acknowledgement of the request.

[0108] As described above, the ISP 212 can be configured to generate a preview stream and a video stream. In some embodiments, the video stream can include all image frames captured by the image sensor 202 and processed by the ISP 212. The preview stream can include a subset of the image frames captured by the image sensor 202 and processed by the ISP 212. For example, the preview stream may utilize a frame rate lower than the frame rate of the image sensor. To generate a preview stream from a sequence of image frames captured at a higher frame rate, the ISP 212 can process only a subset of the image frames in the sequence (and discard other image frames for the preview stream). In some embodiments, the ISP 212 processes intervals of image frames (e.g., every other image frame, every fourth image frame, etc.) to generate a preview stream at a desired frame rate. The image frame stream from the image sensor 202 can include a first sequence of image frames captured at a first frame rate and a second sequence of image frames captured at a second frame rate (where the image sensor 202 indicates a frame rate change). The ISP 212 can configure a subset of image frames to be processed for each sequence. In this way, ISP 212 can keep the frame rate of the preview stream consistent across multiple sequences captured at different frame rates.

[0109] Figure 7 FIG2 is an illustrative flow diagram describing example operations 700 for generating a preview stream. Image sensor 202 provides a stream of image frames to ISP 212, wherein the frame rate is varied during capture of the stream of image frames. In this manner, image sensor 202 provides a first sequence of image frames captured at a first frame rate and a second sequence of image frames captured at a second frame rate.

[0110] At 702, ISP 212 determines a subset of a sequence of captured image frames to be processed for a preview stream at a frame rate. For example, ISP 212 may be pre-configured to process a first subset of image frames for a first sequence of image frames captured at a first frame rate. ISP 212 then begins receiving a second sequence of image frames captured at a second frame rate (wherein image sensor 202 indicates a frame rate change from the first frame rate to the second frame rate). ISP 212 may therefore adjust the subset of image frames to be processed for the second sequence of image frames in response to the frame rate change. For example, ISP 212 may adjust the subset to be processed to maintain a consistent frame rate for the preview stream across sequences of image frames captured at different frame rates.

[0111] In some embodiments, the ISP 212 may determine an interval for processing image frames for the preview stream based on the frame rate of the image frame sequence (704). For example, if the frame rate of the image frame sequence is 240 fps and the frame rate of the preview stream is 30 fps, the ISP 212 may determine the interval for processing to be every eighth image frame (30 / 240). In some embodiments, the ISP 300 ( Figure 3A ) determines a VC ID or stream ID from one or more image frames in the sequence and determines the interval based on the LUT 308 using the VCID or stream ID mapped to the interval.

[0112] In some embodiments, ISP 212 can determine the interval (706) based on the batch size. The interval can correspond to the batch size of the frame rate. For example, the frame rate of the preview stream and the interval used for communicating with processor 204 can be static relative to each other. In this way, ISP 212 can determine which image frames in the processing sequence to use for the preview stream based on the batch size used to determine when to communicate with processor 204. Figure 3A The LUT 308 in the may include a mapping of a frame rate indication (e.g., a VCID or stream ID) to a batch size, and the batch size may be mapped or otherwise correspond to a particular interval. For example, if communication with the processor 204 is approximately every 16.66 ms (corresponding to a 30 fps frame rate) and the frame rate of the preview stream is 30 fps, then the batch size is the same as the interval. For example, if the image sensor captures image frames at 120 fps, then the batch size may be four frames, and the interval for processing image frames for the preview stream may be every fourth frame.

[0113] In response to determining a subset of the sequence of image frames to be processed, the ISP 212 is configured to process the subset of image frames for the preview stream (708). In some embodiments, the ISP 212 is configured to process an interval of image frames for the preview stream (710). In some examples, the ISP 212 may process all image frames for the video stream. The ISP 212 may then output an instance of each processed image frame at the interval to generate the preview stream. For example, if the frame rate of the input frames from the image sensor 202 is 60 fps and the frame rate of the preview is 30 fps, the ISP 212 may generate a video stream including a processed image frame for each received image frame. The ISP 212 may also generate a preview stream including an instance of the first processed image frame, the third processed image frame, and so on, skipping every other image frame to compensate for the frame rate difference between the input image frames from the image sensor 202 and the preview. In this way, there can be two instances of the first image frame (one for the video stream and one for the preview stream), there can be one instance of the second image frame (one only for the video stream), there can be two instances of the third image frame, and so on.

[0114] In some embodiments, the IFE 326 is configured by the ISP 300 to generate multiple instances of processed image frames for the appropriate streams. For example, the IFE 326 may receive processed image frames from the CSID 324 and store an instance of each processed image frame in a buffer for the video stream. The IFE 326 may also store instances of a subset of processed image frames in a buffer for the preview stream. The ISP 300 may configure which instances to generate or store for the preview stream based on a determined interval or a determined batch size from the ISP configuration. The IFE 326 may also include one or more filters to be applied. For example, the IFE 326 may include one or more filters to be applied to image frames for the video stream to improve image quality. However, to improve preview latency, the IFE 326 may not apply one or more filters to image frames for the preview stream. In this manner, the IFE 326 may be configured to output processed image frames for the preview stream before outputting corresponding processed image frames for the video stream.

[0115] After ISP 212 configures itself to process a subset of the sequence of image frames, ISP 212 processes the subset of image frames for the preview stream (712). In some embodiments, ISP 212 may process intervals of image frames for the preview stream (714). For example, IFE 326 may generate instances of processed image frames for the preview stream to output a preview stream and a video stream. The preview stream including the processed image frames may be provided to display 214, processor 204, graphics processor, or other suitable component (e.g., via a memory accessible to both ISP 212 and the component) to display the preview to the user.

[0116] Various techniques are described for an ISP to manage its resources based on frame rate variations indicated by an image sensor. As previously described, the ISP does not require intervention from an application processor to manage its resources based on frame rate variations. Unless specifically described as being implemented in a particular manner, the techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be implemented at least in part by a non-transitory processor-readable storage medium (e.g., a processor-readable medium) comprising instructions 208. Figure 2 The instructions 208, when executed by the ISP 212, the processor 104, or any other suitable component, cause the device 200 to perform one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

[0117] Non-transitory processor-readable storage media may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, other known storage media, etc. Additionally or alternatively, the techniques may be implemented at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer or other processor.

[0118] The various illustrative logical blocks, modules, circuits, and instructions described in connection with the embodiments disclosed herein may be implemented by, for example, Figure 2The example device 200 of the present invention is executed by one or more processors of the processor 204 or ISP 212. Such processors may include, but are not limited to, one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), application-specific instruction set processors (ASIPs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. The term "processor" as used herein may refer to any of the above structures or any other structures suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within a dedicated software module or hardware module configured as described herein. Moreover, the techniques may be fully implemented in one or more circuits or logic elements. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration.

[0119] As described above, although the present disclosure shows illustrative aspects, it should be noted that various changes and modifications may be made herein without departing from the scope of the appended claims. For example, in the above examples, how to configure the filter and determine the batch size of the image frame is described as being based on accessing LUT 308 or LUT 358 ( Figure 3A and Figure 3B ). However, in some other embodiments, the ISP may determine the batch size or configure the filter based on the equation and the frame rate. Any other suitable component may be used to determine the batch size or how to configure the filter. Likewise, the ISP is not limited to using a LUT. Additionally, unless expressly stated otherwise, the functions, steps, or actions of the method claims according to the various aspects described herein need not be performed in any particular order. Furthermore, although elements may be described or claimed in the singular, the plural is contemplated unless a limitation to the singular is expressly stated. Accordingly, the present disclosure is not limited to the illustrated examples and encompasses any component for performing the functionality described herein in the various aspects of the present disclosure.

Claims

1. A method for digital image processing, comprising: receiving, by an image signal processor, a first sequence of image frames from an image sensor at a first frame rate; processing, by the image signal processor, each image frame in the first sequence of image frames at the first frame rate; receiving, by the image signal processor, from the image sensor an indication of a change in frame rate of the image sensor from the first frame rate to a second frame rate; in response to receiving an indication of the frame rate change from the image sensor, configuring one or more filters of the image signal processor to process image frames from the image sensor based on the frame rate change; receiving, by the image signal processor, a second sequence of image frames from the image sensor at the second frame rate; as well as Each image frame of the second sequence of image frames is processed by the image signal processor at the second frame rate.

2. The method according to claim 1, wherein The indication of the frame rate change is included in metadata of image frames subsequent to the first sequence of image frames from the image sensor.

3. The method of claim 1 , further comprising: A batch count for the second sequence of image frames is indicated by the image signal processor, wherein the number of image frames indicated by the batch count is determined by the image signal processor based on the received indication of the frame rate change.

4. The method according to claim 3, wherein: The number of image frames in the batch is based on a frame rate associated with the batch.

5. The method according to claim 1, wherein The image signal processor indicates the frame rate change to an application processor after configuring the one or more filters.

6. The method of claim 1 , further comprising: processing, by the image signal processor, a first interval of image frames of the first sequence of image frames to generate a first partial preview stream, wherein the first interval is based on the first frame rate; processing, by the image signal processor, a second interval of image frames of the second sequence of image frames to generate a second partial preview stream, wherein the second interval is based on the second frame rate; generating, by the image signal processor, a video stream comprising a processed first image frame sequence and a processed second image frame sequence; outputting, by the image signal processor, the preview stream for display; and The video stream is output by the image signal processor.

7. The method according to claim 6, wherein: The preview stream is at a fixed frame rate, and the video stream is at a variable frame rate.

8. The method of claim 6, further comprising encoding the first image frame sequence and the second image frame sequence by an encoder to generate the video stream.

9. The method of claim 1 , further comprising: A lookup table is stored by a memory, the lookup table mapping different frame rates to settings for configuring the one or more filters, wherein the image signal processor uses the lookup table to configure the one or more filters based on the frame rate change.

10. The method of claim 9, wherein: The lookup table also maps different frame rates to the number of image frames per batch for communication between the image signal processor and the application processor.

11. A device for digital image processing, comprising: An image signal processor configured to: receiving a first sequence of image frames from an image sensor at a first frame rate; processing each image frame in the first sequence of image frames at the first frame rate; receiving, from the image sensor, an indication of a change in frame rate of the image sensor from the first frame rate to a second frame rate; in response to receiving an indication of the frame rate change from the image sensor, configuring one or more filters of the image signal processor to process image frames from the image sensor based on the frame rate change; receiving a second sequence of image frames from the image sensor at the second frame rate; and Each image frame of the second sequence of image frames is processed at the second frame rate.

12. The apparatus of claim 11, wherein: The indication of the frame rate change is included in metadata of image frames subsequent to the first sequence of image frames from the image sensor.

13. The apparatus of claim 11, wherein: The image signal processor is further configured to: A batch count for the second sequence of image frames is indicated by the image signal processor, wherein the number of image frames indicated by the batch count is determined by the image signal processor based on the received indication of the frame rate change.

14. The apparatus of claim 13, wherein: The number of image frames in the batch is based on a frame rate associated with the batch.

15. The apparatus of claim 13, wherein: The image signal processor is further configured to indicate the frame rate change to an application processor after configuring the one or more filters.

16. The apparatus of claim 11, wherein: The image signal processor is further configured to: processing a first interval of image frames of the first sequence of image frames to generate a first partial preview stream, wherein the first interval is based on the first frame rate; processing a second interval of image frames of the second sequence of image frames to generate a second partial preview stream, wherein the second interval is based on the second frame rate; generating a video stream comprising a processed first sequence of image frames and a processed second sequence of image frames; outputting the preview stream for display; and Output the video stream.

17. The apparatus of claim 16, wherein: The preview stream is at a fixed frame rate, and the video stream is at a variable frame rate.

18. The apparatus of claim 16, further comprising an encoder to encode the first sequence of image frames and the second sequence of image frames to generate the video stream.

19. The apparatus of claim 16, further comprising a display to display the preview stream.

20. The device of claim 13, further comprising an application processor to execute a camera application associated with the first sequence of image frames and the second sequence of image frames.

21. The apparatus of claim 11, further comprising: A memory is provided to store a lookup table that maps different frame rates to settings for configuring the one or more filters, wherein the image signal processor is further configured to configure the one or more filters using the lookup table based on the frame rate change.

22. The apparatus of claim 21, wherein: The lookup table also maps different frame rates to the number of image frames per batch for communication between the image signal processor and the application processor.

23. The apparatus of claim 11, further comprising the image sensor configured to: capturing the first sequence of image frames; capturing the second sequence of image frames; and The frame rate change is indicated to the image signal processor.

24. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a device, cause the device to: receiving, by an image signal processor, a first sequence of image frames from an image sensor at a first frame rate; processing, by the image signal processor, each image frame in the first sequence of image frames at the first frame rate; receiving, by the image signal processor, from the image sensor an indication of a change in frame rate of the image sensor from the first frame rate to a second frame rate; in response to receiving an indication of the frame rate change from the image sensor, configuring one or more filters of the image signal processor to process image frames from the image sensor based on the frame rate change; receiving, by the image signal processor, a second sequence of image frames from the image sensor at the second frame rate; as well as Each image frame of the second sequence of image frames is processed by the image signal processor at the second frame rate.

25. The computer-readable medium of claim 24, wherein: The indication of the frame rate change is included in metadata of image frames subsequent to the first sequence of image frames from the image sensor.

26. The computer-readable medium of claim 24, wherein: Execution of the instructions further causes the device to: A batch count for the second sequence of image frames is indicated by the image signal processor, wherein the number of image frames indicated by the batch count is determined by the image signal processor based on the received indication of the frame rate change.

27. The computer-readable medium of claim 26, wherein: The number of image frames in the batch is based on a frame rate associated with the batch.

28. The computer-readable medium of claim 26, wherein: The image signal processor indicates the frame rate change to an application processor after configuring the one or more filters.

29. The computer-readable medium of claim 24, wherein: Execution of the instructions further causes the device to: processing, by the image signal processor, a first interval of image frames of the first sequence of image frames to generate a first partial preview stream, wherein the first interval is based on the first frame rate; processing, by the image signal processor, a second interval of image frames of the second sequence of image frames to generate a second partial preview stream, wherein the second interval is based on the second frame rate; generating, by the image signal processor, a video stream comprising the processed first image frame sequence and the processed second image frame sequence; outputting, by the image signal processor, the preview stream for display; and The video stream is output by the image signal processor.

30. The computer-readable medium of claim 29, wherein: The preview stream is at a fixed frame rate, and the video stream is at a variable frame rate.

31. The computer-readable medium of claim 29, wherein: Execution of the instructions further causes the device to encode, by an encoder, the first sequence of image frames and the second sequence of image frames to generate the video stream.

32. The computer-readable medium of claim 24, wherein: Execution of the instructions further causes the device to: A lookup table is stored by a memory, the lookup table mapping different frame rates to settings for configuring the one or more filters, wherein the image signal processor uses the lookup table to configure the one or more filters based on the frame rate change.

33. The computer-readable medium of claim 32, wherein: The lookup table also maps different frame rates to the number of image frames per batch for communication between the image signal processor and the application processor.

34. A device for digital image processing, comprising: means for receiving a first sequence of image frames from an image sensor at a first frame rate; means for processing each image frame in said first sequence of image frames at said first frame rate; means for receiving from the image sensor an indication of a change in frame rate of the image sensor from the first frame rate to a second frame rate; means for, in response to receiving an indication of the frame rate change from the image sensor, configuring one or more filters of the device to process image frames from the image sensor based on the frame rate change; means for receiving a second sequence of image frames from the image sensor at the second frame rate; as well as Means for processing each image frame of the second sequence of image frames at the second frame rate.

35. The apparatus of claim 34, wherein: The indication of the frame rate change is included in metadata of image frames subsequent to the first sequence of image frames from the image sensor.

36. The apparatus of claim 34, further comprising: means for indicating a batch count for the second sequence of image frames, wherein the number of image frames indicated by the batch count is determined based on the received indication of the frame rate change.

37. The apparatus of claim 36, wherein: The number of image frames in the batch is based on a frame rate associated with the batch.

38. The apparatus of claim 36, wherein: The frame rate change is indicated to an application processor after configuring the one or more filters.

39. The apparatus of claim 34, further comprising: means for processing a first interval of image frames of the first sequence of image frames to generate a first partial preview stream, wherein the first interval is based on the first frame rate; means for processing a second interval of image frames of the second sequence of image frames to generate a second partial preview stream, wherein the second interval is based on the second frame rate; means for generating a video stream comprising said first sequence of processed image frames and said second sequence of processed image frames; means for outputting said preview stream for display; and A component for outputting the video stream.

40. The apparatus of claim 39, wherein: The preview stream is at a fixed frame rate, and the video stream is at a variable frame rate.

41. The apparatus of claim 39, further comprising means for encoding the first sequence of image frames and the second sequence of image frames to generate the video stream.

42. The apparatus of claim 34, further comprising: Means for storing a lookup table that maps different frame rates to settings for configuring the one or more filters, wherein the lookup table is used to configure the one or more filters based on the frame rate change.

43. The apparatus of claim 42, wherein: The lookup table also maps different frame rates to the number of image frames per batch for communication with the application processor.

Images