Real-time streaming camera ISP with loosely coupled control loops

Abstract

A system, method, and apparatus for in-run camera image processing are provided that configure a system-on-chip (SoC) configuration register memory with one or more synchronization settings; processing a plurality of camera image frame sequences having different specified exposures to detect a complete set of frames having a first common frame number from the camera image frame sequences; identifying, from the sequence of image frames, an effective combination of different exposures having a second common frame number following the first common frame number; and processing the effective combination of the different exposures from the first and second sequences of camera image frames to generate output data from the ISP pipeline.

Description

Technical Field

[0001] The present disclosure generally relates to the field of image processing. In one aspect, the present disclosure relates to an image signal processing device, system and method for real-time image signal processing technology. Background Art

[0002] Computer vision systems are increasingly being used in diverse applications such as advanced driver assistance systems (ADAS), monitoring, inspection, safety, and remote sensing systems, as well as mainstream applications such as consumer digital imaging and real-time video conferencing. To improve image quality in such applications, camera image signal processing pipeline (ISP) techniques are used to generate high dynamic range (HDR) images by combining multiple images with different exposure times into a single high-quality image. While the raw input images captured by the camera can be stored in memory and then retrieved for HDR image processing, such memory storage and read operations increase memory system cost and processing latency while reducing memory system bandwidth (e.g., DDR). To avoid this drawback, a stream interface is deployed using a camera image signal processing pipeline (ISP) component for processing the raw camera stream on the fly without performing memory storage / read operations. While this camera ISP scheme can be deployed on a relatively small-cost MCU for a single-camera system, the entire ISP subsystem within a system-on-chip (SoC) solution still retains real-time critical hardware parts that require real-time critical DMA transfers and real-time control loop software executed on the application / control CPU. As can be seen from the foregoing, existing image signal processing is extremely difficult at a practical level due to the challenges of performing image signal processing quickly and efficiently, as real-time performance requirements are difficult to meet when using an application / control CPU to perform memory transfers within real-time control loop constraints. Summary of the invention

[0003] According to a first aspect of the present invention, there is provided a method for image processing of a running camera, comprising:

[0004] configuring, by a first processing circuit in a system on chip (SoC), one or more synchronization settings in a configuration register memory;

[0005] processing, by an image signal processing (ISP) pipeline coupled in the SoC and configured by a first synchronization setting having a first value, first and second camera image frame sequences having first and second specified exposures, respectively, to detect a complete set of frames having a first common frame number from the first and second camera image frame sequences;

[0006] identifying, by the ISP pipeline, from the first and second camera image frame sequences, a valid combination of different exposures having a second common frame number subsequent to the first common frame number; and

[0007] The valid combination of the different exposures from the first and second camera image frame sequences are processed by the ISP pipeline to generate output data from the ISP pipeline.

[0008] In one or more embodiments, configuring one or more synchronization settings includes storing a dual exposure VSYNCH locked (DEVL) value, wherein the DEVL value configures the ISP pipeline to detect the complete set of frames from the first and second camera image frame sequences before identifying a valid combination of the different exposures and generating the output data from the frame having the second common frame number from the first and second camera image frame sequences.

[0009] In one or more embodiments, configuring one or more synchronization settings includes storing a single step enable (SSEN) value that configures the ISP pipeline to generate the output data from a single set of frames having the second common frame number from the first and second camera image frame sequences.

[0010] In one or more embodiments, processing the first and second camera image frame sequences includes:

[0011] scanning the first and second camera image frame sequences to identify the complete set of frames having the first common frame number from the first and second camera image frame sequences; and

[0012] Image rows from the set of complete frames are discarded before valid combinations of the different exposures are identified.

[0013] In one or more embodiments, scanning the first and second camera image frame sequences comprises:

[0014] detecting a first vertical sync pulse from a first frame of the first camera image frame sequence; and

[0015] If a specified minimum number of image rows are received for the first frame from the first camera sequence of image frames, the first frame is identified as a complete frame.

[0016] In one or more embodiments, scanning the first and second camera image frame sequences comprises:

[0017] detecting a second vertical sync pulse from a second frame of the second sequence of camera image frames; and

[0018] If a specified minimum number of image rows are received for the second frame from the second camera sequence of image frames, then the second frame is identified as a complete frame.

[0019] In one or more embodiments, the method further includes processing, by the ISP pipeline configured with a second synchronization setting having a reset value, a combination of different exposures from the first and second camera image frame sequences having a third common frame number subsequent to the second common frame number to generate the output data from the ISP pipeline.

[0020] In one or more embodiments, the method further includes the ISP pipeline configured by a second synchronization setting having a setting value discarding exposures having a third common frame number after the second common frame number from the first and second camera image frame sequences, so that the output data is generated only from valid combinations of the different exposures having the second common frame number from the first and second camera image frame sequences.

[0021] According to a second aspect of the present invention, there is provided a system on chip (SoC), comprising:

[0022] a configuration register memory storing one or more synchronization settings;

[0023] a first processing circuit that configures the configuration register memory with the one or more synchronization settings;

[0024] a first camera image frame processing circuit connected to the input camera to receive and process first and second camera image frame sequences having first and second specified exposures, respectively; and

[0025] An image signal processing (ISP) pipeline, the ISP pipeline being connected and configured to:

[0026] In response to a first synchronization setting having a first value, processing the first and second camera image frame sequences to detect a complete set of frames having a first common frame number from the first and second camera image frame sequences;

[0027] identifying from the first and second camera image frame sequences a valid combination of different exposures having a second common frame number subsequent to the first common frame number; and

[0028] Valid combinations of the different exposures from the first and second camera image frame sequences are processed to generate output data from the ISP pipeline.

[0029] In one or more embodiments, the first processing circuit configures the first synchronization setting of the configuration register memory with a dual exposure VSYNCH locked (DEVL) value so that the ISP pipeline detects the complete frame set from the first and second camera image frame sequences before identifying the valid combination of the different exposures and generating the output data from the frame having the second common frame number from the first and second camera image frame sequences.

[0030] In one or more embodiments, the first processing circuit configures the first synchronization setting of the configuration register memory with a single step enable (SSEN) value so that the ISP pipeline generates the output data from a single set of frames having the second common frame number from the first and second camera image frame sequences.

[0031] In one or more embodiments, the ISP pipeline is configured to process the first and second camera image frame sequences by:

[0032] scanning the first and second camera image frame sequences to identify the complete set of frames having the first common frame number from the first and second camera image frame sequences; and

[0033] Image rows from the set of complete frames are discarded before valid combinations of the different exposures are identified.

[0034] In one or more embodiments, the ISP pipeline is configured to scan the first and second camera image frame sequences by:

[0035] detecting a first vertical sync pulse from a first frame of the first camera image frame sequence; and

[0036] If a specified minimum number of image rows are received for the first frame from the first camera sequence of image frames, the first frame is identified as a complete frame.

[0037] In one or more embodiments, the ISP pipeline is configured to scan the first and second camera image frame sequences by:

[0038] detecting a second vertical sync pulse from a second frame of the second sequence of camera image frames; and

[0039] If a specified minimum number of image rows are received for the second frame from the second camera sequence of image frames, then the second frame is identified as a complete frame.

[0040] In one or more embodiments, the ISP pipeline is further configured to process combinations of different exposures having a third common frame number following the second common frame number from the first and second camera image frame sequences in response to a second synchronization setting having a reset value to generate the output data from the ISP pipeline.

[0041] In one or more embodiments, the ISP pipeline is further configured to discard exposures having a third common frame number after the second common frame number from the first and second camera image frame sequences in response to a second synchronization setting having a set value, such that the output data is generated only from valid combinations of the different exposures having the second common frame number from the first and second camera image frame sequences.

[0042] According to a third aspect of the present invention, there is provided a real-time stream camera image signal processing system, comprising:

[0043] a memory device for storing a dual exposure VSYNCH lock (DEVL) synchronization setting and a single step enable (SSEN) synchronization setting;

[0044] a real-time camera stream pipeline hardware circuit connected and configured to receive and process first and second camera image frame sequences having first and second specified exposures, respectively; and

[0045] an automatic exposure control unit coupled to the memory device and connected to the real-time camera stream pipeline hardware circuit;

[0046] The real-time camera stream pipeline hardware circuit is connected and configured as follows:

[0047] responsive to the DEVL synchronization setting having a first value, processing the first and second camera image frame sequences to detect a complete set of frames having a first common frame number;

[0048] identifying from the first and second camera image frame sequences a valid combination of different exposures having a second common frame number subsequent to the first common frame number; and

[0049] Valid combinations of the different exposures from the first and second camera image frame sequences are processed to generate output data from the real-time camera stream pipeline hardware circuitry.

[0050] In one or more embodiments, the real-time camera stream pipeline hardware circuit is further configured to process combinations of different exposures having a third common frame number subsequent to the second common frame number from the first and second camera image frame sequences in response to a single step enable (SSEN) synchronization setting having a reset value to generate the output data from the ISP pipeline.

[0051] In one or more embodiments, the real-time camera stream pipeline hardware circuit is further configured to discard exposures having a third common frame number after the second common frame number from the first and second camera image frame sequences in response to a single step enable (SSEN) synchronization setting having a set value, such that the output data is generated only from valid combinations of the different exposures having the second common frame number from the first and second camera image frame sequences.

[0052] In one or more embodiments, the real-time camera stream pipeline hardware circuit is configured to process the first and second camera image frame sequences by:

[0053] scanning the first and second camera image frame sequences to identify the complete set of frames having the first common frame number from the first and second camera image frame sequences; and

[0054] Image rows from the set of complete frames are discarded before valid combinations of the different exposures are identified.

[0055] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention and its numerous objects, features and advantages achieved will be understood when the following detailed description of the preferred embodiments is considered in conjunction with the following drawings.

[0057] Figure 1 A simplified block diagram depicts a real-time streaming camera system employing an image signal processor with an automatic exposure control unit connected in a loosely coupled control loop to process input camera stream frames from an input camera using a real-time camera stream processing module in response to one or more synchronization settings in accordance with selected embodiments of the present disclosure.

[0058] Figure 2-3 Depicted is a streaming interface timing diagram for introducing camera input data into an image signal processor in accordance with selected embodiments of the present disclosure.

[0059] Figure 4is a timing diagram illustration of ISP output generated when a trigger is received while the channel stream interface is active and the dual exposure VSYNC lock value is set in accordance with a first selected embodiment of the present disclosure.

[0060] Figure 5 is a timing diagram illustration of ISP output generated when a trigger is received while the channel stream interface is inactive and a dual exposure VSYNC lock value is set in accordance with a second selected embodiment of the present disclosure.

[0061] Figure 6 is a timing diagram illustration of ISP output generated when a trigger is received while the channel stream interface is active and the dual exposure VSYNC lock value is reset in accordance with a third selected embodiment of the present disclosure.

[0062] Figure 7 A simplified flow chart showing logic for implementing dual exposure VSYNC lock is shown in accordance with selected embodiments of the present disclosure.

[0063] Figure 8 is a timing diagram illustration of an ISP output generated from a dual exposure camera when both a dual exposure VSYNC lock value and a single step frame value are set in accordance with selected dual exposure camera embodiments of the present disclosure.

[0064] Fig.9A is a timing diagram illustration of an ISP output generated from an HDR camera when both a dual exposure VSYNC lock value and a single step frame value are set in accordance with selected HDR camera embodiments of the present disclosure.

[0065] Fig. 9B is an illustration of a timing diagram of an ISP output generated from an HDR camera when setting a single-step frame value according to a selected back-to-back frame processing embodiment of the present disclosure.

[0066] Fig.10 A block diagram of a computer system according to selected embodiments of the present disclosure is schematically depicted. DETAILED DESCRIPTION

[0067] A real-time streaming camera ISP hardware device, system, architecture, method, and program code are described, wherein the camera ISP hardware device is configured to identify and continuously process valid combinations of exposures belonging to the same frame number from a camera input module using one or more synchronization settings. In selected embodiments, the camera ISP hardware device is configured with a dual exposure VSYNCH lock (DEVL) setting to respond to a trigger by scanning valid combinations of exposures (i.e., DCG and VS) with the same frame number, discarding corresponding image rows during scanning, generating ISP outputs from each valid combination of exposures belonging to the same frame number, and then discarding each fully scanned frame before the camera ISP hardware device starts generating ISP outputs from subsequent valid combinations of exposures for the next frame number, etc. In other selected embodiments, the camera ISP hardware device is configured with a single step enable (SSEN) setting to respond to a trigger by scanning valid combinations of exposures (i.e., DCG and VS) to be processed to generate an ISP output for a single ISI frame before processing of the next frame begins, and then discarding or discarding all subsequent frames from ISP processing until the ISP receives a subsequent trigger. In selected embodiments, the DEVL and SSEN settings can be configured individually or in combination to configure the camera ISP hardware device to efficiently process single exposure operations or dual exposure operations, so that the ISP pipeline can be implemented with existing DMA and CPU resources of a single SoC solution without requiring dedicated ISP-specific DMA and CPU resources of a separate SoC solution, thereby reducing die size by reusing existing SoC components. In addition, by using adaptively configurable DEVL and SSEN settings to enable real-time processing of input camera streams by the camera ISP hardware device without saving the input stream frames to external memory (e.g., DDR), bandwidth requirements, power consumption, and / or memory size requirements for external memory access are reduced.

[0068] To provide additional details to better understand the context of the present disclosure, reference is now made to Figure 1 , Figure 1A simplified block diagram of a real-time streaming camera system 1 is depicted, which employs an image signal processor 11 with an automatic exposure control unit 15, which is connected in a loosely coupled control loop to process input camera stream frames from an input camera 10 using real-time camera stream processing hardware / module 18 in response to one or more synchronization settings 16, 17. In selected embodiments, the input camera 10 can be a single camera or a multi-camera system, such as a Mobile Industry Processor Interface (MIPI) camera that is increasingly used for edge or industrial market applications and automotive market applications. As will be more fully understood and described below, the input camera 10 can generate input camera data as a streaming interface, the input camera data including input image frames and frame timing signals, such as VSYNC, HSYNC, enable data (Data-Enable), etc.

[0069] As shown, the image signal processor 11 also includes an automatic exposure control unit 15, which can be implemented with any suitable microcontroller, such as an ARM Cortex-A53 CPU. In addition, the image signal processor 11 includes a dynamic range compression (DRC) block 13, which is configured to perform calculations on the input image frame to generate table entries for storage in the table 14. The image signal processor 11 also includes an automatic exposure control unit 15, which is connected and configured to access the table entries from the table 14 and calculate the image processing algorithm (e.g., 3A algorithm) and other values ​​used in programming the feedback of the input camera 10 and / or programming the ISP configuration registers used by one or more of the image processing circuits 12A-G.

[0070] In order to process the input camera stream frames from the input camera 10, digital camera image signal processing increasingly requires an auto-exposure / balance (2A) algorithm or an auto-focus / exposure / white balance (3A) algorithm for camera ISP applications, where the ISP processing capabilities require efficient MEM-to-MEM ISP operations for multi-camera inputs and / or additional stream interfaces for single camera inputs. In selected embodiments, such image processing algorithms may perform high dynamic range (HDR) image processing, which combines multiple image frames with different exposure times into a single high-quality image. To this end, a real-time camera stream processing hardware / module 18 is provided to implement an image signal processing pipeline (ISP) to perform a pipeline sequence of image processing operations 12A-G. For example, the real-time camera stream processing hardware / module 18 may include an optical black / white balance circuit (OB / WB) circuit 12A, a Bayer noise reduction (BNR) circuit 12B, a high dynamic range (HDR) circuit 12C, a color temperature measurement (CTEMP) circuit 12D, an image demosaicing circuit 12E, a color correction circuit 12F, and a color space conversion circuit 12G. However, as will be appreciated by those skilled in the art, it will be appreciated that fewer or additional image processing circuits may be used, including but not limited to synchronization circuits, color matrix circuits, gamma conversion circuits, RGB conversion circuits, edge enhancement circuits, noise reduction circuits, resizing circuits, vignetting circuits, and / or image compression / expansion circuits, etc. It should be noted that while each of the depicted circuits 12A-G may process image data serially along an image data stream in the view, it is permissible for each circuit to process image data individually.

[0071] To avoid the memory cost (e.g., bandwidth requirements, power consumption, and / or memory size requirements) of storing incoming image frames in external memory, the real-time camera stream processing hardware / module 18 can be connected with the auto-exposure control unit 15 to form a loosely coupled control loop, where the input camera stream is fed directly to the ISP without saving the stream frames to external memory to save DDR bandwidth and power. However, there are costs and constraints for such real-time critical ISP control loops. For example, and as the timing Figure 2As shown, each frame resynchronization pulse 3A-3D signals the start of an input image frame and is used to trigger the ISP 11 to generate a corresponding ISP output 4A-4D after a short startup processing delay. In an example sequence where the control loop software from the automatic exposure control unit 15 operates on a given frame (e.g., frame A) to extract image parameter or metadata values ​​from frame A at time t1, the automatic exposure control unit 15 may require processing time 6 to process the image parameter values ​​for one or more subsequent frames (e.g., frame B). However, given the tight periodicity of the input image frames 2, the processing of the next frame (e.g., frame B) may require a shorter processing time 5 than can be achieved by a CPU-based software solution. Due to the longer software-based processing time 6, the processing of the image parameter values ​​extracted from frame A at time t1 is not completed until time t3, which is too late for processing the next frame B at time t2. Another disadvantage of conventional control loop solutions is that the measurement parameters calculated internally by the ISP (e.g., statistics, histograms, etc.) require dedicated real-time efficient DMA transfers and a dedicated CPU for ISP processing. This requirement increases the die size and cost used to implement the real-time streaming camera ISP processing.

[0072] To provide additional details to better understand the context of the present disclosure, reference is now made to Figure 2 , Figure 2 A first stream interface timing diagram 20 is depicted for introducing camera input data into an image signal processor. Specifically, a clock signal 21 is an input clock to an image signal processor that controls the stream interface timing for processing camera stream frames from an input camera. Additionally, a pixel data signal 22 indicates an input pixel data stream represented by 32 bits per pixel (bpp) that will be processed as "valid" or "invalid" data, depending on the other stream interface timing signals. Specifically, a horizontal synchronization (HSYNC) signal 23 is provided as a low level active signal to identify the start of each row in an image frame with a PW_H pulse, and a data enable (DE) signal 24 is provided to signal when the pixel data signal 22 is valid. In the case where the HSYNC signal 23 and the DE signal 24 are provided to multiple ISP receivers, a valid signal 25 is provided to each ISP receiver to indicate that the HSYNC signal 23 and the DE signal 24 are valid for processing by the ISP receiving the valid signal 25.

[0073] To provide additional details to better understand the context of the present disclosure, reference is now made to Figure 3 , Figure 3 A second stream interface timing diagram 30 is depicted for introducing camera input data into an image signal processor. Specifically, the HSYNC signal 31 is coupled to the Figure 2, but on a different time scale, and acts as a clock for the vertical synchronization (VSYNC) signal 33 described herein. In addition, the row data signal 32 indicates when an input row is to be processed as a "valid" or "invalid" row, depending on the other stream interface timing signals. Specifically, the VSYNC signal 33 is provided as a low-level active signal to identify the beginning of each new frame with a PW_V pulse, and the data enable (DE) signal 34 is provided with a pulse signal to signal when the row data signal 32 is valid, wherein the DE pulse signal is separated from the VSYNC signal pulse 33 at the front end by the BP_V pulse duration. In the case where the VSYNC signal 33 and the DE signal 34 are provided to multiple ISP receivers, a valid signal 35 is provided to each ISP receiver to indicate that the VSYNC signal 33 and the DE signal 34 are valid for processing by the ISP receiving the valid signal 35.

[0074] like Figure 2-3 As can be seen from the depicted channel stream interface (CSI) timing diagram, the camera captures a single frame 36 under the control of control signals 31-35, wherein the VSYNC signal 33 provides a frame start signal for the frame 36, and wherein the HSYNC signal 31 and the data enable signal 34 are combined to verify the input row data signal 32 for each row of the frame 36. Subsequent frames are also captured using corresponding control signaling. In addition, it should be understood that the camera can use the same control signaling to capture or generate multiple frame images of the same scene, albeit with different exposures. For example, the first image frame captured from the scene can be a dual conversion gain (DCG) exposure, and the second image frame captured from the same scene can be a very short (VS) exposure. Due to the different image processing requirements of the camera for different types of exposures, the channel stream interface timing for different frame exposures may be slightly different.

[0075] As will be appreciated, the camera's channel stream interface state must be coordinated or synchronized with the operation of the image signal processor 11 to ensure that incoming input camera stream frames from the input camera 10 can be received and processed by the ISP 11. However, there are many challenges in coordinating the reception and processing of frames from the channel stream interface. For example, and as described above, there may be a timing offset between receiving different frame exposures of the same scene. In addition, the channel stream interface can be in any state (i.e., active or inactive) when the ISP 11 is configured and triggered. Another challenge comes from the real-time constraints imposed by using the ISP control feedback loop to implement real-time streaming camera image signal processing (described above).

[0076] To address these challenges of coordinating the reception of frames from the channel stream interface for ISP processing under relaxed timing constraints, the real-time stream camera system 1 can be configured with one or more synchronization settings 16, 17 that are set by the host CPU 20 to control the real-time camera stream processing hardware / module 18 to identify and continuously process valid combinations of exposures belonging to the same frame number from the camera input module. In a selected embodiment, the real-time stream camera system 1 includes a register configuration space 19 that is configured by the CPU 20 with a dual exposure VSYNCH lock (DEVL) setting 16 to configure the image signal processing hardware device 11 to respond to a trigger by scanning valid combinations of exposures (i.e., DCG and VS) with the same frame number, discarding corresponding image rows during scanning, generating ISP outputs from each valid combination of exposures belonging to the same frame number, and then discarding each fully scanned frame before the camera ISP hardware device begins generating ISP outputs from subsequent valid combinations of exposures of the next frame number, etc.

[0077] To provide additional details to better understand selected embodiments of the present disclosure, reference is now made to Figure 4 , Figure 4 4 is a timing diagram 40 showing how an ISP output 46 is generated by locking onto two different exposure frame sequences 43, 44 in response to a trigger 45 received while the channel stream interface is in an active state and a dual exposure VSYNC lock (DEVL) value is set to a first value (e.g., DEVL=1). In this depicted example, the ISP 11 is operatively configured with a DEVL value (e.g., DEVL=1), and when the trigger signal 45 is received at time t0, the channel stream interface transitions from an "inactive" state (e.g., CSI inactive) 41 to an "active" state (e.g., CSI active) 42. In this mode of operation, the ISP 11 must be enabled to process valid combinations of exposures belonging to the same frame number from the input camera 10. In the depicted example, the first sequence of image frames 43 may be dual conversion gain (DCG) exposures, and the second sequence of image frames 44 may be very short (VS) exposures.

[0078] At time t0, the trigger 45 is issued while the CSI is active and just after the first VSYNC DCG frame starts, so before the trigger, "frame n" from the VSYNC DCG sequence 43 is already in progress. At time t1, the ISP system cannot detect the complete VSYNC DCG and VSYNC VS frames of "frame n" because only a portion of "frame n" from the VSYNC DCG sequence 43 is detected. Additionally, the ISP system cannot detect the complete VSYNC DCG and VSYNC VS frames of "frame n+1" because "frame n+1" from the VSYNC DCG sequence 43 has only started streaming. At time t2, the ISP system can detect the complete VSYNC DCG and VSYNC VS frames of "frame n+1" because both frames have been completely received, at which time lock of the VSYNC DCG "frame n+1" and VSYNC VS "frame n+1" is detected. Only after lock is detected does the ISP system begin processing the ISP output 46 calculated from VSYNC DCG "frame n+2" and VSYNC VS "frame n+2". Thus, when enabled using the DEVL field, the ISP 11 scans the incoming frame sequence 43, 44 for a valid combination of exposures (i.e., DCG and VS) for the same frame number to detect whether two full frames of different exposures have been received. When in frame detect scan mode, the ISP 11 discards the corresponding image rows and does not generate the ISP output 46, but once two full frames of different exposures have been scanned, the ISP 11 begins processing from the next frame (e.g., frame n+3, etc.) to generate the ISP output 46. All subsequent frames are processed and output as needed.

[0079] While the ISP system can perform frame tracking by detecting when all horizontal lines of a frame have been received, this may constrain the ISP system's ability to process frame data in sufficient time to enable processing of subsequent frames. Accordingly, selected embodiments of the ISP system may be configured to perform frame tracking by detecting when a predetermined portion of horizontal lines of a frame have been received. For example, an internal counter of the ISP system may be incremented with each HSYNC pulse, and when half the frame height is detected from the HSYNC pulse count, this may be used to signal that a "full" frame has been detected.

[0080] To provide additional details to better understand selected embodiments of the present disclosure, reference is now made to Figure 5 , Figure 55 is a timing diagram 50 showing how an ISP output 56 is generated by locking onto two different exposure frame sequences in response to a trigger 55 received when the channel stream interface is in an inactive state and a dual exposure VSYNC lock (DEVL) value is set to a first value (e.g., DEVL=1). In this depicted example, the ISP 11 is operatively configured with a DEVL value (e.g., DEVL=1), and after receiving the trigger signal 55 at time t0, the channel stream interface transitions from an "inactive" state (e.g., CSI inactive) 51 to an "active" state (e.g., CSI active) 52. In this operating mode, the ISP 11 must be enabled to process valid combinations of exposures belonging to the same frame number from the input camera 10. In the depicted example, the first sequence of image frames 53 may be dual conversion gain (DCG) exposures, and the second sequence of image frames 54 may be very short (VS) exposures.

[0081] At time t0, when the CSI is inactive and before the first VSYNC DCG frame starts, a trigger 55 is issued, so "frame n" from the VSYNC DCG sequence 53 has not yet started. At time t1, the ISP system cannot detect the complete VSYNC DCG and VSYNC VS frames of "frame n" because only a portion of "frame n" from the VSYNC DCG sequence 53 is detected. At time t2, the ISP system can detect the complete VSYNC DCG and VSYNC VS frames of "frame n" because both frames have been received in their entirety, at which point lock of VSYNC DCG "frame n" and VSYNC VS "frame n" is detected. Only after lock is detected does the ISP system begin calculating the ISP output 56 based on VSYNC DCG "frame n+1" and VSYNC VS "frame n+1". Thus, when enabled using the DEVL field, the ISP 11 scans the incoming frame sequence 53, 54 for valid combinations of exposures (i.e., DCG and VS) for the same frame number to detect whether two full frames of different exposures are received. When in frame detect scan mode, the ISP 11 discards the corresponding image rows without generating the ISP output 56, but once two full frames of different exposures are scanned, the ISP 11 begins processing from the next frame (e.g., frame n+2, frame n+3, etc.) to generate the ISP output 56. All subsequent frames are processed and output as needed. As described above, the ISP system can perform frame tracking by detecting when all horizontal rows of a frame or a predetermined portion of a row of a frame have been received.

[0082] To provide additional details to better understand selected embodiments of the present disclosure, reference is now made to Figure 6 , Figure 664. is a timing diagram 60 showing how the ISP output 66 is generated without first locking onto two full frames from different exposure frame sequences 63, 64. In this mode, the CPU 20 first ensures that the ISP 11 is triggered only when the CSI is in the active state 62, but this mode does not allow restarting of the ISP 11. As depicted, a trigger 65 is received while the channel stream interface is in the inactive state 61 and while the dual exposure VSYNC lock (DEVL) value is set to a second value (e.g., DEVL=0). Once the channel stream interface is in the active state 62, the ISP 11 is enabled to immediately begin processing valid combinations of exposures 63, 64 belonging to the same frame number from the input camera 10. Specifically, while the CSI is in the inactive state 61, at time t0, the trigger 65 is issued, and in response to the CSI becoming active 62, the ISP 11 begins generating the ISP output 66 from "frame n" of the VSYNC DCG sequence 63 and the VSYNC VS sequence 64 at time t1, even before two full frames have been detected.

[0083] In order to provide more details for a better understanding of selected embodiments of the present disclosure, reference is now made to Figure 7 , Figure 7 A simplified flowchart 70 is depicted showing logic for implementing dual exposure VSYNC lock, such as may be used by an image signal processing unit and / or a real-time camera stream processing hardware unit in a computer system. In flowchart 70, the method begins at step 700 when the real-time camera stream processing hardware unit receives a trigger while a channel stream interface (CSI) is active or inactive. Once the CSI is active, two or more different exposure frame sequences are received from the camera and are processed separately in a parallel processing sequence by the real-time camera stream processing hardware unit.

[0084] In the first processing sequence 701-704, at step 701, the real-time camera stream processing hardware unit monitors the first exposure frame sequence (exposure A) to detect the first VSYNC pulse signal (VSYNC A,N ) is received. If the first VSYNC pulse signal (VSYNC A,N ), the detection process continues (as indicated by a negative result in step 701). However, after the first VSYNC pulse signal (VSYNC A,N ), which indicates that the "frame start" of the Nth frame has been received (as indicated by the positive result of step 701), and then the first corresponding flag is set in step 702, for example, by setting VSYNC for exposure A. A,NFlag. Next, in a first processing sequence, at step 703, the real-time camera stream processing hardware unit scans the first exposure frame sequence (exposure A) to detect the receipt of an incoming frame (e.g., frame N) in the first exposure frame sequence (exposure A). For example, the scanning step 703 can detect when a first predetermined portion of the incoming frame (e.g., frame N) has been received, such as by counting the number of received frame rows (indicated by received HSYNC pulses) of the Nth frame of the first exposure frame sequence (exposure A) to detect when half of the Nth frame of exposure A has been received. If the desired portion of the Nth frame of exposure A has not been received, the scanning process continues (as indicated by a negative result of step 703). However, when the desired portion of the Nth frame of exposure A is detected (as indicated by a positive result of step 703), then a second corresponding flag is set at step 704, such as by setting a VSYNC pulse for exposure A. A,N+1 Logo.

[0085] In parallel with the first processing sequence 701-704, the real-time camera stream processing hardware unit performs the second processing sequence 711-714, although in selected embodiments, the second processing sequence 711-714 may be performed in whole or in part before or after the first processing sequence 701-704. As depicted, the second processing sequence 711-714 begins by monitoring the second exposure frame sequence (exposure B) at step 711 to detect receipt of the second VSYNC pulse signal (VSYNCB,N). If the second VSYNC pulse signal (VSYNCB,N) is not received, the second VSYNC pulse signal (VSYNCB,N) is received. B,N ), the detection process continues (as indicated by a negative result in step 711). However, upon detecting the second VSYNC pulse signal (VSYNC B,N ), which indicates that the "frame start" of the Nth frame has been received (as indicated by the positive result of step 711), and then in step 712, a third corresponding flag is set, for example, by setting VSYNC for exposure B. B,N Flag. Next, in a second processing sequence, at step 713, the real-time camera stream processing hardware unit scans the second exposure frame sequence (exposure B) to detect the receipt of an incoming frame (e.g., frame N) in the second exposure frame sequence (exposure B). For example, the scanning step 713 can detect when a first predetermined portion of the incoming frame (e.g., frame N) has been received, such as by counting the number of received frame rows (indicated by received HSYNC pulses) of the Nth frame of the second exposure frame sequence (exposure B) to detect when half of the Nth frame of exposure B has been received. If the desired portion of the Nth frame of exposure B is not received, the scanning process continues (as indicated by a negative result of step 713). However, when the desired portion of the Nth frame of exposure B is detected (as indicated by a positive result of step 713), then a fourth corresponding flag is set at step 714, such as by setting a VSYNC pulse for exposure B.B,N+1 Logo.

[0086] At step 715, the real-time camera stream processing hardware unit determines whether two "full" frames from different exposure frame sequences (exposure A, B) have been received by detecting whether all flags have been set. Specifically, the first to fourth flags are checked at this time. If all flags have not been set (as indicated by a negative result of step 715), this indicates that two "full" frames from different exposure frame sequences (exposure A, B) have not been received for frame N, so the counter value N is then incremented at step 716, and the process returns to step 701 to start over, where the next incoming frame (actually, frame N+1) from the different exposure frame sequence (exposure A, B) is evaluated using the first processing sequence 701-704 and the second processing sequence 711-714 at step 701. However, once it is detected that all required flags have been set (as indicated by an affirmative result in step 715), then in step 717, dual exposure VSYNCH lock has been achieved between different exposure frame sequences (exposure A, B), and the real-time camera stream processing hardware unit is configured to generate an ISP output by starting with processing "frame N+1" of the different exposure frame sequence (exposure A, B), and then continuously processing subsequent frames (e.g., frame N+2, frame N+3, etc.) as needed. In addition, the generated ISP output can be saved in a memory as embedded data. For example, an id-based or row-based identification mechanism can be used for the embedded data stored or saved in the memory with the ISP output. In a selected embodiment, the embedded data has frame-specific parameters, such as frame number, analog gain of the sensor, etc.

[0087] In addition to using a dual exposure VSYNCH locking mechanism to address the challenges of coordinating the reception of frames from the channel stream interface as described above, the real-time stream camera system 1 can also be configured with a single step enable (SSEN) setting 17 to respond to a trigger by scanning a valid combination of exposures (i.e., DCG and VS) to be processed to generate a single ISI frame of ISP output before processing of the next frame begins, and then discarding or dropping all subsequent frames from ISP processing until the ISP receives a subsequent trigger. In such an embodiment, the real-time stream camera system 1 includes a register configuration space 19 that is configured by the CPU 20 with the SSEN setting 17 to configure the image signal processing hardware device 11 to respond to a trigger by scanning a valid combination of exposures (i.e., DCG and VS) having the same frame number, discarding corresponding image rows during the scanning, generating ISP outputs from each valid combination of exposures belonging to the single frame number, and then ceasing to generate ISP outputs from subsequent frames.

[0088] To provide additional details to better understand selected embodiments of the present disclosure, reference is now made to Figure 8 , Figure 8 8 is a timing diagram 80 of an embodiment in which ISP outputs 83, 85 are generated in response to triggers 82, 84 from a dual exposure camera, respectively, when both the dual exposure VSYNC lock value and the single step frame value are set. In this depicted example, the ISP 11 is operatively configured with DEVL and SSEN values ​​being first values ​​(e.g., DEVL=SSEN=1), and the channel stream interface is in an "active" state when the first trigger signal 82 is received at time t0. In this operating mode, the ISP 11 is enabled to generate ISP outputs by processing valid combinations of exposures belonging to the same frame number from the input camera 10, but each ISP output is generated by processing only a single frame from the valid combination of exposures. In the depicted example, the first sequence of image frames 80 can be dual conversion gain (DCG) exposures, and the second sequence of image frames 81 can be very short (VS) exposures.

[0089] like Figure 8As depicted, at time t0, when the CSI is active and in the middle of the first frame ("frame n") of the VSYNC DCG sequence 80 and the VSYNCVS sequence 81, a trigger 82 is issued. Due to the DEVL setting (DEVL = 1), at time t0, the ISP system cannot detect the complete VSYNC DCG and VSYNC VS frames of "frame n" because only a portion of "frame n" from the VSYNC DCG sequence 80 and the VSYNC VS sequence 81 is detected; however, at time t1, the ISP system can detect the complete VSYNC DCG and VSYNC VS frames of "frame n+1" because both frames have been completely received, at which time a double lock of VSYNC DCG "frame n+1" and VSYNC VS "frame n+1" is detected. Only after the double lock is detected does the ISP system begin to calculate the ISP output 83 based on the VSYNC DCG "frame n+2" and VSYNC VS "frame n+2". However, due to the SSEN setting (SSEN=1), ISP11 stops generating ISP output 83, and the ISP system discards all subsequent frames until the next trigger 84 is received and the process starts over. Specifically, when the second trigger 84 is issued at time t2, in the middle of the fourth frame ("Frame n+3") of the VSYNC DCG sequence 80 and the VSYNC VS sequence 81, due to the DEVL setting (DEVL=1), the ISP system cannot detect the complete VSYNC DCG and VSYNC VS frames of "Frame n+3". However, at time t3, the ISP system can detect the complete VSYNC DCG and VSYNC VS frames of "Frame n+4" because both frames have been completely received, and a double lock of VSYNC DCG "Frame n+4" and VSYNCVS "Frame n+4" is detected. Only after the double lock is detected does the ISP system begin to calculate the ISP output 85 based on VSYNC DCG "Frame n+5" and VSYNC VS "Frame n+5". However, due to the SSEN setting (SSEN=1), the ISP 11 stops generating the ISP output 85 and the ISP system discards all subsequent frames until the next trigger is received and the process starts over.

[0090] To provide additional details to better understand selected embodiments of the present disclosure, reference is now made to Fig.9A , Fig.9A90A is a timing diagram of an embodiment in which ISP outputs 93, 95 are generated in response to triggers 92, 94 from an HDR camera, respectively, when a dual exposure VSYNC lock value is set and a single step frame value is set. In this depicted example, the ISP 11 is operatively configured with a reset DEVL value (DEVL=0) and a set SSEN value (SSEN=1), and the channel stream interface is in an "active" state when the first trigger signal 92 is received at time t0 during reception of the VSYNC HDR pulse sequence 91. In this operating mode, the ISP 11 is enabled to generate ISP outputs at the next frame received from the input camera 10, but each ISP output is generated by processing only a single frame of the CSI sequence from the HDR image frame. As Fig.9A As depicted, at time t0, a trigger 92 is issued when the CSI is active and in the middle of the first frame ("frame n") from the VSYNC HDR sequence 91. Due to the DEVL setting (DEVL=0), the ISP system does not wait for detection of a full frame, so at time t1, the ISP system begins to calculate the ISP output 93 based on VSYNC HDR "frame n+1". However, due to the SSEN setting (SSEN=1), the ISP 11 stops generating the ISP output 93 after VSYNC HDR "frame n+1", and the ISP 11 can perform a save / restore on local memory before processing of the next frame begins. In addition, the ISP 11 is configured by the SSEN setting (SSEN=1) so that all subsequent frames are discarded until the next trigger 94 is received and the process starts over. Specifically, at time t2, when the second trigger 94 is issued in the middle of the fifth frame ("frame n+4") from the VSYNC HDR sequence 91, the ISP system begins generating the ISP output 95 calculated based on the VSYNC HDR "frame n+5" without waiting for the detection of the full frame. Also, due to the SSEN setting (SSEN=1), the ISP 11 stops generating the ISP output 95, and the ISP system discards all subsequent frames until the next trigger is received and the process starts over.

[0091] To provide additional details to better understand selected back-to-back frame processing embodiments of the present disclosure, reference is now made to Fig. 9B , Fig. 9B90B is a timing diagram of an embodiment in which ISP outputs 97, 99 are generated in response to triggers 96, 98 from an HDR camera, respectively, when a dual exposure VSYNC lock value is set and a single step frame value is set. In this depicted example, the ISP 11 is operatively configured with a reset DEVL value (DEVL=0) and a set SSEN value (SSEN=1), and the channel stream interface is in an "active" state when the first trigger signal 97 is received at time t0 during reception of the VSYNCHDR pulse sequence 91. In this operating mode, the ISP 11 is enabled to generate ISP outputs at the next frame received from the input camera 10, but each ISP output is generated by processing only a single frame of the CSI sequence from the HDR image frame. As Fig. 9B As depicted, at time t0, when the CSI is active and in the middle of the first frame ("frame n") from the VSYNC HDR sequence 91, a trigger 96 is issued. Due to the DEVL setting (DEVL=0), the ISP system does not wait for detection of a full frame, so at time t1, the ISP system begins ISP output 97 calculated from VSYNC HDR "frame n+1". However, due to the SSEN setting (SSEN=1), the ISP 11 stops generating ISP output 97 after VSYNC HDR "frame n+1", but when a back-to-back trigger 98 is received in the middle of the second frame ("frame n+1") from the VSYNC HDR sequence 91 at time t2, the ISP system begins generating ISP output 99 calculated from VSYNC HDR "frame n+2" without waiting for detection of a full frame. Likewise, due to the SSEN setting (SSEN=1), the ISP 11 stops generating ISP output 99, and the ISP system discards all subsequent frames until the next trigger is received and the process starts over.

[0092] To provide additional details to better understand selected embodiments of the present disclosure, reference is now made to Fig.10 , Fig.10A simplified architectural block diagram of a microcontroller system 100 for automotive and industrial high performance radar applications is shown, the microcontroller system 100 implementing a real-time streaming camera image signal processor with loosely coupled control loops. As depicted, the microcontroller system 100 includes a direct memory access (DMA) block 1001, a system control block 1002, a main CPU platform 1003, a vision accelerator block 1006, and a video accelerator 1009. In the main CPU platform 1003, one or more control processors or central processing unit (CPU) subsystems 1004 and on-chip memory 1005 (e.g., volatile or non-volatile memory) are provided. In addition, the vision accelerator block 1006 can include a dual exposure frame lock detector 1007, which enables the vision accelerator 1006 to generate an ISP output by identifying and continuously processing valid combinations of exposures belonging to the same frame number from the camera interface module 1013 after detecting a corresponding full frame from each of a plurality of different exposure frame sequences in response to a predetermined VSYNCH lock (DEVL) setting. Additionally, the vision accelerator block 1006 may include a single-step ISI frame detector / processor 1008 that enables the vision accelerator 1006 to generate an ISP output from a single corresponding set of frames from the camera interface module 1013 in response to a predetermined SSEN lock setting.

[0093] The DMA block 1001, the system control block 1002, the main CPU platform 1003, the vision accelerator block 1006, and the video accelerator 1009 can be connected to a memory interface 1011 (e.g., a DDR interface), one or more connectivity modules 1012 (e.g., PCIe 2x, FlexPWM, eTimer, IIC, DSPI, etc.), a camera interface module 1013, and other modules 1014 via an on-chip interconnect 1010. In selected embodiments, the microcontroller system 100 can be implemented as a circuit system on a single integrated circuit or system on chip (SoC). In addition, the interconnect 1010 can be any type of bus structure, including but not limited to a non-coherent interconnect, an advanced high-performance bus (AHB), or an advanced peripheral bus (APB). In addition, the control processor 1004 can be any type of processing circuit, including but not limited to a microprocessor (MPU), a microcontroller (MCU), a digital signal processor (DSP), or another type of processor or processor core with multiple cache memory levels. Although not shown, the microcontroller system 100 may include a peripheral device or a dedicated processor, a communication interface, a timer, an encoder / decoder, and one or more external memory interfaces, such as a DDR interface or a flash memory interface. Then, the external memory interface 1011 may be connected to an external memory, such as a DDR memory or a flash memory.

[0094] As disclosed, the microcontroller system 100 can use the control processor 1004 to configure the DEVL and SSEN register settings at the vision accelerator 1006 individually or in combination, thereby configuring the vision accelerator 1006 to efficiently process single exposure operations or dual exposure operations, so that the ISP pipeline can be implemented with the existing DMA 1001 and control processor 1004 resources of a single SoC solution without requiring dedicated ISP-specific DMA and CPU resources of a separate SoC solution, thereby reducing the die size by reusing existing SoC components. In addition, by using the adaptively configurable DEVL and SSEN register settings at the vision accelerator 1006 to enable real-time processing of the input camera stream at the vision accelerator 1006 without saving the input stream frames to an external memory (e.g., DDR), the bandwidth requirements, power consumption, and / or memory size requirements of external memory access are reduced.

[0095] The term "module" may be defined to include multiple executable modules. A module may include software, hardware, or some combination thereof that may be executed by a processor such as the control processor 1004. A software module may include instructions stored in a memory such as the memory 1005 or another memory device that may be executed by the control processor 1004 or other processors. A hardware module may include various devices, components, circuits, gates, circuit boards, etc. that may be executed, directed, and / or controlled in performance by the control processor 1004.

[0096] Computer-readable media or machine-readable media may include any non-transitory memory device that contains or stores software for use by or in conjunction with an instruction-executable system, device, or apparatus. A machine-readable medium may be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus. Examples may include a portable magnetic or optical disk; volatile memory, such as random access memory "RAM," read-only memory "ROM," or erasable programmable read-only memory "EPROM"; or flash memory. Machine-readable storage may also include non-transitory tangible media on which the software is stored. The software may be stored electronically as an image or in another format (e.g., by optical scanning) and then compiled, interpreted, or otherwise processed.

[0097] As will be appreciated, the term "computer-readable medium" may include a single medium or multiple media, such as a centralized or distributed database, and / or associated caches and servers storing one or more sets of instructions. Computer-readable media may also include any medium capable of storing, encoding, or carrying a set of instructions for execution by a processor or causing a computer system to perform any one or more of the disclosed methods or operations. Computer-readable media may be non-temporary and may be tangible. In addition, computer-readable media may include solid-state memory, such as a memory card or other package that accommodates one or more non-volatile read-only memories. Computer-readable media may be random access memory or other volatile rewritable memory. Computer-readable media may include magneto-optical or optical media, such as a disk or tape or other storage device, for capturing carrier signals, such as signals transmitted through a transmission medium. Digital file attachments to e-mails or other independent information archives or archives may be considered as distributed media in the form of tangible storage media. As will be appreciated, any one or more of computer-readable media or distributed media and other equivalent and subsequent media may be included for storing data or instructions.

[0098] In selected embodiments, the visual accelerator 1006 may be implemented with dedicated hardware, such as application specific integrated circuits, programmable logic arrays, and other hardware devices, which may be constructed to implement various parts of the system. Applications that may include the apparatus and system may widely include various electronic and computer systems. One or more of the examples described may implement functions using two or more specific interconnected hardware modules or devices, wherein related control and data signals may be transmitted between and through the modules, or may be implemented as part of an application specific integrated circuit. The described system may include a software program that may be executed by a computer system. Implementations may include distributed processing, component / object distributed processing, and parallel processing. Alternatively, virtual computer system processing may be constructed to implement various parts of the system.

[0099] It should now be appreciated that an apparatus, method, program code, and system for performing on-the-fly camera image processing have been provided. In the disclosed embodiments, a first processing circuit in a system on chip (SoC) configures one or more synchronization settings in a configuration register memory. In addition, an image signal processing (ISP) pipeline is connected in a control feedback loop in the SoC and configured by a first synchronization setting having a first value to process first and second camera image frame sequences having first and second specified exposures, respectively, thereby detecting a complete set of frames having a first common frame number from the first and second camera image frame sequences. In addition, the ISP pipeline identifies valid combinations of different exposures having a second common frame number after the first common frame number from the first and second camera image frame sequences. Finally, the ISP pipeline processes valid combinations of different exposures from the first and second camera image frame sequences to generate output data from the ISP pipeline. In selected embodiments, one of the configured synchronization settings is stored as a dual exposure VSYNCH lock (DEVL) value, which configures the ISP pipeline to detect a complete set of frames from the first and second camera image frame sequences before identifying valid combinations of different exposures and generating output data from frames having a second common frame number from the first and second camera image frame sequences. In other selected embodiments, one of the configured synchronization settings is stored as a single step enable (SSEN) value that configures the ISP pipeline to generate output data from a single set of frames having a second common frame number from the first and second camera image frame sequences. In selected embodiments, the first and second camera image frame sequences are processed at the ISP pipeline by: scanning the first and second camera image frame sequences to identify a complete set of frames having a first common frame number from the first and second camera image frame sequences; and discarding image rows from the complete set of frames before identifying a valid combination of different exposures. In such embodiments, scanning the first and second camera image frame sequences may include: detecting a first vertical synchronization pulse from a first frame from the first camera image frame sequence; and if a specified minimum number of image rows are received for the first frame from the first camera image frame sequence, identifying the first frame as a complete frame. Additionally, scanning the first and second camera image frame sequences may include: detecting a second vertical synchronization pulse from a second frame from the second camera image frame sequence; and if a specified minimum number of image rows are received for the second frame from the second camera image frame sequence, identifying the second frame as a complete frame. In selected embodiments, the ISP pipeline can be additionally configured via a second synchronization setting having a reset value to process combinations of different exposures from the first and second camera image frame sequences having a third common frame number subsequent to the second common frame number to generate output data from the ISP pipeline.Additionally or in the alternative, the ISP pipeline may be further configured to discard exposures having a third common frame number following the second common frame number from the first and second camera image frame sequences via a second synchronization setting having a set value, such that the output data is generated only from valid combinations of different exposures having the second common frame number from the first and second camera image frame sequences. Although the described exemplary embodiments disclosed herein focus on configuring the ISP hardware device to identify and continuously process valid combinations of exposures belonging to the same frame number from the camera input module, the present invention is not necessarily limited to the example embodiments shown herein, and may be applied to any image processing system in which the DEVL and SSEN configurations are set to effectively synchronize and control single or dual exposure operations when generating ISP outputs. Therefore, the specific embodiments disclosed above are merely illustrative and should not be construed as limitations on the present invention, as the present invention may be modified and implemented in different but equivalent manners that are apparent to those skilled in the art having the benefit of the teachings herein. Accordingly, the above description is not intended to limit the invention to the specific forms set forth, but on the contrary it is intended to cover such alternatives, modifications and equivalents as may be included within the spirit and scope of the invention as defined by the appended claims, so that those skilled in the art should understand that they can make various changes, substitutions and alterations in its broadest form without departing from the spirit and scope of the invention.

[0100] Benefits, other advantages, and solutions to problems are described above with respect to specific embodiments. However, those benefits, advantages, solutions to problems, and any elements that may cause any benefit, advantage, or solution to occur, or become more significant, shall not be construed as key, required, or essential features or elements of any or all claims. As used herein, the term "comprises" or any other variation thereof is intended to encompass non-exclusive inclusions, such that a process, method, article, or apparatus that includes a list of elements may include not only those elements, but may include other elements that are not expressly listed or inherent to such process, method, article, or apparatus.

Claims

1. A method for image processing of a running camera, characterized in that include: configuring, by a first processing circuit in the system-on-chip SoC, one or more synchronization settings in a configuration register memory; processing, by an image signal processing (ISP) pipeline connected in the SoC and configured by a first synchronization setting having a first value, first and second camera image frame sequences having first and second specified exposures, respectively, to detect a complete set of frames having a first common frame number from the first and second camera image frame sequences; identifying, by the ISP pipeline, from the first and second camera image frame sequences, a valid combination of different exposures having a second common frame number subsequent to the first common frame number; as well as The valid combination of the different exposures from the first and second camera image frame sequences are processed by the ISP pipeline to generate output data from the ISP pipeline.

2. The method according to claim 1, characterized in that Configuring one or more synchronization settings includes storing a dual exposure VSYNCH lock DEVL value, which configures the ISP pipeline to detect the complete set of frames from the first and second camera image frame sequences before identifying a valid combination of the different exposures and generating the output data from the frames having the second common frame number from the first and second camera image frame sequences.

3. The method according to claim 1, characterized in that Configuring one or more synchronization settings includes storing a single-step enable SSEN value that configures the ISP pipeline to generate the output data from a single set of frames having the second common frame number from the first and second camera image frame sequences.

4. The method according to claim 1, characterized in that: Processing the first and second camera image frame sequences comprises: scanning the first and second camera image frame sequences to identify the complete set of frames having the first common frame number from the first and second camera image frame sequences; and Image rows from the set of complete frames are discarded before valid combinations of the different exposures are identified.

5. The method according to claim 4, characterized in that Scanning the first and second camera image frame sequences comprises: detecting a first vertical sync pulse from a first frame of the first camera image frame sequence; and If a specified minimum number of image rows are received for the first frame from the first camera sequence of image frames, the first frame is identified as a complete frame.

6. The method according to claim 5, characterized in that Scanning the first and second camera image frame sequences comprises: detecting a second vertical sync pulse from a second frame of the second sequence of camera image frames; and If a specified minimum number of image rows are received for the second frame from the second camera sequence of image frames, then the second frame is identified as a complete frame.

7. The method according to claim 1, characterized in that Also included is the ISP pipeline configured by a second synchronization setting having a reset value to process a combination of different exposures having a third common frame number following the second common frame number from the first and second camera image frame sequences to generate the output data from the ISP pipeline.

8. The method according to claim 1, characterized in that: Also included is the ISP pipeline configured by a second synchronization setting having a set value to discard exposures having a third common frame number after the second common frame number from the first and second camera image frame sequences, so that the output data is generated only from valid combinations of the different exposures having the second common frame number from the first and second camera image frame sequences.

9. A system on chip (SoC), characterized in that: include: a configuration register memory storing one or more synchronization settings; a first processing circuit that configures the configuration register memory with the one or more synchronization settings; a first camera image frame processing circuit connected to the input camera to receive and process first and second camera image frame sequences having first and second specified exposures, respectively; as well as An image signal processing (ISP) pipeline, the ISP pipeline being connected and configured to: In response to a first synchronization setting having a first value, processing the first and second camera image frame sequences to detect a complete set of frames having a first common frame number from the first and second camera image frame sequences; identifying, from the first and second camera image frame sequences, a valid combination of different exposures having a second common frame number subsequent to the first common frame number; as well as Valid combinations of the different exposures from the first and second camera image frame sequences are processed to generate output data from the ISP pipeline.

10. A real-time streaming camera image signal processing system, characterized in that: include: a memory device for storing a dual exposure VSYNCH lock DEVL synchronization setting and a single step enable SSEN synchronization setting; a real-time camera stream pipeline hardware circuit connected and configured to receive and process first and second camera image frame sequences having first and second specified exposures, respectively; as well as an automatic exposure control unit coupled to the memory device and connected to the real-time camera stream pipeline hardware circuit; The real-time camera stream pipeline hardware circuit is connected and configured as follows: responsive to the DEVL synchronization setting having a first value, processing the first and second camera image frame sequences to detect a complete set of frames having a first common frame number; identifying, from the first and second camera image frame sequences, a valid combination of different exposures having a second common frame number subsequent to the first common frame number; as well as Valid combinations of the different exposures from the first and second camera image frame sequences are processed to generate output data from the real-time camera stream pipeline hardware circuitry.

Images