Robust frame size error detection and recovery mechanism

Abstract

Receive (405) an image data frame from an external source. In response to determining (410) that a first frame size of the received image data frame is incorrect, perform an error concealment operation on the received image data frame. Determine that the first frame size of the image data frame is incorrect based on at least one frame synchronization signal associated with the image data frame. Perform an image processing operation on the received image data frame on which the error concealment operation has been performed, so that the image processing module can perform the image processing operation without entering a deadlock state, and thereby prevent the main processor from having to perform a hardware reset of the deadlock module.

Description

Background Art

[0001] Autonomous driving is one of the most challenging computing problems in the world. Different functions related to autonomous driving or vehicle maneuvering can be implemented using an Advanced Driver Assistance System (ADAS) that relies on a sensor suite providing data on the environment of the host vehicle. Such a sensor suite can include RADAR (Radio Detection and Ranging), LIDAR (Light Detection and Ranging), cameras for imaging, etc. The range of ADAS functions implemented can range from distance sensing and parking assistance to complex ADAS functions such as cruise control, lane change assistance, collision mitigation, emergency braking, full autonomous driving, etc. A large amount of data from image sensors, RADAR, LIDAR, and high-definition maps must be processed to generate commands for safely and comfortably controlling a vehicle in real time in such ADAS systems. This challenging task requires one or more dedicated computing devices that are energy-efficient and low-power, run complex high-performance software, and rely on breakthroughs in artificial intelligence, machine learning, deep learning, and computer vision. Such computing devices can be implemented as energy-efficient and space-saving System-on-Chip (SoC) that can be integrated into a flexible and scalable platform capable of enabling various autonomous vehicles.

[0002] Typical computing devices (e.g., SOC) include multiple internal components that exchange data. For example, a typical processor can include multiple functional blocks such as an image signal processor that exchanges and processes image data, a display engine, and / or a media encoder. Errors in the incoming continuous image data stream processed by the image signal processor module on the SoC can not only result in corrupted pixel data (rendering the image frame unusable for safety-critical applications such as autonomous driving, computer vision), but the faulty frame can also cause other types of problems in the image processing pipeline (e.g., deadlock conditions), which require a hardware reset of one or more modules of the SoC to correct the problem. Using a reset to restart the processing module has adverse side effects and typically results in the loss of additional image frames, which violates the high-performance, high-reliability, and low-latency requirements of safety-critical applications. For example, multiple image data frame losses in an ADAS system that relies on image data frames for autonomous driving may be unacceptable. Summary of the Invention

[0003] A simplified overview of the disclosed subject matter is presented below in order to provide a basic understanding of some aspects of the present disclosure. This overview is not an exhaustive review of the technology disclosed herein. It is not intended to identify the important or critical elements of the present disclosure or to delineate the scope of the present disclosure. Its sole purpose is to present some concepts in a simplified form as a prelude to a more detailed description discussed later.

[0004] In one example, an image processing device includes: an image data receiver configured to receive an image data frame from an external source; an error handling module communicatively coupled to receive the image data frame from the image data receiver and configured to perform an error concealment operation on the received image data frame in response to determining that a first frame size of the image data frame at the error handling module is incorrect, wherein the error handling module determines that the first frame size of the image data frame is incorrect based on at least one frame synchronization signal associated with the image data frame; and an image processing unit communicatively coupled to receive the image data frame on which the error concealment operation has been performed from the error handling module, wherein the image processing unit is configured to perform an image processing operation on the received image data frame.

[0005] In another example, an image processing method includes: receiving an image data frame from an external source; performing an error concealment operation on the received image data frame in response to determining that a first frame size of the received image data frame is incorrect, wherein it is determined that the first frame size of the image data frame is incorrect based on at least one frame synchronization signal associated with the image data frame; and performing an image processing operation on the received image data frame on which the error concealment operation has been performed.

[0006] In another example, the method is embodied in computer-executable program code and stored in a non-transitory storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] For a detailed description of various examples, reference will now be made to the accompanying drawings, in which:

[0008] Figure 1 A block diagram of an image processing system according to one or more embodiments is shown.

[0009] Figure 2 A timing diagram showing frame-level error detection and handling operations performed by an image processing system according to one or more embodiments is shown.

[0010] Figure 3 Another timing diagram showing frame-level error detection and handling operations performed by an image processing system according to one or more embodiments is shown.

[0011] Figure 4 A flowchart of an image processing method for frame-level error detection and handling that can be performed by an image processing system according to one or more embodiments is shown.

[0012] Figure 5 Row-level error detection and handling operations performed by an image processing system according to one or more embodiments are shown.

[0013] Figure 6A flowchart of an image processing method for row-level error detection and handling that can be performed by an image processing system according to one or more embodiments is shown.

[0014] Figure 7 An illustrative simplified block diagram of a computing system according to one or more embodiments is shown. Detailed Description

[0015] The present disclosure relates to detecting and effectively handling (e.g., hiding) image frame size and / or synchronization signal protocol errors to minimize frame loss in safety-critical applications. The techniques disclosed herein are aimed at avoiding deadlock situations in modules or units of an image processing pipeline, which are caused by the improper handling of unexpected missing (i.e., undersized) and / or extra (i.e., oversized) pixel data of an image data frame or the improper handling of an otherwise corrupted image frame (e.g., protocol error). When receiving a continuous stream of image frames from a source (e.g., an image sensor) and processing the continuous stream in real-time and on-the-fly in an image processing pipeline, one or more image processing units of the pipeline can be configured to receive these image frames, each having a specific frame size (reference size) and / or conforming to a predetermined protocol (e.g., synchronization signal protocol).

[0016] Various types of errors may be introduced into the bitstream at the source, during transmission, or at the receiver module of the pipeline, and may cause data corruption of the image frame, such that the corrupted image frame no longer conforms to its size and / or the protocol configuration expected by the receiver. That is, uncorrected image sensor data and / or transmission errors in the incoming image data bitstream may result in the captured frame having the wrong frame size due to missing pixel image data and / or faulty frame synchronization information of the input stream and due to data loss after an internal overflow caused by system traffic overload.

[0017] Processing such corrupted image frames at the receiver may lead to various types of problems in the image processing pipeline, some of which may be more critical than others. For example, a corrupted image frame that is directly stored in memory and erroneously becomes larger than its expected size (e.g., an oversized frame) may cause a memory overflow error due to the additional writes to memory caused by the oversized input frame. As another example, a corrupted image frame that undergoes on-the-fly processing by one or more image processing units of the image processing pipeline may unexpectedly cause the receiver image processing unit to enter an unknown state (e.g., a locked state, a deadlock state, etc.) because the frame size and / or protocol is different from what the receiver expects.

[0018] Although corrupted pixel image data only results in the corruption of the image data stored in the memory, frame size or protocol errors during immediate processing may cause many side effects, including memory corruption or the image processing units (and subunits) of the image processing pipeline entering a deadlock state due to faulty frames. Once the module enters a deadlock state, the only option for the main processor implementing the image processing pipeline to resolve the deadlock may be to perform a hardware reset. However, using a hardware reset to restart the image processing unit that processes incoming image data in a real-time application requires the main processor to undergo software interrupt handling. Due to the time required to reset the stalled components (and sub-components) of the image processing pipeline, this interrupt handling may in turn result in the loss of multiple consecutively received frames. In safety-critical applications such as ADAS, infotainment, imaging, computer vision, etc., which rely on a continuous stream of image data to make real-time decisions, such frame loss may be unacceptable.

[0019] To prevent the image processing units of the pipeline from entering a deadlock state and thus prevent the main processor from having to perform a hardware reset, the techniques disclosed herein are designed to effectively and quickly handle faulty (e.g., illegal or incorrect-sized) incoming frames so that even if the image data of the frame is corrupted to the extent that the pixel image data of the frame may no longer be available for subsequent processing, the error detection and handling module implemented in the image processing pipeline can detect the frame size error of the faulty incoming frame and perform appropriate error handling steps to quickly hide the size error during real-time processing so that the "size-corrected" frame can pass through the image processing pipeline without causing any deadlock state.

[0020] For example, when a too-small input frame is detected, the error handling module can operate to maintain the full input frame size expected by the downstream image processing units (i.e., full-size frame emulation for the downstream processing module) by generating the missing pixels and rows specified for the current frame. The handler can discard any remaining incoming data of the faulty too-small frame while filling the remaining part (pixels and rows) of the frame with "virtual" data. As another example, when a too-large input frame is detected, the error handler can discard the remaining incoming data and initialize to wait for the start of the next incoming frame. In either case of size error, a software interrupt can be issued to the main processor to identify the faulty frame saved to the memory (and provide details of the fault to the software layer, e.g., too large, too small, protocol error, etc.). The host can then implement advanced error handling options (e.g., discard the faulty frame and repeat good frames, etc.).

[0021] Other embodiments of the error handling module may perform error handling operations on a stream of image frames on a "line-by-line" basis. That is, the error handling module may process each input line of a frame that has an error in size (either too small or too large) by generating missing pixel data for the current faulty line or discarding the remainder of the incoming pixel data for a line that is too large in size, and then disposing of the next line of pixel data for the frame independently of the previous line. Thus, line-based error handling may copy only the missing pixel data within the current line and dispose of the next line as a normal line (performing any applicable error handling based on the detected size error). This scheme may minimize full-frame rejection due to only local error conditions (e.g., an error in only one line of an otherwise "normal" frame). In the case of line-based error handling, the software interrupt of the main processor may then include additional error information that is reported to assist the software in making a decision on whether to fully reject the error frame based on the error location information, the region of interest information, and a pixel confidence factor that indicates the number of pixels / lines that must be copied in the error frame (e.g., the number of virtual pixels / lines).

[0022] Figure 1 FIG. shows a block diagram of an image processing system 100 according to one or more embodiments. The image processing system 100 may be implemented for a variety of different applications that require real-time processing of an incoming stream of image data (e.g., ADAS, infotainment, imaging, computer vision applications). As Figure 1 shown, a control unit 105 (e.g., an image processing device) that may be implemented as a system-on-chip (SoC) may be used to receive a continuous stream of image data from multiple image data sources using different interfaces. In Figure 1 the embodiment shown, the control unit 105 may receive a video stream from an external source, including a digital parallel video stream received via a video interface 110 and connected to an external video decoder 115 or another processor. The control unit 105 may also receive a continuous stream of image data frames via an image data receiver 120 connected to an image data transmitter 125 through a serial interface.

[0023] In one embodiment, the serial interface connection to transmitter 125 provides high-speed image data transfer via a standardized communication interface. For example, transmitter 125 and receiver 120 may be connected via a Camera Serial Interface (CSI) which is a specification of the Mobile Industry Processor Interface (MIPI) Alliance standard. CSI defines the interface between a transmitter (e.g., an image sensor, a camera, etc.) and control unit 105 (e.g., a main processor, a SoC, etc.). The image data stream received by receiver 120 may be a continuous sequence of image frames having a given resolution (e.g., 1 megapixel or 2 megapixels) (e.g., received at 30 frames per second).

[0024] The image data received via video interface 110 may be at a lower speed compared to the high-speed CSI-MIPI compliant serial image data transfer via receiver 120. The data input via interface 110 may be stored in system memory 130 via system interconnect 135 and memory interface 140. The video data may also be displayed on display 145 via display interface 150. On the other hand, when control unit 105 is used in certain applications (e.g., automotive) where low processing latency is critical, the data received via receiver 120 may be transferred to an image processing pipeline implemented using one or more image processing units 155 before the processed image data frames stream is written out to system memory 130 for further processing or displayed on display panel 145. For example, image processing unit 155 may be an Image Signal Processor (ISP) or a Hardware Accelerator (HWA) module which converts the raw input image data into one of the internal processing color space forms (e.g., YUV or RGB color space format) and / or performs additional on-the-fly color image processing. Although Figure 1 only one image processing unit 155 is shown to implement the image processing pipeline, this is not necessarily the case. More than one image processing unit 155 may form an image processing pipeline on control unit 105.

[0025] When implemented as a CSI-MIPI compliant interface, the serial image data transfer via receiver 120 may include multiple channels of image data streams transferred through the same physical interface. Receiver 120 of control unit 105 may extract pixel image data (e.g., payload data) from the received continuous stream of image data, and extract synchronization signal information (e.g., start / end of a frame, start / end of each row of a frame, etc.) from the input bit stream, and transfer the input stream of the received image data frames to system memory 130 or to the connected image processing unit 155 (e.g., HWA module or ISP module) of the image processing pipeline via a Direct Memory Access (DMA) controller 160.

[0026] Although the bitstream is continuously transmitted from the transmitter 125 to the receiver 120 and then either directly stored in the system memory 130 or post-processed via an image processing pipeline, various error conditions may be introduced into the input image data at various points along the signal chain. For example, due to various fault conditions, errors may be introduced at the image sensor source (e.g., transmitter 125), on the transported bitstream between the transmitter 125 and the receiver 120, at the receiver 120, and / or at one or more image processing units 155 of the image processing pipeline downstream of the receiver 120. Possible sources of error conditions include: glitches in the image sensor or image data transmitter 125 that cause errors in the pixel data and / or synchronization signals at the source, transmission errors from the transmitter 125 to the control unit 105 that are not detected / corrected by the receiver 120, and errors introduced by the receiver 120 due to incorrect user configurations of frame sizes that do not match the received frame size, detection of unsupported formats, and undetected design issues. Other sources of errors in the bitstream can include overflow conditions in the receiver 120 that cause loss of pixel data and / or synchronization signal information, and random transient bit corruptions due to electromagnetic interference in the signal path that, for example, cause incorrect handling of input frames at the receiver 120.

[0027] When the pixel image data has an error (e.g., a faulty pixel value), even if the corresponding image frame may not be available for display or for making an autonomous driving decision, the data can still be passed down to the image processing pipeline for processing without causing a fatal deadlock condition. On the other hand, when the frame data has an incorrect row size or frame size or other protocol error, the subsequent image processing pipeline may not be able to handle the corresponding faulty image frame. Failure to detect and / or correct error conditions (especially with respect to row / frame sizes and synchronization information) can lead to problems such as memory corruption (e.g., overwriting) caused by frames with incorrect sizes (e.g., oversized frames) being directly written into the memory 130. Failure to detect and / or correct error conditions can also lead to frame corruption due to misaligned synchronization signals and, in some cases, deadlock conditions in the image processing pipeline due to unexpected absence of frames and / or incorrect handling of extra pixels / rows.

[0028] Error detection and correction mechanisms such as cyclic redundancy check (CRC) and error correction code (ECC) can be implemented on the transport stream between the transmitter 125 and the receiver 120 to check the payload data and single-bit error correction and 2-bit error detection in the data packet header to protect the integrity of the transmitted data. However, CRC and ECC do not provide protection against frame size and protocol errors, which can originate from the upstream image capture module and can lead to a fatal deadlock condition in the image processing pipeline. Frame size and protocol errors can also originate from modules downstream of the receiver 120. The error detection module 121 and the error handling module 156 can detect and appropriately handle such size or protocol errors.

[0029] The image data receiver 120 extracts image data frames in real time and sends them to the memory 130 and / or the downstream image processing unit 155. The image data receiver 120 can include an error detection module 121 that implements a frame size error detection mechanism based on the actual frame size (e.g., the second frame size) against a user-configured frame reference size. In particular, the error detection module 121 can perform error detection by checking the row width and frame height (actual size) of the input image frame against the expected frame size (e.g., the reference size) according to the user configuration to detect size deviations. The error detection module 121 can also provide memory protection by discarding the extra pixel data of one or more rows when a size-over condition is detected (e.g., extra pixel data in one or more rows or extra rows in a frame). By discarding the data, the error detection module 121 protects the memory 130 from the extra writes that might otherwise be caused by an over-sized input frame.

[0030] When the error detection module 121 detects an error, the software issues an error interrupt to notify the main processor to handle the generated error frame. For example, the main processor may mark the frame that can be saved to the memory 130 as having a size error. Although the error detection module 121 performs frame size error detection and memory protection on frames that are too large in size, the error detection module 121 can still pass frames with short lines (e.g., missing pixel data in one or more lines; short lines) or frames with fewer than the standard number of lines (e.g., missing one or more lines of the frame; short frames) or frames with other protocol errors to the downstream image processing unit 155 for image processing. Such frames passed downstream to the image processing unit 155 can cause the image processing unit 155 to enter an unknown state that results in a locked condition. That is, the image processing unit 155 configured to receive the incoming image data stream may expect an exact frame size and may lack error handling when receiving an input frame with an unexpected frame size. Deviations in the frame size or protocol configuration can lead to corrupted output or even a fatal deadlock condition, which requires a hardware reset of the unit 155 and any downstream modules to resolve the deadlock. For example, the deviation can keep the image processing unit 155 of the image processing pipeline in a deadlock state or an error state. In the deadlock state, the unit 155 is waiting for more data to complete the processing of the current image data frame. In the error state, the unit 155 fails to properly handle the additional (unexpected) image pixel data of one or more lines of the frame.

[0031] To resolve such deadlock states and resume the normal processing of the next image frame in the continuous capture stream of image frames passing through the image pipeline by the unit 155, the main processor may have to perform a hardware reset to clear the affected processing unit(s) 155 and restart the pipeline. Additionally, the deadlock state of the processing unit 155 may only be detected when another error (e.g., an overflow error) is detected and later reported. This may inevitably result in error recovery with multiple frame losses, which is unacceptable for safety-critical automotive applications.

[0032] To effectively handle faulty frames passing through the image processing pipeline, the image processing unit 155 may include an error handling module 156. The error handling module 156 can be configured to perform frame size error detection and error hiding / disposal operations before performing image processing operations at the image processing unit 155. For example, the error handling module 156 can detect protocol violations of the frame synchronization signal and frame size errors for each incoming image frame. The error handling module 156 can further perform frame size error hiding operations on the detected faulty frames (or lines) and perform corresponding error reporting and restart operations.

[0033] To detect faulty frames with illegal frame or line dimensions, the error handling module 156 can perform a size error check on the image data stream of incoming frames by comparing the currently measured frame size (e.g., the first frame size) against a reference frame and line dimensions (e.g., reference dimensions) provided by the user (i.e., user configuration). For example, the error handling module 156 can detect line width errors, including detecting short lines and long lines, where in a short line, a given input line of the frame has less data than the expected number of valid pixels, and in a long line, a given input line of the frame has more data than the expected number of valid pixels. The error handling module 156 can further detect frame height errors, including detecting short frames and long frames, where in a short frame, the current input frame has less data than the expected number of valid lines, and in a long frame, the current input frame has more data than the expected number of valid lines. When the error handling module 156 detects a frame size error (e.g., long line, short line, long frame, short frame, etc.), the error handling module 156 generates an interrupt to indicate the frame size error to the main processor and enters an error handling mode (e.g., error concealment operation).

[0034] To detect frame size errors, when an image data frame is being transferred between the receiver 120 and the image processing unit 155, the error handling module 156 can look for protocol violations in the frame synchronization signals in the incoming frame by using a set of frame synchronization markers (frame synchronization signals) marking the start and end of a line and the start and end of the frame. Synchronization markers that can be defined and used include:

[0035]

[0036]

[0037] When the image processing unit 155 is receiving a continuous stream of image data frames, the error handling module 156 can check each incoming frame to ensure that the synchronization markers or signals transition correctly for each normal frame input. The error handling module 156 can utilize illegal line or frame transitions detected based on the received frame synchronization markers as an early indicator of frame size errors in the current input frame. In one embodiment, the error handling module 156 can use the frame synchronization signals or markers to perform the following checks to detect protocol violations in the frame synchronization signals of incoming frames from the receiver 120 to the image processing unit 155.

[0038] VE without HE Received end of frame (VE) without end of line (HE) VS without HS Received start of frame (VS) without start of line (HS) VS-VS check Start of frame (VS) not paired with end of frame (VE) HS-HS check Start of line (HS) not paired with end of line (HE) HE-HE check End of line (HE) followed by second end of line (HE) VE-VE check End of frame (VE) followed by second end of frame (VE)

[0039] When one or more frame synchronization signal protocol violations are detected, the error handling module 156 generates an interrupt and reports the detected violations (e.g., protocol violations or errors). Protocol errors (e.g., missing synchronization signals, misaligned synchronization signals, etc.) typically require discarding the frame. To ensure that the frame size remains at the expected level (reference size) for downstream modules, the error handling module 156 can further use the synchronization signals to detect illegal frame sizes. That is, the synchronization signals are used to diagnose frame size errors and report them to the main processor. After detecting a frame size error based on the frame synchronization signal, the error handling module 156 enters an error handling mode to perform a fast error concealment operation, thereby preventing a deadlock state in the image processing unit 155 or beyond. The image processing unit 155 expects an exact number of pixels and rows of data for each received input image frame to operate properly (i.e., normal operation without errors). In addition, the DMA controller 165 (and task controller) responsible for handling data transfer between one or more image processing units 155 that handle the image processing pipeline and the system memory 130 can rely on the image processing unit 155 to output each processed frame with an exact (as expected according to user configuration) frame size. When a frame size error is detected, the error handling module 156 immediately switches to an error recovery mode to perform error handling (error concealment operation) for undersized or oversized input frames. Figure 2 and Figure 3 illustrates the undersized input frame handling performed by the error handling module 156 after detecting an illegal frame size for the current frame.

[0040] As Figure 2 shown in the timing diagram 200 in, when the error handling module 156 detects an undersized input frame (e.g., short frame, short row) while processing input frame 1 due to receiving the frame synchronization marker VE indicating the end of frame 1 earlier than expected (e.g., the first frame or row size is smaller than the reference frame or row size) (1 - error detected), the error handling module 156 enters the error handling mode for the current frame (error concealment operation), discards any remaining input data of the current frame received from the receiver 120 (2 - discard input data from the receiver; e.g., discard the remaining part of the image data frame) and generates "virtual" data to complete the current frame 1 (3 - short frame correction). Thus, in the error handling mode, the error handling module 156 operates to maintain the full (or expected) input frame size of frame 1 for processing by the image processing unit 155.

[0041] The error handling module 156 generates a complete frame for the image processing unit 155 by generating data for the missing pixels and lines specified for the current frame. Since the frame is considered an error frame, the error handling module 156 can repeat the image data of the last known good pixel to complete the frame. In other embodiments, other data from the frame or data other than the current frame data can be used to generate virtual data. The generation of virtual data is performed at the full processing clock speed, which is typically much faster than the incoming data from the receiver 120 at the pixel clock rate (VP.PCLK). Therefore, the error handler can complete the current frame that is considered an error or faulty frame as soon as possible and be ready to start the next frame (frame 2) at the next corresponding initialization signal (4 - check the VS signal flag of the input data from the receiver and normally capture the next frame; and 5 - capture frame 3).

[0042] As Figure 2 shown in the exemplary timing diagram 200 of, by performing short frame error correction in accordance with the error handling and concealment operations of the error handling module 156, it is possible to ensure that a frame of "normal" size is received at the image processing unit 155 (even if the normal size frame has bad pixel data), and thus, it is possible to prevent the image processing unit 155 from entering any unknown or deadlock state. In Figure 2 the case where, if it were not for the error handling of the undersized input frame by the error handling module 156, the image processing unit 155 might receive short frame 1 and enter a deadlock state, which would require the main processor to perform software interrupt handling and perform a hardware reset of the image processing unit 155 to restart the unit and any sub-units. Such additional steps may cause delays, and the delays will result in the loss of one or more additional image frames (e.g., frame 2, frame 3, etc.) from the continuous stream of input image frames.

[0043] Figure 3 The timing diagram shown illustrates an alternative to the short frame error handling (undersized input frame handling; error concealment operation) performed by the error handling module 156 in one or more embodiments. While completing the current error frame by performing the error concealment operation and inserting virtual data, the error handling module 156 can discard the input from the receiver 120 to clear any remaining error frame data. As Figure 3 shown and in the same case as the timing diagram of Figure 2 when the error handling module 156 detects an undersized input frame (e.g., short frame, short line) due to receiving the frame synchronization marker VE indicating the end of frame 1 earlier than expected while processing input frame 1 (1 - detect an error), the error handling module 156 enters the error handling mode for the current frame, discards any remaining input data of the current frame received from the receiver 120 (2 - discard the input data from the receiver) and generates "virtual" data to complete the current frame 1 (3 - short frame correction).

[0044] In such a scenario, if a new frame (frame 2) starts to be received on the continuous image data stream received from the receiver 120 while the clearing of the current frame 1 is still valid, the frame synchronization marker VS indicating the start of frame 2 can arrive before the error handling module 156 initializes the DMA and is ready to start frame 2 (4 - input frame 2 skip detected). In this case, frame 2 can also be skipped, and the error handling module 156 remains in the error handling mode and discards any remaining input data of the current frame 2 received from the receiver 120, and issues an interrupt to notify the main controller that an "extra" frame (i.e., frame 2) is lost during error recovery. Under normal circumstances, when performing error handling and concealment operations on short frames, this kind of "extra" frame loss does not occur because of the vertical blanking period between incoming frames and because the pixel clock rate (i.e., the rate at which new frames enter) is much slower compared to the full processing clock speed for discarding error frames and generating virtual data.

[0045] Return Figure 1 , when the error handling module 156 detects an oversized input frame (e.g., long frame, long line) (e.g., the size of the first frame or line is greater than the reference frame or line size; for at least one of multiple lines of an image data frame, the number of pixels for which the error handling module 156 receives data is greater than the reference number of pixels per line of the image data frame (long line); for an image data frame, the number of lines for which the error handling module 156 receives data is greater than the reference number of lines of the image data frame (long frame)) while processing the current input frame received from the receiver 120, the error handling module 156 enters the error handling mode by discarding all remaining incoming streams of the current oversized image frame to perform oversized input frame handling (error concealment operation), and enters the error frame fast completion mode by generating the remaining pixel / line data to complete the oversized frame as soon as possible using virtual data. Since the frame is regarded as an error frame, the error handling module 156 can repeat the image data of the last known good pixel to complete the frame according to its normal expected size. In other embodiments, other data from the frame or data other than the current frame data can be used to generate virtual data and complete the frame.

[0046] The generation of virtual data is performed at the full processing clock speed, which is generally much faster than the incoming data from the receiver 120 at the pixel clock rate (VP.PCLK). Since error handling operations are performed on the oversized frame, a complete frame with the normal expected size can be provided to the image processing unit 155. Waiting to complete receiving the oversized frame and then only discarding the data of extra pixels and lines may take an unknown period of time. In addition, since the error condition in the remaining part of the image data to be received for the current oversized error frame is unknown, waiting to complete receiving the oversized frame may cause the system to enter an unknown state.

[0047] Once the error handler determines that the current frame is an over-sized frame of a fault, the error handling module 156 implements the shortest possible completion time of the current fault frame by completing the current over-sized frame as soon as possible (e.g., by discarding the incoming data of the current frame and quickly generating virtual data to fill the frame), so that the error handling module 156 can complete initializing the DMA and get ready for the start of a new frame before the frame synchronization marker VS indicating the start of the next frame arrives. The error concealment operation of the error handling module 156, which enables the image processing unit 155 to complete the current frame with the correct frame size, prevents any deadlock condition of the control logic in the rest of the image processing unit 155 or other downstream modules of the image processing pipeline, and enables seamless error recovery with minimal frame loss and without resetting the hardware (subsystem) due to frame size errors.

[0048] Figure 4 A flowchart of an image processing method 400 that can be executed by the image processing system 100 according to one or more embodiments is shown. As Figure 4 shown, method 400 begins at block 405, where the image data receiver 120 of the control unit 105 receives an image data frame. As previously explained, the receiver 120 can receive a continuous stream of image data frames from an external source (e.g., the image data transmitter 125) that complies with a predetermined communication standard (e.g., the CSI-MIPI-standard). At block 410, the error detection module 121 of the receiver 120 can perform an error detection operation on the received image data frame by checking the line width of one or more lines of the input image frame and the frame height of the frame (actual received size; second frame size) against the expected line size and frame size according to user configuration (reference size) to detect size deviations. If the error detection module 121 detects a frame size error at block 410 (yes at block 410), the operation proceeds to block 415, where the error detection module 121 can issue an error interrupt (e.g., a software interrupt) to notify the main processor that the currently received image frame has a frame size error. Then, the main processor can perform software interrupt handling and disposal based on the received interrupt to implement advanced error handling options (e.g., discarding the error frame).

[0049] At block 420, error detection module 121 checks to determine whether the current frame with a frame size error detected at block 410 is a frame that is too large in size. That is, error detection module 121 determines whether the data corresponding to the current image frame received by receiver 120 includes data for more pixels than a specified number of rows or data for more rows than a specified number of frames. If error detection module 121 detects that the frame is too large in size (a "yes" at block 420), then at block 425, error detection module 121 discards the data for the extra pixels and / or rows. For example, once the data for a specified number of pixels for a specified number of rows has been received, error detection module 121 can clear the remainder of the data received for the current image frame. Thus, at block 425, error detection module 121 provides memory protection by discarding the extra pixel data for one or more rows when a condition of being too large in size is detected. At block 425, if the frame is not a frame that is too large in size (a "no" at block 420) (e.g., the frame is a short frame or a frame that is too small in size that is missing one or more pixels of image data for one or more rows; i.e., a short frame or short row), then error detection module 121 can pass only the frame that is too small in size to downstream modules (e.g., error handling module 156, image processing unit 155) for further processing.

[0050] If error detection module 121 does not detect a frame size error at block 410 (a "no" at block 410), or if error detection module 121 detects that the frame is not too large in size (a "no" at block 420), then the operation proceeds to block 430, where receiver 120 (or error detection module 121) determines whether the current image frame that has undergone error detection and memory protection is to be processed immediately in real time by the image processing pipeline implemented on control unit 105. That is, at block 430, if the frame is not to undergo processing at the image processing pipeline, then receiver 120 can route the received image frame directly to DMA controller 160 for storage on system memory 130 (a "no" at block 430; block 460). Alternatively, if it is determined at block 430 that the frame is to be processed at the image processing pipeline, then receiver 120 can route the received image frame to error handling module 156 for image processing by image processing unit 155 (a "yes" at block 430).

[0051] If the frame is to be processed immediately, then method 400 proceeds to block 435, where the image frame is transmitted to the image processing pipeline implemented on control unit 105. That is, at block 435, receiver 120 (or error detection module 121) transmits the currently received image frame to error handling module 156, which in turn transmits the frame to one or more image processing units 155 that make up the image processing pipeline after error detection and concealment operations.

[0052] Then, method 400 proceeds to block 440, where error handling module 156, which has received the current image frame from receiver 120, detects whether the currently received frame violates any of the frame synchronization signal protocols. At block 440, error handling module 156 uses the frame synchronization markers described previously that are valid during the start and end of each line and the start and end of each frame to check whether any of the synchronization signals received at module 156 are missing or misaligned (e.g., VE without HE, VS without HS, VS-VS check, etc.). If error handling module 156 detects any violation of the frame synchronization signal protocol at block 440 (yes at block 440), then error handling module 156 issues an interrupt to the main processor indicating a protocol error for the current frame (block 442). A protocol error typically causes the frame to be discarded by the main processor during subsequent processing. To prevent downstream modules from entering an unknown state and causing a deadlock condition, at block 444, error handling module 156 further checks whether the current frame has an illegal frame size. At block 444, error handling module 156 checks the incoming video stream against a reference size to detect line width errors (e.g., long lines, short lines) and frame height errors (e.g., long frames, short frames) based on the synchronization markers.

[0053] If error handling module 156 detects an illegal frame size at block 444 based on the received synchronization signals and the user-configured reference size (yes at block 444), then error handling module 156 enters an error handling mode to quickly hide the size error and maintain the frame size expected by the downstream modules. This also prevents subsequent frame loss by avoiding a deadlock condition in the image processing pipeline. In the error handling mode, error handling module 156 performs under-sized input frame handling (block 445A) or over-sized input frame handling (block 445B) based on whether the frame with the illegal frame size detected at block 444 is under-sized (e.g., short frame, short line) or over-sized (e.g., long frame, long line).

[0054] At block 450, error handling module 156 issues an interrupt to the main processor indicating a frame size error for the current faulty frame for which error handling and hiding operations are performed. After the error hiding operation, the faulty frame (whose frame size is now the expected size) is passed to the downstream modules of the image processing pipeline. On the other hand, if error handling module 156 does not detect any illegal frame size for the current frame (no at block 444), then error handling module 156 transmits the error-free frame downstream to the image processing pipeline (e.g., image processing unit 155) for image processing.

[0055] At block 455, the current frame is transmitted to a downstream module of the image processing pipeline (e.g., image processing unit 155) for image processing. Since the frame transmitted at block 455 and received by the image processing unit 155 has the correct size (i.e., the size expected by the modules of the pipeline), deadlock conditions in the control logic of the pipeline can be prevented and seamless error recovery can be achieved without hardware reset, thereby minimizing frame loss, which is extremely important in safety-critical applications (such as ADAS, computer vision). At block 460, the image frame to be processed by the image processing pipeline is saved to the system memory 130. Then, method 400 proceeds to block 465, where the operations of blocks 405 - 465 are repeated for each incoming image frame received by the control unit 105 from an external image data source (e.g., transmitter 125).

[0056] Figures 1-4 The disclosure in [references] relates to frame (or frame-level) based error handling and concealment, where once the error handling module 156 detects a size error at any point in a given image frame, the remaining portion of the error frame is discarded and filled with virtual data, and an interrupt is issued to notify the main processor of the error frame. That is, in frame (or frame-level) based error handling as described above, once an error is detected in a given frame, the frame is marked as a faulty frame and the entire frame can be cleared (e.g., if the size error is in the first row of the frame), even if only a small portion of the frame has an error. In other words, in frame-based error handling, even in the case of a local error condition, the data of all remaining frame pixels after the first error detection are cleared. For example, pixel loss due to momentary FIFO overflow may result in a single short row in an otherwise good frame. However, under frame-based error handling, when a single short row is detected, the remaining portion of the frame pixel data can be cleared and replaced with virtual data. In the case where a single short row appears at the end of the frame (e.g., in the last row of the frame), the frame can still be marked as a faulty frame and discarded based on an interrupt to the main processor. However, discarding the entire frame based on a local error condition can be unnecessary or wasteful.

[0057] In Figure 5 and Figure 6In the illustrated embodiment, the error handling and concealment operations performed by the error handling module may be performed on a line-by-line basis (i.e., line-based or line-level error handling) to minimize full-frame rejection (i.e., discarding the entire frame) for local error conditions. Line-based error handling only copies the data of the missing pixels within the current line of the current frame in which an error has been detected (e.g., based on a corresponding frame synchronization signal protocol violation), and, if applicable, treats the next line of the current frame as a normal line using the same error handling and concealment operations. Then, the line-based error handling module may perform an error information report that provides additional information (e.g., error location information, frame confidence value or factor information, etc.) along with a software interrupt to the main processor to assist the processor in making a frame rejection decision (e.g., not rejecting the frame if the error is local and otherwise outside the region of interest of the frame). For example, a status register associated with the software interrupt may indicate additional information about the error type (e.g., frame or line with too small a size, frame or line with too large a size, error location information, etc.). A protocol error detected based on a synchronization signal protocol violation (e.g., synchronization signal misalignment) is fatal and typically renders the frame useless. However, in the case of a local frame size error (e.g., a frame with an illegal frame size), it may be possible to use the frame (i.e., choose not to discard the faulty frame) even if there is an error if the software determines (based on error information read by the software from an additional register associated with the error interrupt) that the region of interest corresponding to the frame with the local frame size error is error-free (or has error pixel data below a predetermined threshold).

[0058] The line-based error handling module may track the error location (e.g., which lines of the current frame are faulty) and report the error region or location information to the host when an error interrupt is issued. The host may then use this information to determine whether to use or discard the current frame data. For example, if the pixels or lines with size errors are outside the vertical and / or horizontal regions of the region of interest, the error may be ignored. The line-based error handling module may also track the number of pixels and / or lines of the current frame that must be copied and may report the error count information to the host when an error interrupt is issued. The module may also provide a confidence value (e.g., frame confidence level or confidence value) along with the interrupt to the host based on the number of pixels and / or lines of the current frame that must be copied. The host may then use this information to better determine whether to completely reject the current faulty image frame.

[0059] As Figure 5As shown, a line-level error handling module (not shown) performs error detection and handling / hiding operations on a line-by-line basis. When the error handler detects a short line at line n+1 based on the corresponding received line synchronization signal, pixel image data, and reference line size, the line-level error handling module performs line-level error detection and handling (input line error handling with too small a size; error hiding operation) operations for line n+1 of the current frame. In Figure 5 In the example of Figure 5 , when the line-level error handling module detects an input frame with too small a size (e.g., short line n+1) due to receiving the frame synchronization marker HE indicating the end of line n+1 earlier than expected while processing the current input, the line-level error handling module enters the error handling mode for the current line n+1 and generates "virtual" data to complete the current line n+1 using the corresponding frame synchronization signal. Therefore, in the line-level error handling mode, the error handler operates to maintain the full (or expected) input line size of line n+1 for downstream module processing. In one embodiment, the line-level error handling module may repeat the image data of the last known good pixel of line n+1 to complete the line. In other embodiments, other data from line n+1 or data other than the current line or frame may be used to generate the virtual data. The generation of the virtual data is performed at the full processing clock speed, which is typically much faster than the incoming data from the receiver at the pixel clock rate (VP.PCLK). Therefore, the error handler can complete the current line n+1, which is considered an error or faulty line, as soon as possible and prepare for the next line n+2.

[0060] In Figure 1 In Figure 1 , if applicable, after completing the error handling and disposal of the short line n+1, the line-level error handler may start processing the next line n+2 of the current frame as a normal line using the same error handling and hiding operations. In the case of frame-based error handling, when the error handling module 156 detects a short line at line n+1, all remaining lines starting from line n+1 (including line n+2) may be cleared and filled with virtual data. On the other hand, in the case of line-level error handling, the next line is considered a normal line, thus minimizing the discard of "good" pixel data in the case of a frame with local errors (e.g., errors in only one or a few lines).

[0061] Return Figure 5, when the error handler detects a long line at line n+m based on the corresponding received line synchronization signal, pixel image data, and reference line size, the line-level error handling module performs line-level error detection and handling operations (oversized input line error handling; error concealment operation) for line n+m of the current frame, where the line-level error handling module discards all remaining incoming streams of the current oversized image line n+m and enters the error frame fast completion mode by generating remaining pixel data to complete the oversized current line n+m as soon as possible with virtual data. The error handler can repeat the image data of the last known good pixel to complete line n+m according to its normal expected size. In other embodiments, other data from line n+m or data other than the current line or frame can be used to generate virtual data and complete line n+m.

[0062] Due to the error handling operations performed for the oversized line n+m, a complete line with the normal expected size can be provided to downstream modules. Waiting to complete receiving the oversized line n+m and then discarding the extra pixels may take an unknown period of time. Additionally, since the error condition in the remaining part of the line n+m image data to be received for the current error line is unknown, waiting to complete receiving the oversized line data may cause the system to enter an unknown error state. Once the error handler determines that the current line n+m is a faulty oversized line, by completing the current oversized line as soon as possible (e.g., by discarding the incoming data of the current line and quickly generating virtual data to fill the line), the error handler achieves the shortest possible completion time for the current line, such that it can be ready to start the new line n+m+1 of the current frame and receive line n+m+1 without errors.

[0063] In addition, as Figure 5 shown, when an interruption regarding the current faulty frame is issued to the main processor, the line-level error handling module can report the error pixel count (e.g., the number of pixels that must be copied for the short line n+1, the number of pixels that must be discarded for the long line n+m+1), as well as the error location or region within the current frame (e.g., the error start line n+1 and the error end line n+m+1). The handler can similarly perform processing on subsequent lines of the current frame, performing any error detection and handling on each line as needed.

[0064] Figure 6 FIG. shows a flowchart of an image processing method 600 that can be performed by an image processing system including a line-based error handling module according to one or more embodiments. Only those steps of the line-level error handling method 600 that are different from the Figure 4 frame-level error handling method 400 are shown in Figure 6 and are described in detail below. Similar to Figure 4The steps in the frame-level error handling method 400 thereof are omitted. In at least some embodiments, regarding Figure 4 and Figure 6 both the frame-level size error check and line-based error recovery / hiding described are performed simultaneously.

[0065] As Figure 6 shown, at block 630, the line-level error handling module of the current image frame has received data of the pixel lines of the current frame one line at a time from the receiver of the control unit 105. For the line obtained at block 630, the line-level error handling module detects at block 640 whether the current received line of the current frame violates any of the frame synchronization signal protocols. At block 640, as described above, the error handler uses the frame synchronization markers valid during the start and end of each line to check whether the synchronization signal is missing or misaligned (e.g., HS-HS check, HE-HE check, etc.). If the line-level error handling module detects any violation of the frame synchronization signal protocol at block 640 (yes at block 640), then the module flags the current frame to issue an interrupt to the main processor indicating a protocol error for the current frame and line (block 642).

[0066] To ensure that the downstream modules receive frames and lines of the correct size, at block 644, the line-level error handling module further checks whether the current line has an illegal line size. At block 644, the line-level error handling module checks the incoming video stream (the actual received line size) against a reference size to detect line width errors (e.g., long lines, short lines) based on the synchronization markers.

[0067] If the line-level error handling module detects an illegal line size at block 644 based on the received synchronization signal and the user-configured reference size (yes at block 644), then the error handling module enters an error handling mode to quickly hide the line size error and maintain the line and frame sizes expected by the downstream modules. This also prevents subsequent frame loss by avoiding deadlock conditions in the image processing pipeline. In the line-level error handling mode, the module performs undersized input line handling (block 645A) or oversized input line handling (block 645B) based on whether the line with the illegal line size detected at block 644 is undersized (e.g., short line) or oversized (e.g., long line). At block 650, the error handling module flags the current frame to issue an interrupt to the main processor indicating a line size error. At block 655, the error handling module determines whether there are more lines in the current input image frame, and if so, starts processing the next line of the current frame.

[0068] As Figure 6As shown, the operations corresponding to block 630 - block 655 are performed repeatedly and separately for each row of the frame. If, at block 655, the error handler determines that there are no more rows in the current frame for error handling and concealment processing ( "No" at block 655), then method 600 proceeds to block 660, where an interrupt is issued to the main processor based on flags that may be set for one or more rows of the current input frame at block 642 and / or block 650. Thus, at block 660, the row-level error handling module issues an interrupt to the main processor to indicate that the current frame for which row-level error handling and concealment operations have been performed (for at least one row of the frame) is a faulty frame. At block 665, the row-level error handling module may further transmit error report information (based on flags that may be set for one or more rows of the current input frame at block 642 and / or at block 650) to the main processor, such as error location and frame confidence values, to assist the main processor in determining whether to discard the frame entirely or use it for subsequent processing based on the region-of-interest data corresponding to the frame.

[0069] An image processing system with frame-level or row-level error detection and recovery operations as disclosed herein provides a hardware-based solution to prevent deadlock situations by maintaining full frame size and concealing frame errors. The solution provides frame-level and row-level handling of frame errors and graceful error recovery in hardware (or software or both), thereby minimizing the number of lost frames compared to the case of performing a reset issued by software in response to an interrupt for error recovery. Additionally, in row-level error detection and handling operations, error information (e.g., error location information) is provided to the host to minimize unnecessary frame rejection. By implementing error detection (and concealment) operations at multiple layers (e.g., at the receiver, at downstream image processing units in the image processing pipeline), protection along the entire image data signal path can be ensured.

[0070] Figure 7 An illustrative simplified block diagram of a computing system 700 in accordance with one or more embodiments is shown. Computing system 700 may correspond to a computer and / or any other computing device (such as a workstation, server, mainframe, supercomputer, and / or portable computing device), or may be a part thereof. Refer to Figure 1, the computing system 700 may correspond to the control unit 105. The computing system 700 includes a processor 702, which may include one or more processors (CPU, GPU, or other types of integrated circuits) and / or other types of system-on-chip (SoC) components to process image data. As an example, the processor 702 may include processing components (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs)) and memory for performing various image processing operations associated with the image or pixel processing pipeline as described herein. The processor 702 may communicate (e.g., via the system bus 770) and / or provide instructions to other components within the computing system 700, such as the input interface 704, the output interface 706, and / or the memory 708. In one embodiment, the processor 702 may include one or more multi-core processors and / or a storage medium (e.g., cache memory) that serves as a buffer and / or storage device for data. Although Figure 7 it is shown that the processor 702 may be a single processor, the processor 702 is not limited thereto, but may represent multiple processors.

[0071] Figure 7 it is shown that the memory 708 may be operably coupled to the processor 702. The memory 708 may be a non-transitory medium configured to store various types of data. For example, the memory 708 may include one or more memory devices, which include auxiliary storage devices, read-only memory (ROM), and / or random access memory (RAM). Auxiliary storage devices typically consist of one or more disk drives, optical drives, solid-state drives (SSDs), and / or tape drives and are used for non-volatile storage of data. In some cases, if the allocated RAM is not large enough to hold all working data, the auxiliary storage device may be used to store overflow data. The auxiliary storage device may also be used to store programs when they are selected to be loaded into RAM for execution. ROM is used to store instructions and possibly data that are read during program execution. ROM is a non-volatile memory device and typically has a small memory capacity compared to the larger memory capacity of the auxiliary storage device. RAM is used to store volatile data and may also store instructions.

[0072] Memory 708 may be used to hold instructions and logic for implementing the various embodiments described herein. In an embodiment, memory 708 may include error detection and error handling / hiding logic that may be accessed and implemented by processor 702. Additionally or alternatively, the logic may be stored and accessed within a memory (e.g., cache memory) embedded in processor 702 or implemented in hardware or some combination of hardware and software. In one embodiment, memory 708 may interface with system bus 770 (e.g., a computer bus) to communicate and / or transfer information stored in memory 708 to processor 702 during the execution of a software program (such as a software application including program code) and / or computer-executable process steps incorporating the functions described herein.

[0073] One of ordinary skill in the art will recognize that software programs may be developed, coded, and compiled in a variety of computing languages for a variety of software platforms and / or operating systems and subsequently loaded and executed by processor 702. In one embodiment, the compilation process of a software program may transform program code written in one programming language into another computer language such that processor 702 can execute the programming code. For example, the compilation process of a software program may generate an executable program that provides processor 702 with encoded instructions (e.g., machine code instructions) to perform a specific, non-general, particular computing function, such as performing the error detection and handling / hiding operations described herein.

[0074] After the compilation process, the error detection and handling / hiding operations described herein may be loaded from a storage device (e.g., memory 708, one or more storage media, removable media drives, and / or other storage devices) into processor 702 and / or embedded in processor 702 as computer-executable instructions or process steps. Processor 702 may execute the stored instructions or process steps to perform the instructions or process steps that transform computing system 700 into a non-general, particular, specially programmed machine or apparatus. During the execution of the computer-executable instructions or process steps, processor 702 may access stored data, such as data stored by a storage device, to direct one or more components within computing system 700.

[0075] Alternatively, one of ordinary skill in the art will recognize that the stored instructions can be transformed and implemented as hardware customized for a particular use (e.g., an SOC for ADAS, infotainment, imaging, and computer vision applications), rather than programming and / or loading the executable instructions onto the memory 708 and / or the processor 702 to form a non - general - purpose, specific machine or apparatus. In one embodiment, implementing operations (such as the error detection and handling / hiding operations described herein) by loading executable software into a computing device can be transformed into a hardware implementation by well - known design rules. For example, the compilation process of a software program can build a sequence of instruction bits that control and arrange a sequence of control gate - level components that write data to buses, write latches, and registers across channels, memory, and / or other components of the processor 702 and / or the memory 708. The compilation of an image - processing operation can produce gate - level components with fixed relationships designed to perform specific, non - general - purpose, particular computational functions.

[0076] The decision between implementing a concept in software and hardware can depend on multiple design choices, which include the stability of the design, the number of units to be produced, and the issues involved in the transformation from the software domain to the hardware domain. Generally, a design can be developed and tested in software form and then transformed into an equivalent hardware implementation in an ASIC or other application - specific hardware by well - known design rules that hard - wire the instructions or process steps of the software. A non - general - purpose, specific, specially - programmed machine or apparatus operates in the same manner as a machine controlled by a new ASIC. Similarly, a computing device (e.g., a computer) programmed and / or loaded with executable instructions or process steps should be considered a non - general - purpose, specific, specially - programmed machine or apparatus.

[0077] Figure 7It is also shown that the processor 702 can be operatively coupled to an input interface 704 configured to receive input sensor data and / or directly reported data, and an output interface 706 configured to output and / or display, for example, image data. The input interface 704 can be configured to obtain input sensor data and / or directly reported data and / or other information via a cable, connector, wireless connection, and / or other communication protocols. In one embodiment, the input interface 704 can be a network interface that includes a plurality of ports configured to receive and / or transmit data via a network. In particular, the network interface can transmit data via a wired link, wireless link, and / or logical link. Other examples of the input interface 704 can be a Universal Serial Bus (USB) interface, CD-ROM, DVD-ROM, and / or a connection to one or more sensors. The output interface 706 can include one or more connections for a graphical display (such as a monitor), a printing device that produces a hard copy of the generated results, and / or a plurality of ports that transmit data via a cable, connector, wireless connection, and / or other communication protocols.

[0078] Figure 7 It is also shown that the processor 702 can be operatively coupled to one or more device sensors 715. The device sensors 715 can include, but are not limited to, an optical sensor array, a sound sensor, an image sensor, a CMOS sensor, an ambient light sensor, a thermal sensor, a light sensor, a differential light sensor, a pixel array, a micro-pixel array, etc. Those of ordinary skill in the art will appreciate that the computing system 700 can include other components well known in the art that are Figure 7 not explicitly shown herein, such as other sensors, a power supply, and / or an analog-to-digital converter.

[0079] It should be understood that the above description is intended to be illustrative and not restrictive. The presented material is provided to enable those skilled in the art to make and use the claimed subject matter herein and is provided in the context of specific embodiments, and variations of these embodiments will be apparent to those skilled in the art (e.g., some of the disclosed embodiments can be used in combination with each other). Additionally, some of the described operations (image processing methods 400 and 600) can perform their respective steps in a different order than presented herein or in combination with other steps presented herein. Moreover, some of the disclosed steps can be omitted. More generally, if there is hardware support, some of the operations described in Figures 1-6 connection can be performed in parallel.

[0080] References to "an embodiment" or "embodiments" in the present disclosure mean that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least one embodiment of the present disclosure, and repeated references to "an embodiment" or "embodiments" should not be construed as necessarily referring to the same embodiment. Unless explicitly so defined, the terms "a", "an", and "the" are not intended to refer to a single entity but rather include a general category that can be illustrated by a particular example. Thus, the use of the term "a" or "an" can denote any quantity of at least one, including "one", "one or more", "at least one", and "one or more than one". The term "or" refers to any alternative and any combination of alternatives, including all alternatives, unless the alternatives are explicitly indicated as mutually exclusive. When combined with a list of items, the phrase "at least one of... " refers to a single item in the list or any combination of items in the list. Unless explicitly so defined, this phrase does not require all of the listed items.

[0081] At least one embodiment is disclosed, and variations, combinations, and / or modifications of (one or more) embodiments and / or features of (one or more) embodiments made by a person of ordinary skill in the art are within the scope of the present disclosure. Alternative embodiments resulting from the combination, combination, and / or omission of features of (one or more) embodiments are also within the scope of the present disclosure. In cases where numerical ranges or limitations are explicitly stated, such explicit ranges or limitations can be understood to include iterative ranges or limitations of similar magnitudes that fall within the explicitly stated ranges or limitations (e.g., from about 1 to about 10 includes 2, 3, 4, etc.; greater than 0.10 includes 0.11, 0.12, 0.13, etc.). Unless otherwise stated, the use of the term "about" means ±10% of the subsequent number.

[0082] After reading the above description, many other embodiments will be apparent to those skilled in the art. Accordingly, the scope of the present disclosure should be determined with reference to the appended claims and the full scope of equivalents to which these claims are entitled. In the appended claims, the terms "including" and "in which" are used as plain English equivalents of the corresponding terms "comprising" and "wherein".

[0083] The term "coupled" is used throughout the specification. This term can encompass a connection, communication, or signal path that achieves a functional relationship consistent with the description of the present disclosure. For example, if device A generates a signal to control device B to perform an action, then in a first example, device A is coupled to device B, or in a second example, if an intermediate component C does not significantly change the functional relationship between device A and device B, then device A is coupled to device B through this intermediate component C, such that device A controls device B via the control signal generated by device A.

[0084] Throughout the specification and claims, certain terms are used to refer to specific system components. As will be understood by those skilled in the art, different parties may refer to components by different names. This document is not intended to distinguish between components that have different names but the same function. In this disclosure and claims, the terms "comprising" and "including" are used in an open-ended manner and should thus be interpreted to mean "including but not limited to...". Additionally, the term "coupled" or "coupling" is intended to mean either an indirect or direct wired or wireless connection. Thus, if a first device is coupled to a second device, that connection may be through a direct connection or an indirect connection via other devices and connections. The recitation "based on" is intended to mean "at least in part based on". Thus, if X is based on Y, then X may be a function of Y and any number of other factors. The recitation of "about" before a recited value is intended to cover all values within a range of ±10% of that value.

[0085] The foregoing discussion is intended to illustrate the principles of the disclosure and various embodiments. Once the foregoing disclosure is fully understood, many variations and modifications will become obvious to those skilled in the art. The appended claims are intended to be construed to include all such variations and modifications.

Claims

1. An image processing apparatus, comprising: an error handling circuit configured to receive an image data frame; determine that a size of the image data frame is incorrect based on a first marker indicating a start of the image data frame and further based on a second marker indicating an end of the image data frame; and perform an error concealment operation on the image data frame in response to determining that the size of the image data frame is incorrect, the error concealment operation including discarding data of a remaining portion of the image data frame received by the error handling circuit; and when the image data frame is determined to be too small in size, determine that pixel data of at least one row among multiple rows of the image data frame is missing, and generate virtual data in response to the missing pixel data, and when the image data frame is determined to be too large in size, generate virtual data to complete the image data frame; and an image processing circuit communicatively coupled to receive the image data frame on which the error concealment operation has been performed from the error handling circuit, and perform an image processing operation on the image data frame.

2. The image processing apparatus according to claim 1, wherein, When performing the error concealment operation when the image data frame is determined to be too small in size and pixel data of at least one row among multiple rows of the image data frame is determined to be missing, the error handling circuit is further configured to: generate the virtual data for the missing pixel data such that the size of the image data frame becomes equal to a reference size.

3. The image processing apparatus according to claim 2, wherein the error handling circuit generates the virtual data by repeating data of the last received good pixels of the image data frame.

4. The image processing apparatus according to claim 1, wherein in order to determine that the image data frame is too large in size, the error handling circuit is further configured to determine that the size of the image data frame is greater than a reference size, and in response to determining that the size of the image data frame is greater than the reference size, the error handling circuit is further configured to: generate the virtual data to complete the image data frame such that the size of the completed image data frame becomes equal to the reference size.

5. The image processing apparatus according to claim 4, wherein in order to determine that the size of the image data frame is greater than the reference size, the error handling circuit is configured to, for at least one row among multiple rows of the image data frame, determine that a number of pixels whose data is received by the error handling circuit is greater than a reference number of pixels for each row of the image data frame.

6. The image processing apparatus according to claim 4, wherein in order to determine that the size of the image data frame is greater than the reference size, the error handling circuit is configured to determine that a number of rows of the image data frame whose data is received by the error handling circuit is greater than a reference number of rows of the image data frame.

7. The image processing apparatus according to claim 1, wherein the error handling circuit is further configured to detect a frame synchronization signal protocol violation for the image data frame based on at least one of the first flag, the second flag, or the third flag, and Wherein the first mark includes: a vertical start signal that is valid when the error handling circuit receives data of a first pixel of the image data frame, the second flag includes a vertical end signal that is valid when the error handling circuit receives data of a last pixel of the image data frame, and the third flag includes a horizontal start signal that is valid when the error handling circuit receives data of a first pixel of each row of multiple rows of the image data frame, or a horizontal end signal that is valid when the error handling circuit receives data of a last pixel of each row of the multiple rows of the image data frame.

8. The image processing apparatus according to claim 7, wherein the error handling circuit is further configured to interrupt the main processor in response to detecting the frame synchronization signal protocol violation for the image data frame.

9. The image processing apparatus according to claim 1, further comprising an image data receiver circuit communicatively coupled to the error handling circuit, wherein the image data receiver circuit is configured to receive a continuous stream of image data frames, and wherein the error handling circuit is configured to perform the error concealment operation for each image data frame of the continuous stream in response to determining that the size of the corresponding image data frame is incorrect.

10. The image processing apparatus according to claim 1, further comprising an error detection circuit communicatively coupled to the error handling circuit, wherein the size of the image data frame is a first size, and wherein the error detection circuit is configured to determine whether a second size of the image data frame is incorrect based on a reference size.

11. The image processing apparatus according to claim 10, wherein the error detection circuit is further configured to: determine whether the second size of the image data frame is greater than the reference size; and in response to determining that the second size is greater than the reference size, discard extra pixel data of one or more rows of the image data frame such that the second size becomes equal to the reference size.

12. The image processing apparatus according to claim 1, wherein the error handling circuit is further configured to interrupt the main processor in response to determining that the size of the image data frame is incorrect.

13. The image processing apparatus according to claim 12, wherein the error handling circuit is further configured to perform the error concealment operation for each row of multiple rows of the image data frame one row at a time.

14. The image processing apparatus according to claim 13, wherein the error handling circuit is further configured to report error information to the main processor, wherein the error information includes at least one of error location information and frame confidence value information.

15. An image processing method, comprising: receiving an image data frame; It is incorrect to determine the size of the image data frame based on a first marker indicating the start of the image data frame and further based on a second marker indicating the end of the image data frame; In response to determining that the size of the image data frame is incorrect, perform an error concealment operation on the image data frame, where the error concealment operation includes: Discard the data of the remaining part of the received image data frame; and When the image data frame is determined to be too small in size and it is determined that pixel data of at least one row in multiple rows of the image data frame is missing, generate virtual data for the missing pixel data; and When the image data frame is determined to be too large in size, generate virtual data to complete the image data frame; and Perform an image processing operation on the image data frame on which the error concealment operation has been performed.

16. The image processing method according to claim 15, wherein generating virtual data for the missing pixel data includes: Generate the virtual data for the missing pixel data such that the size of the image data frame becomes equal to a reference size.

17. The image processing method according to claim 15, wherein generating the virtual data to complete the image data frame includes: Generate the virtual data to complete the image data frame such that the size of the completed frame becomes equal to a reference size.

18. The image processing method according to claim 15, further includes: Detect a frame synchronization signal protocol violation for the image data frame based on at least one of the first marker, the second marker, and the third marker, where the first marker includes a vertical start signal valid when receiving the data of the first pixel of the image data frame, the second marker includes a vertical end signal valid when receiving the data of the last pixel of the image data frame, and the third marker includes a horizontal start signal valid when receiving the data of the first pixel of each row in multiple rows of the image data frame or a horizontal end signal valid when receiving the data of the last pixel of each row in the multiple rows of the image data frame.

19. The image processing method according to claim 15, wherein the size of the image data frame is a first size, and the method further includes determining whether a second size of the image data frame is incorrect based on a reference size, where the second size of the image data frame is measured by a first module of an image processing device, and the first module is upstream of a second module of the image processing device that performs the error concealment operation.

20. The image processing method according to claim 19, further includes: Determine whether the second size of the image data frame is greater than the reference size; In response to determining that the second size is greater than the reference size, discard the extra pixel data of one or more rows of the image data frame by the first module such that the second size becomes equal to the reference size; and Transmit the image data frame with the discarded extra pixel data to the second module.

21. The image processing method according to claim 15 further includes interrupting the main processor in response to determining that the size of the image data frame is incorrect.

22. A non-transitory computer-readable medium storing instructions thereon, the instructions, when executed by one or more processors, cause the one or more processors to: Receive an image data frame; Determine that the size of the image data frame is incorrect based on a first marker indicating the start of the image data frame and further based on a second marker indicating the end of the image data frame; In response to determining that the size of the image data frame is incorrect, perform an error concealment operation on the image data frame, wherein when performing the error concealment operation, the one or more processors: Discard the data of the remaining part of the received image data frame; and Generate virtual data for the missing pixel data when the image data frame is determined to be too small in size and it is determined that pixel data of at least one row in multiple rows of the image data frame is missing; And Generate virtual data to complete the image data frame when the image data frame is determined to be too large in size; And Perform an image processing operation on the image data frame on which the error concealment operation has been performed.

Images