Image processing fault detection

Abstract

The invention relates to a method and a system for detecting a fault in an image processing system. The image processing system comprises an image signal processor. An example method includes dividing an image into a first image tile and a second image tile, wherein the first image tile and the second image tile each include an overlapping portion of the image; processing the first image tile using the ISP to generate a first processed image tile including a first processed overlap portion; processing the second image tile using the ISP to generate a second processed image tile including a second processed overlap portion; calculating the characteristics of the overlapped part after the first processing and calculating the same characteristics of the overlapped part after the second processing; and comparing the characteristic of the overlapping portion after the first processing with the characteristic of the overlapping portion after the second processing.

Description

Technical Field

[0001] This disclosure relates to systems and methods for detecting faults in image processing systems. Background Technology

[0002] Image processing systems can generally be considered as systems that apply functions to input images to obtain processed output images and / or useful information about the images. Image processing systems typically include an image signal processor (ISP); that is, a dedicated component configured to acquire unprocessed image files and apply signal processing techniques to enhance the image before further processing by the image processing system.

[0003] Image processing systems are used in a wide range of technologies and fields, many of which are critical to safety. In the automotive industry, image processing systems are increasingly used to support driver assistance systems, vehicle safety systems, and autonomous driving capabilities. For example, vehicles can use image processing systems to analyze images to detect road signs, lane markings, other vehicles, and so on. The output can be used to provide warnings or information to the driver, or to provide information support for autonomous driving.

[0004] In safety-critical applications, it is crucial to regularly verify the functionality of the image processing system (especially the image processing unit, ISP). Hardware errors in these systems can lead to potentially hazardous outputs (e.g., providing incorrect information to the driver).

[0005] However, providing sufficient verification can be challenging. Given the complexity of modern computing architectures, it is difficult to robustly verify all hardware components in real time. Furthermore, performing verification significantly increases the computational demands on the image processing system. Summary of the Invention

[0006] According to a first aspect, a computer-implemented method for detecting faults in an image processing system is provided. The image processing system includes an image signal processor (ISP). The computer-implemented method includes:

[0007] Receive image;

[0008] The received image is divided into a first image patch and a second image patch, wherein the first image patch and the second image patch each include the overlapping portion of the received image;

[0009] The first image patch is processed using ISP to produce a first processed image patch that includes the overlapping portion after the first processing;

[0010] The second image patch is processed using ISP to produce a second processed image patch that includes the overlapping portion of the second processed patch;

[0011] Calculate the characteristics of the overlapping portion after the first processing and calculate the same characteristics of the overlapping portion after the second processing; and

[0012] The characteristics of the overlapping portion after the first processing are compared with the characteristics of the overlapping portion after the second processing.

[0013] Because the first and second processed overlapping portions correspond to the same parts of the received image (input image), the calculated characteristics of the processed overlapping portions should be equal. Therefore, when comparing these characteristics, the differences between the calculated characteristics can indicate a fault in the image processing system (particularly the ISP that has processed the overlapping portions in each of the first and second image patches). Thus, this disclosure advantageously provides an effective method for detecting hardware faults in an image processing system.

[0014] The computer-implemented method may further include generating a fault output if a comparison between the characteristics of the first processed overlapping portion and the characteristics of the second processed overlapping portion reveals a difference.

[0015] Fault output can include numbers or text-based strings. For example, fault output can be provided to a user (e.g., via a visual display or audible alarm) to notify them that a fault has been detected. Fault output can be provided to a computer processor to initiate additional processes. For example, in the case of this disclosure used in a semi-autonomous vehicle, fault output may disengage the vehicle from semi-autonomous driving mode.

[0016] Processing a first image patch using ISP includes calculating the properties of the overlapping portion after the first processing. Processing a second image patch using ISP includes calculating the properties of the overlapping portion after the second processing.

[0017] Using ISP to compute the characteristics of the overlapping portions after the first and / or second processing can advantageously reduce the number of additional components required to perform fault detection and / or reduce the additional computational requirements for this. As part of normal image processing, ISP may have already determined the characteristics of the image patches that can be used for fault detection (e.g., signal-to-noise ratio or Fourier transform output). Therefore, separate characterization computations are not required—the only additional computation required compared to the usual processing of the input image is the repeated processing of the overlapping portions.

[0018] Calculating the properties of the overlapping region after the first processing can be performed after processing the first image patch using ISP. Calculating the properties of the overlapping region after the second processing can be performed after processing the second image patch using ISP. The calculation of the properties of the overlapping regions after the first and second processing can be performed using the characterization module.

[0019] Using a separate characterization module to compute characteristics after the ISP processes image tiles (as opposed to computing characteristics using the ISP itself) can advantageously enable fault detection of all other components of the image processing system. By providing a characterization module at the end of the normal processing pipeline (i.e., after the ISP, where the processed image is typically passed to another image processing module), faults associated with any components upstream of the pipeline (e.g., all logic and memory components) can be detected by comparing characteristics.

[0020] Calculating the characteristics of the overlapping portion after the first processing may include calculating the characteristics of a first plurality of sub-regions within the first processed image patch, wherein at least some of the first plurality of sub-regions are located within the overlapping portion after the first processing. Calculating the characteristics of the overlapping portion after the second processing may include calculating the characteristics of a second plurality of sub-regions within the second processed image patch, wherein at least some of the second plurality of sub-regions are located within the overlapping portion after the second processing.

[0021] Calculating properties across sub-regions can advantageously reduce the computational requirements associated with calculating those properties. Calculating properties of sub-regions (e.g., 16×16 or 32×32 pixel areas of an image patch) rather than individual pixels may require less computation, while still ensuring sufficient area is tested for each image patch. Sub-regions that do have the calculated properties cover at least a portion of the first and second processed overlaps, allowing for direct comparison of these portions. Multiple sub-regions can cover only the processed overlaps or a portion thereof, thereby minimizing the computation required to calculate properties of portions that only appear in the first or second image patch and therefore cannot be directly compared to infer the faulty portion.

[0022] Calculating the characteristics of the overlapping portion after the first processing and the overlapping portion after the second processing may include:

[0023] The Cyclic Redundancy Check (CRC) algorithm is applied to the first processed image patch and the second processed image patch.

[0024] Using the algorithms described above to calculate characteristics can advantageously reduce the computational requirements associated with providing fault detection. The algorithms themselves are computationally inexpensive, requiring only the computation and comparison of relatively small amounts of data (e.g., each characteristic may be 2 or 4 bytes). Furthermore, these algorithms may already be used as part of a wider range of image processing systems, such as those that apply CRC as part of other system operations.

[0025] The computer-implemented method may further include:

[0026] The characteristics of the overlapping portion after the first processing and the characteristics of the overlapping portion after the second processing are stored in memory; and

[0027] Use an external verification module to verify the characteristics of the overlapping portion after the first processing and / or the characteristics of the overlapping portion after the second processing.

[0028] Storing the calculated characteristics and analyzing them using an additional verification module can advantageously enable additional verification of the image processing system. In addition to the comparison-based fault detection provided by this disclosure, additional verification can be used to provide checks for additional potential faults. These allow the image processing system to utilize other well-established fault detection techniques and be verified to desired standards, such as meeting the Automotive Safety Integrity Level (ASIL) B standard.

[0029] The computer-implemented method may further include storing a first image patch and a second image patch in a memory comprising multiple segments, wherein the overlapping portion of the first image patch is at least partially stored in one or more memory segments that are different from the overlapping portion of the second image patch.

[0030] By storing the overlapping portion of the first image patch in a different memory segment than the overlapping memory segment of the second image patch, the method advantageously enables fault detection of the memory segment. A fault (temporary or permanent) in the memory segment storing the overlapping portion of the first image patch will not affect the overlapping portion of the second image patch. Therefore, the fault can produce a detectable difference between the calculated characteristics of the two image patches.

[0031] Storing the second image block may include shifting a plurality of memory segments by a predetermined number of segments, such that the second image block is shifted by a predetermined number of segments in memory compared to the first image block.

[0032] Shifting memory segments between processing different image tiles can advantageously allow all memory segments to be used across multiple tiles. This may mean that a fault on any segment can be detected once all memory segments have been cycled through. Memory segments can be shifted one or more segments based on the size of the image tile (e.g., the size of each column / row of data in the image tile).

[0033] The computer-implemented method may further include inserting known data into at least one of the first image tile and the second image tile.

[0034] Inserting known data into one or more image tiles (e.g., by appending known data or filling portions of image tiles) advantageously provides another means of detecting faults. Characteristics can be calculated for the known data portions, and the calculated characteristics can be compared with expected / pre-calculated characteristics, where the fault causes the difference. The computational requirements are reduced compared to pre-calculated characteristics because the known data only needs to be processed once. If known data is appended to both a first and a second image tile, the characteristics of the known data portions of the two image tiles can be calculated and compared, similar to the overlapping portions discussed above.

[0035] The computer-implemented method may further include:

[0036] Remove the overlapping portions of the first-processed image patches and / or remove the overlapping portions of the second-processed image patches; and

[0037] The first processed image tiles and the second processed image tiles are recombined to produce the final processed image.

[0038] The final processed image, after removing overlapping portions, can be identical to the original input image that has already been processed by the ISP. Therefore, the current method advantageously allows the described fault detection method to be applied to any known image processing system without interrupting the system's output; the final processed image can be identical and thus can be used for further processing as usual.

[0039] According to a second aspect, an image processing system including an image signal processor (ISP) is provided. The image processing system is configured to:

[0040] Receive image;

[0041] The received image is divided into a first image patch and a second image patch, wherein the first image patch and the second image patch each include the overlapping portion of the received image;

[0042] The first image patch is processed using ISP to produce a first processed image patch that includes the overlapping portion after the first processing;

[0043] The second image patch is processed using ISP to produce a second processed image patch that includes the overlapping portion of the second processed patch;

[0044] Calculate the characteristics of the overlapping portion after the first processing and calculate the same characteristics of the overlapping portion after the second processing; and

[0045] The characteristics of the overlapping portion after the first processing are compared with the characteristics of the overlapping portion after the second processing.

[0046] An ISP can be part of a System-on-a-Chip (SoC).

[0047] The described fault detection method is particularly advantageous when used with a SoCISP because it requires minimal additional components for fault detection. For example, any suitable ISP can be utilized, and a separate characterization module can be provided for said ISP. The image processing system can be implemented using any suitable computing device, and is not limited to SoCs.

[0048] The image processing system can be configured to perform the computer-implemented method of the first aspect.

[0049] According to a third aspect, an imaging system is provided. The imaging system includes:

[0050] camera;

[0051] User interface; and

[0052] An image processing system, comprising an image signal processor (ISP), wherein the image processing system is configured to:

[0053] Receive image;

[0054] The received image is divided into a first image patch and a second image patch, wherein the first image patch and the second image patch each include the overlapping portion of the received image;

[0055] The first image patch is processed using ISP to produce a first processed image patch that includes the overlapping portion after the first processing;

[0056] The second image patch is processed using ISP to produce a second processed image patch that includes the overlapping portion of the second processed patch;

[0057] Calculate the characteristics of the overlapping portion after the first processing and calculate the same characteristics of the overlapping portion after the second processing; and

[0058] The characteristics of the overlapping portion after the first processing are compared with the characteristics of the overlapping portion after the second processing.

[0059] The camera is configured to capture images and provide them to an image processing system, and

[0060] The user interface is configured to provide output to the user based on a comparison of the characteristics of the overlapping portion after the first processing with the characteristics of the overlapping portion after the second processing.

[0061] The camera can be any suitable image-acquiring device and can be an existing device associated with a wide range of platforms. For example, in the case of applying an imaging system to a vehicle, the vehicle may already be equipped with cameras for other driving functions and capabilities.

[0062] Providing output to the user via a user interface can advantageously enable the user to react to detected faults. For example, in the case of detecting differences in the characteristics of image tiles, a visual and / or auditory alert indicating a detected fault can be provided to the user (e.g., the driver)—for example, a warning accompanied by an alarm noise can be displayed on the vehicle's display system. The output can also be provided to and utilized by other systems in a wider system. For example, autonomous driving capabilities dependent on ISP processing can be disabled when the fault is detected, while the user is appropriately notified.

[0063] The imaging system may include the image processing system of the second aspect.

[0064] According to a fourth aspect, a computer program is provided, the computer program comprising instructions that cause a computer processor to execute the method according to the first aspect.

[0065] A computer program may be provided that, when run on a computer, causes the computer to configure any device, including the circuits, controllers, sensors, filters, or devices disclosed herein, or to perform any of the methods disclosed herein. The computer program may be implemented in software, and the computer may be considered any suitable hardware, including, as non-limiting examples, digital signal processors, microcontrollers, and implementations in read-only memory (ROM), erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM). The software implementation may be an assembler.

[0066] Computer programs may be provided on a non-transitory computer-readable medium, which may be a physical computer-readable medium, such as a disk or storage device, or may embody a transient signal. Such a transient signal may be a network download, including an internet download.

[0067] These and other aspects of the invention will become apparent and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0068] The embodiments will now be described by way of example only with reference to the accompanying drawings, in which:

[0069] Figure 1 This is a schematic diagram of an input image being divided into two image patches with overlapping portions, according to an example of this disclosure;

[0070] Figure 2 This is a schematic diagram of an example image processing system according to the present disclosure;

[0071] Figure 3 This is a schematic diagram of an example imaging system according to the present disclosure;

[0072] Figure 4 This is a schematic diagram of image tiles being saved to segmented memory according to the present disclosure;

[0073] Figure 5 This is a flowchart of a method for detecting faults in an image processing system according to the present disclosure.

[0074] It should be noted that the accompanying drawings are schematic and not drawn to scale. For clarity and convenience in the drawings, the relative dimensions and scales of the components have been enlarged or reduced. The same reference numerals are generally used to refer to corresponding or similar features in modified and different embodiments. Detailed Implementation

[0075] Figure 1 The image 10 is schematically shown to be divided into a first image patch 14a and a second image patch 14b. Image 10 includes an image data array. Each element 11 of the image data array may correspond to a pixel in image 10. For example, the image data may be an RGB image or a raw image.

[0076] Image 10 is divided into a first image portion 12a and a second image portion 12b. Adjacent portions 13a of the second image portion 12b are appended / filled onto the first image portion 12a to form a first image patch 14a. Similarly, adjacent portions 13b of the first image portion 12a are appended / filled onto the second image portion 12b to form a second image patch 14b. Thus, the first image patch 14a and the second image patch 14b include an overlapping portion 15; pixels of the appended adjacent portions 13a and 13b exist within these two image patches.

[0077] For example, image 10 can be a 1280-pixel wide image. The image can be vertically divided into a first image portion 12a and a second image portion 12b, each portion being 640 pixels. Along the edges of the image portions 12a and 12b that divide image 10, each can be appended with a 16-pixel wide strip / column from the other of image portions 12a and 12b. This process of appending each of image portions 12a and 12b to its adjacent pixels creates a first image patch 14a and a second image patch 14b, each image patch having a total width of 656 pixels. Both the first image patch 14a and the second image patch 14b include a 32-pixel wide vertical strip—an overlapping portion 15—found in both image patches 14a and 14b.

[0078] The aforementioned padding of image patches 14a and 14b inherently implies that the overlapping portion 15 is processed twice by the image processing system hardware (e.g., ISP). As further discussed below, the two processing iterations generate characteristics of the resulting processed overlapping portion 15, which can be stored and compared to detect both permanent and transient faults. By extending each image patch by several columns by appending adjacent portions 13a and 13b of adjacent image patches to each of the image patches 14a and 14b, image artifacts in the final processed image can be avoided.

[0079] Although described as a vertical bisection of image 10, it should be understood that this is merely one example of dividing image 10 into multiple overlapping patches. For example: image 10 can be divided horizontally; the two image portions 12a, 12b do not necessarily need to have equal dimensions; the number of overlapping pixels can vary; more than two image portions can be generated (image 10 can be divided into four quadrants, for example, with overlapping portions along the edges of each division); and / or the image portions do not necessarily need to be rectangular or located at the edges of image 10 (for example, the first image portion 12a can be 'cut out' from within the second image portion 12b).

[0080] The above partitioning process can be implemented using any suitable computational operation. For example, image 10 can be received and saved to a memory storage device (e.g., external memory, such as double data rate (DDR) memory). Adjacent portions 13a of the first image portion 12a and the second image portion 12b can be read from memory for use in subsequent processes, thereby generating a first image patch 14a upon reading this data. Similarly, the second image patch 14b can be read subsequently. Overlapping portions can be saved only once, wherein the overlapping portions are read twice as part of the first and second image patches. This reduces the amount of image data that needs to be stored.

[0081] Figure 2 A schematic diagram of an image processing system 100 is shown. The image processing system 100 includes an image signal processor (ISP) 110 and a memory storage device 112. The image 10 is stored in the memory storage device 112.

[0082] Use the above about Figure 1The process reads image 10 from memory storage device 112 and divides image 10 into a first image patch 14a and a second image patch 14b. The first image patch 14a and the second image patch 14b are provided to the ISP for processing. Any suitable digital image processing / algorithm can be used to process image patches 14a and 14b. This can depend on the ISP itself and the application of image processing device 100. For example, the ISP can be configured to perform denoising, sharpening, compression, and / or filtering (spatial filtering or Fourier-based filtering). The ISP can be used to provide, for example, high dynamic range (HDR), camera correction matrix, color space conversion, chromatic aberration correction, and / or de-mosaic.

[0083] The ISP outputs a first processed image patch 18a and a second processed image patch 18b. The first processed image patch 18a includes a first processed image portion 16a (i.e., the processed form of the first image portion 12a) and a processed adjacent portion 17a (i.e., the processed form of the adjacent portion 13a). Similarly, the second processed image patch 18b includes a second processed image portion 16b and a processed adjacent portion 17b. Because the input image patches 14a and 14b both include overlapping portions 15, the overlapping portions 15 are processed in both image patches, thereby generating a first processed overlapping portion in the first processed image patch 18a and a second processed overlapping portion in the second processed image patch 18b. Similar to... Figure 1 As shown, the first processed overlapping portion will include a portion of the first processed image portion 16a and a portion of the processed adjacent portion 17a. Similarly, the second processed overlapping portion will include a portion of the second processed image portion 16b and a portion of the processed adjacent portion 17b.

[0084] The adjacent portions 13a and 13b of the processed image patches 18a and 18b, which have been processed by the ISP to produce the processed adjacent portions 17a and 17b, can be removed. Alternatively, the entire overlapping portion can be removed from one of the processed image patches 18a and 18b. The processed image patches are then recombined to generate the processed output image 20. That is, the processed first image portion 16a and the second image portion 16b are recombined to generate a version of the input image 10 that has been processed by the ISP. The image processing system 100 also generates a characteristic 22, also known as a signature. The characteristic 22, discussed further below, contains statistical or numerical information about the processed image patches 18a and 18b, including information about the overlapping portion of the two patches. Both the processed output image 20 and the characteristic 22 can be stored in the memory storage device 112.

[0085] Figure 3A schematic diagram of an imaging system 200 is shown. The imaging system 200 may include, as described above... Figure 2 The image processing system 100 under discussion includes an image processing unit (ISP) 110 and a memory storage device 112. It should be understood that the ISP 110 can be configured as follows: Figure 2 It works as described in the text; for clarity, Figure 3 The generation of image tiles and processed image tiles is omitted.

[0086] Imaging system 200 includes at least one imaging device 210 (e.g., a camera) configured to capture an image 10 received by image processing system 100. The image captured by imaging device 210 can pass through capture pipeline 212 before being received by image processing system 100. Capture pipeline 212 may include, for example, a communication channel, an image sensor interface (ISI) module, and / or a mobile industrial processor interface (MIPI) module.

[0087] Figure 3 The image processing system 100 further includes a characterization module 120, which receives processed image tiles from the ISP 110. The characterization module 120 is configured to calculate characteristics 22 of the processed tiles.

[0088] Any suitable algorithm / technique can be applied to the processed image patch so that the calculated characteristic can be any suitable characteristic. For example, the characterization module 120 can be configured to apply an error detection algorithm (e.g., cyclic redundancy check) to the processed image patch so that the determined characteristic 22 is a check value.

[0089] Despite Figure 3 While shown as a separate characterization module 120, the computation of feature 22 can be performed as part of a process executed by the ISP 110. Statistical information generated by the ISP during processing can be output as feature 22. For example, the ISP can be configured to calculate the signal-to-noise ratio (SNR), sum the pixel values ​​of a sub-region, or perform a Fourier transform.

[0090] Features 22 can be calculated for discrete sub-regions or patches within each image patch. For example, an image patch can be segmented into 32×32 pixel or 16×16 pixel sub-regions, with features calculated across each sub-region. Therefore, features 22 can include a sequence or array of calculated features. The size of features 22 can be much smaller than image 10; for example, each feature can have a 2-byte or 4-byte value. At least a portion of the overlapping portion of each of the first and second processed image patches will have the calculated features. For example, one or more corresponding sub-regions within the overlapping portion can have the calculated features. Calculating features can include calculating features across the entire image patch (including the overlapping portion), only the overlapping portion, or at least a portion of the overlapping portion.

[0091] The calculated characteristic 22 is received by the comparison module 130. The comparison module 130 is configured to compare the characteristics of the first processed image patch with the characteristics of the second processed image patch. The comparison module 130 compares the characteristics of a sub-region within the first processed overlapping region with the characteristics of a corresponding sub-region within the second processed overlapping region. As described above, this can be performed on the entire overlapping portion or at least a portion of the overlapping portion.

[0092] Because the overlapping portions in each of the first and second image patches correspond to the same pixels in the input image, the calculated characteristic 22 in each of the processed overlapping portions in the first and second processed overlapping portions should be equal. By comparing characteristics, the comparison module 130 can therefore detect faults within the hardware of the image processing system 100 (specifically, the ISP 110). The comparison of characteristics may include finding numerical differences, for example, by calculating the difference between the value associated with the characteristic of the first processed overlapping portion and the corresponding value associated with the characteristic of the second processed overlapping portion.

[0093] The fault will manifest as a difference in characteristics between corresponding sub-regions of the first-processed image patch and the second-processed image patch. Faults may include:

[0094] i) Transient faults—temporary and / or fluctuating faults that occur or disappear between the feature calculations of the first and second image tiles, resulting in differences between the calculated features. Image tiles can be processed sequentially to maximize the likelihood that transient faults will occur between the processing of two tiles; and / or

[0095] ii) Permanent Failure – A permanent failure in the hardware. Different hardware may be used to process the first image tile and the second image tile. For example, the image tile and / or the processed image tile may be stored in different memory locations, or processed using different processing cores of the ISP 100. Therefore, a hardware failure in any of these components will result in differences between the calculated characteristics (because only one image tile is processed / stored on the hardware with the permanent failure).

[0096] By performing feature calculations at the end of the image processing pipeline (e.g., using a separate feature module 120 after ISP 110), image processing system 100 can provide hardware fault detection capabilities to all upstream components (e.g., input logic, utilized memory locations, execution logic, registers, and parameter memory affecting the behavior of the execution logic). Although described as hardware faults, the system is also capable of detecting software faults. Software / firmware errors can cause hardware components to operate incorrectly, thus also generating detectable differences in feature 22.

[0097] The comparison module 130 is configured to generate a fault output if a fault is detected (i.e., if there is a difference in characteristics between the first processed image patch and the second processed image patch, or if the difference exceeds a predetermined threshold). The fault output is received by the user interface 220. The user interface 220 can be configured to alert the user of the imaging system 200 to the fact that a fault has been detected. The user interface 220 may include visual elements (e.g., a display) and / or auditory elements (e.g., an alarm) to alert the user. The user interface 220 may also include a storage medium so that the detected fault can be retrieved later.

[0098] like Figure 2 As shown, the output image 20 (corresponding to the recombined first and second image portions) can be stored in the memory storage device 112. The output image 20 (i.e., the input image 10 processed by the ISP 110) can be used by an additional module / processor 230 of the imaging system 200. The additional module 230 can vary depending on the application of the imaging system 200. For example, in the case where the imaging system 200 is used to provide a driver assistance system in a vehicle, the additional module 230 can be configured to detect road markings present in the output image 20 (e.g., by using edge detection and image segmentation techniques).

[0099] Feature 22 can also be stored (e.g., in memory storage device 112) and used in additional modules / processors that may vary depending on the application of imaging system 200. Feature 22 can be provided to verification module 240. Verification module 240 can be configured to analyze feature 22, for example, to compare feature 22 with predetermined expected values, thereby providing additional verification that feature 22 has been correctly calculated. Analysis can be performed on all or some of features 22. Depending on the calculated feature 22, any suitable additional analysis and processing can be performed. For example, verification module 240 can be configured to perform Automotive Safety Integrity Level (ASIL) decomposition. In Quality Management (QM) ISP, characterization module 120 can perform potential fault checks via external ASIL-B software and hardware. In this configuration, verification module 240 can be QM(B) and external independent ASIL-B(B) hardware and software can verify feature 22.

[0100] It should be understood that Figure 2 The image processing system 100 shown and Figure 3 The implementation of the imaging system 200 shown is merely an example. The methods and systems shown can be implemented using any suitable computational solution and architecture. For example, the ISP 100, characterization module 120, and comparison module 130... Figure 3 These components are shown as separate components. However, all of these components can be implemented on the same computing device. Specifically, the ISP 110 and / or other components can be provided as a system-on-a-chip (SoC). Characteristic calculations and comparisons can therefore enable fault detection to be provided to the entire SoC.

[0101] Clearly, the methods and systems of the present invention can be repeatedly applied to provide fault detection across multiple input images. This can be performed in parallel (e.g., performing fault detection on multiple images captured simultaneously by multiple cameras) and / or sequentially (e.g., performing fault detection on all or some images captured as part of a video recording). Therefore, these methods and systems can provide continuous monitoring of problems such as memory corruption or damaged logic circuitry systems.

[0102] refer to Figure 4 The diagram illustrates image patches 14a-d stored in segmented memory 114. Segmented memory 114 includes N memory segments (numbered 0, 1, 2, ..., N-2, N-1, N). Segmented memory may include multiple buffer segments (e.g., row buffers). Different portions of an image / image patch can be stored in different memory segments. For example, one or more corresponding columns / rows of multiple image patches can be stored in different memory segments of segmented memory 114.

[0103] Specifically, overlapping portions of various image tiles can be saved to different memory segments. For example... Figure 4 As shown, for the first image patch 14a, the first image portion 12a is saved to memory segments 0, 1, 2, 3, and the adjacent portion 13a of the second image portion 12b is saved to memory segments ..., N-2. Simultaneously, for the second image patch 14b, the second image portion 12b is saved to memory segments 3, ..., N-2, N-1, and the adjacent portion 13b of the first image portion 12a is saved to 1, 2. As a result, for the overlapping portions of the images found in both the first image patch 14a and the second image patch 14b, at least some of the columns of the overlapping portions in the two image patches are saved to memory segments different from the other image patch. For example, the leftmost column of the 'overlap' shown for the first image patch 14a and the second image patch 14b is taken. For the first image patch 14a, this column (in the first image portion 12a) is stored in memory segment 2. Simultaneously, for the second image patch 14b, this column (in the adjacent portion 13b) is stored in memory segment 1. By storing the overlapping portions in different memory segments, faults in the segments can be detected by calculating characteristics (as described above). For example, if memory segment 1 is corrupted, the characteristic calculation of the overlapping portion in the second image patch 14b will be affected, while the characteristic calculation of the overlapping portion in the first image patch 14a will not be affected, thus creating a difference between the two corresponding calculated characteristics.

[0104] The memory segments of segmented memory 114 can be rotated / shifted between each image tile in image tiles 14a-d. For example... Figure 4 As shown, for each of the four image blocks 14a-d, a space is shifted to the left. For example, the first image block 14a begins reading from / writing to memory block 0, the second image block 14b begins reading from / writing to memory block 1, and so on. As a result of this rotation, all memory blocks of the segmented memory 114 can be used during the processing of multiple image blocks, thereby enabling the detection of faults in all memory blocks. Figure 4 An example is shown in which a first input image is segmented into a first image patch 14a and a second image patch 14b, and then the second input image is segmented into a third image patch 14c (including the third image portion 12c and the adjacent portion 13c of the fourth image portion 12d) and a fourth image patch 14d (including the fourth image portion 12d and the adjacent portion 13d of the third image portion 12c). However, the same process can also be applied to a single input image that is divided into more than two image patches.

[0105] This method of saving overlapping portions of image tiles to different memory segments can be applied to any and / or all parts of the described fault detection method / system for saving or retrieving images. For example, memory segments can be rotated between saves of different input images; different image tiles can be saved to different memory segments when provided to the ISP (as described above); and / or different processed image tiles can be saved to different memory segments after processing by the ISP. In all cases, differences in calculated characteristics can be detected because some parts of the image / image tile are affected by the fault while others are not. Although in Figure 4 The diagram shows a shift of a location within a memory segment, but other shift amounts can be used appropriately. For example, when calculating the characteristics of a sub-region spanning two memory segments, the memory segments can be shifted by two locations at a time, causing the entire sub-region to shift to a different memory segment.

[0106] Before processing at the ISP, known data and / or padding can be added to at least some of the image tiles. For example, known data can be added only to the first image tile. Figure 4 As shown, each of the image patches 14-ad has been appended with known data portions 19a-d (e.g., known test image patterns). Due to the rotation of the memory segments discussed above, these known data portions 19a-d are also stored in different memory segments across the image patches 14a-d, namely memory segment N-1, where N is for the first image patch 14a, and memory segments 1 and 2 are for the fourth image patch 14d. As for the image portions and appended portions, the known data portions can have characteristics calculated using the characterization module. Since the data contained within the known data portions 19a-d is known, the characteristics of each known data portion 19a-d can be compared with expected characteristics. Similar to the comparison of characteristics of overlapping portions, the characteristics of the calculated first known data portion 19a can be compared with the characteristics of the calculated second known data portion 19b using the comparison module. Therefore, the known data portions 19a-d provide another way to verify the calculated characteristics, and thus can also be used to identify faults in the described image processing system (particularly hardware faults in certain memory segments of the segmented memory 114). In addition to providing verification of internal segmented memory (as described above), the known data can also advantageously allow verification of a wider range of image processing systems, including registers and computational logic.

[0107] Figure 5 A flowchart of a method 300 for detecting faults in an image processing system, including an image processing system (ISP), is shown. The method includes:

[0108] Receive 302 image;

[0109] The received image is divided into a first image patch and a second image patch by 304, wherein the first image patch and the second image patch each include the overlapping part of the received image;

[0110] The first image patch 306 is processed using ISP to produce a first processed image patch that includes the overlapping portion after the first processing.

[0111] The second image patch 308 is processed using ISP to produce a second processed image patch that includes the overlapping portion of the second processed patch;

[0112] Calculate the characteristics of the overlapping portion after the first processing (310) and calculate the same characteristics of the overlapping portion after the second processing; and

[0113] The characteristics of the overlapping portion after the first processing are compared with the characteristics of the overlapping portion after the second processing. 312.

[0114] The calculation of feature 310 can be performed after processing the first and second image patches 306 and 308 using ISP, such as... Figure 5 As shown. As described above, performing feature calculation 310 subsequently allows the method of this disclosure to run and test the entire image processing pipeline.

[0115] However, it should be understood that Figure 5 The steps shown do not necessarily have to be performed in this order; the order of some operations can vary. For example, as also mentioned above, the characteristics of the first and second processed overlapping portions can be characteristics or statistics generated during the processing of image tiles with ISP—that is, without separate calculation of the characteristics. In such an example, calculating characteristic 310 can be understood as being performed concurrently with processing the first and second image tiles 306 and 308.

[0116] Alternatively, the calculation of the characteristics of the first processed overlapping portion can be performed separately from the calculation of the characteristics of the second processed overlapping portion. In one example, ISP processing 306 is used on a first image patch to produce a first processed image patch including the first processed overlapping portion. The characteristics of the first processed overlapping portion are then calculated and saved to memory. After the processing and characteristic calculation of the first image patch, ISP processing 308 is used on a second image patch to produce a second processed image patch including the second processed overlapping portion. Essentially, the steps of calculating 310 the characteristics of the first processed overlapping portion and the characteristics of the second processed overlapping portion can be divided into two separate operations, wherein the calculation of the characteristics of the first processed overlapping portion is performed after ISP processing 306 on the first image patch, and the calculation of the characteristics of the second processed overlapping portion is performed after ISP processing 308 on the second image patch.

[0117] By performing the calculation of the characteristics of the first image patch before processing the second image patch, the amount of characteristic data that needs to be stored can be reduced and / or the bandwidth of the image processing system can be improved. With the characteristics of the first image patch already calculated, the final image, including the entire processed received image, can be generated 'in real time' while processing the second image patch. The characteristics of the second image patch can be compared with those of the first image patch while the characteristics of the second image patch are being calculated, instead of having to simultaneously calculate all comparisons of various sub-regions (discussed above).

[0118] Alternatively, the calculation of the characteristics of the overlapping portion after the first processing and the overlapping portion after the second processing can be performed before processing the first image patch and the second image patch (or at least a portion of the processing) using the ISP. For example, the first image patch and the second image patch can be read into the ISP (e.g., read into...). Figure 4 The characteristics are calculated after the segmented memory discussed earlier, but before the ISP actually processes the image tiles. This order of operations still allows for fault detection in the preceding pipeline steps (e.g., fault detection in the segmented memory and any operations performed before the ISP), but can reduce the processing requirements of the system. After the characteristics associated with the first and second image tiles have been calculated, it may only be necessary to save and process the overlapping portions for one of the tiles by the ISP.

[0119] The above methods can be implemented using any suitable computing device, including the image processing system / imaging apparatus or components thereof disclosed herein.

[0120] Other variations and modifications will be apparent to those skilled in the art from this disclosure. Such variations and modifications may involve equivalent and other features known in the field of image processing systems, and may replace or use these equivalent and other features in addition to those described herein.

[0121] Although the appended claims relate to specific combinations of features, it should be understood that the scope of the disclosure of this invention also includes any novel feature or any novel combination of features or any generalization thereof explicitly or implicitly disclosed herein, whether or not it relates to the same invention as currently claimed in any of the claims, and whether or not it alleviates any or all of the same technical problems as those of this invention.

[0122] Features described in the context of a single embodiment may also be provided in combination in that single embodiment. Conversely, for the sake of brevity, the various features described in the context of a single embodiment may also be provided individually or in any suitable sub-combination. The applicant hereby informs that new claims may be formulated during the examination of these features and / or combinations of these features.

[0123] For completeness, it is also stated that the term "comprising" does not exclude other elements or steps, the term "a" or "an" does not exclude multiple, a single processor or other unit may perform the functions of several components recited in the claims, and the reference numerals in the claims should not be construed as limiting the scope of the claims.

Claims

1. A computer-implemented method (300) of detecting a fault in an image processing system (100), the image processing system (100) comprising an image signal processor (110), ISP, characterized in that, The computer-implemented method (300) comprises: receiving (302) an image (10); dividing (304) the received image (10) into a first image tile (14a) and a second image tile (14b), wherein the first image tile (14a) and the second image tile (14b) each comprise an overlapping portion (15) of the received image (10); processing (306) the first image tile (14a) using the ISP (110) to produce a first processed image tile (18a) comprising a first processed overlapping portion; processing (308) the second image tile (14b) using the ISP (110) to produce a second processed image tile (18b) comprising a second processed overlapping portion; computing (310) a characteristic (22) of the first processed overlapping portion and computing the same characteristic (22) of the second processed overlapping portion; and comparing (312) the characteristic (22) of the first processed overlapping portion with the characteristic (22) of the second processed overlapping portion.

2. The computer-implemented method (300) according to claim 1, wherein Further comprising: generating a fault output if the comparison of the characteristic (22) of the first processed overlapping portion with the characteristic (22) of the second processed overlapping portion results in a difference.

3. The computer-implemented method (300) of claim 1 or claim 2, wherein: processing (306) the first image tile (14a) using the ISP (110) comprises computing the characteristic (22) of the first processed overlapping portion; and processing (308) the second image tile (14b) using the ISP (110) comprises computing the characteristic (22) of the second processed overlapping portion.

4. The computer-implemented method (300) of claim 1 or claim 2, wherein: computing (310) the characteristic (22) of the first processed overlapping portion is performed after processing (306) the first image tile (14a) using the ISP (110); and computing (310) the characteristic (22) of the second processed overlapping portion is performed after processing (308) the second image tile (14b) using the ISP (110), wherein computing (310) the characteristic (22) of the first processed overlapping portion and the characteristic (22) of the second processed overlapping portion is performed using a characterization module (120).

5. The computer-implemented method (300) of any preceding claim, wherein: computing (310) the characteristic (22) of the first processed overlapping portion comprises computing the characteristic of a first plurality of sub-regions within the first processed image tile (18a), wherein at least some of the first plurality of sub-regions are located within the first processed overlapping portion; and computing (310) the characteristic (22) of the second processed overlapping portion comprises computing the characteristic of a second plurality of sub-regions within the second processed image tile (18b), wherein at least some of the second plurality of sub-regions are located within the second processed overlapping portion. Computing (310) the property (22) of the second processed overlapping portion comprises computing the property of a second plurality of sub-regions within the second processed image tile (18b), wherein at least some of the second plurality of sub-regions are located within the second processed overlapping portion.

6. The computer-implemented method (300) according to any one of the preceding claims, wherein: Computing (310) the property (22) of the first processed overlapping portion and the property (22) of the second processed overlapping portion comprises: applying a cyclic redundancy check, CRC, algorithm to the first processed image tile (18a) and the second processed image tile (18b).

7. The computer-implemented method (300) according to any one of the preceding claims, wherein Further comprising: storing the property (22) of the first processed overlapping portion and the property (22) of the second processed overlapping portion in a memory (112); and verifying the property (22) of the first processed overlapping portion and / or the property (22) of the second processed overlapping portion using a verification module.

8. An image processing system (100) comprising an image signal processor (110), ISP, characterized in that, The image processing system (100) is configured to perform the method (300) according to any one of claims 1 to 7.

9. An imaging system (200), characterized by Comprising: a camera (210); a user interface (220); and The image processing system (100) according to claim 8, wherein the camera (210) is configured to capture an image (10) and provide the image (10) to the image processing system (100), and wherein the user interface (220) is configured to provide an output to a user based on the comparison of the property (22) of the first processed overlapping portion and the property (22) of the second processed overlapping portion.

10. A computer program, characterized in that, Instructions causing a computer processor to perform the method (300) according to any one of claims 1 to 7.

Images