Image processing system

Abstract

An image processing system includes one or more image sensors configured to capture images, one or more image processors configured to process images captured by the one or more image sensors to produce output images, and one or more circuits configured to use the output images produced by the one or more image processors. The one or more image processors include a set of one or more image processing stages, and each stage is configured to process images using a respective image processing operation. The image processing system also includes processing circuitry configured to verify image processing performed by one or more image processing stages of the set of one or more image processing stages by inserting one or more probes into one or more images prior to processing of the one or more images by the one or more image processing stages of the set of one or more image processing stages, and attempting to identify the one or more probes in output of the one or more processing stages.

Description

Technical Field

[0001] This application relates to an image processing system. Background Technology

[0002] The techniques described herein relate to image processing systems, and more specifically to image processing systems that modify images using a processor before they are used (e.g., displayed).

[0003] Automotive mirror replacement systems have been designed to replace or supplement vehicle mirrors (e.g., exterior mirrors) using a camera-based system. Images captured by the camera are modified by an image processor to be displayed appropriately to the vehicle operator. However, processing errors within such systems can significantly impact vehicle safety.

[0004] The applicant believes there is still room for improvement in the image processing system. Summary of the Invention

[0005] In a first aspect, this disclosure relates to an image processing system comprising:

[0006] One or more image sensors, the one or more image sensors being configured to capture images;

[0007] One or more image processors are configured to process images captured by the one or more image sensors to produce an output image, wherein the one or more image processors include a set of one or more image processing stages, each stage being configured to process the image using a corresponding image processing operation;

[0008] One or more circuits, the one or more circuits being configured to use an output image generated by the one or more image processors; and

[0009] A processing circuit system configured to verify image processing performed by one or more image processing stages from the set of one or more image processing stages in the following manner:

[0010] Before one or more images are processed by one or more image processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images; and

[0011] An attempt is made to identify one or more probes in the output of the one or more processing stages. In a second aspect, this disclosure relates to an image processor comprising:

[0012] An image processing circuit system configured to process an image to produce an output image, wherein the image processing circuit system includes a set of one or more image processing stages, each stage being configured to process the image using a corresponding image processing operation; and

[0013] A processing circuit system configured to verify image processing performed by one or more image processing stages from the set of one or more image processing stages in the following manner:

[0014] Before one or more images are processed by one or more processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images; and

[0015] Attempt to identify one or more probes in the output of the one or more processing stages.

[0016] In a third aspect, this disclosure relates to a method of operating an image processor, the method comprising:

[0017] An image is processed using a set of one or more image processing stages to produce an output image; and

[0018] The image processing performed by one or more image processing stages from the set of one or more image processing stages is verified in the following way:

[0019] Before one or more images are processed by one or more processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images; and

[0020] Attempt to identify one or more probes in the output of the one or more processing stages.

[0021] In a fourth aspect, this disclosure relates to a non-transitory computer-readable storage medium storing computer software code that, when executed on a processor, performs a method of operating an image processor, the method comprising:

[0022] An image is processed using a set of one or more image processing stages to produce an output image; and

[0023] The image processing performed by one or more image processing stages from the set of one or more image processing stages is verified in the following way:

[0024] Before one or more images are processed by one or more processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images; and

[0025] Attempt to identify one or more probes in the output of the one or more processing stages. Attached Figure Description

[0026] Various embodiments of the technology described herein will now be described by way of example only, with reference to the accompanying drawings, wherein:

[0027] Figure 1 A conventional image processing system is illustrated schematically.

[0028] Figure 2 An image processing system configured according to the technology described herein is illustrated schematically;

[0029] Figure 3 An image processing system configured according to the technology described herein is illustrated schematically;

[0030] Figure 4 A conceptual illustration of an image with an embedded digital watermark; and

[0031] Figure 5 An image processing system configured according to the techniques described herein is illustrated schematically.

[0032] Where appropriate, similar reference numerals may be used for similar elements throughout the accompanying drawings. Detailed Implementation

[0033] A first embodiment of the technology described herein includes an image processing system, the image processing system comprising:

[0034] One or more image sensors configured to capture images;

[0035] One or more image processors are configured to process images captured by the one or more image sensors to produce an output image, wherein the one or more image processors include a set of one or more image processing stages, each stage being configured to process the image using a corresponding image processing operation;

[0036] One or more circuits, configured to use an output image generated by the one or more image processors; and

[0037] A processing circuit system configured to verify image processing performed by one or more image processing stages from the set of one or more image processing stages in the following manner:

[0038] Before one or more images are processed by one or more image processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images; and

[0039] Attempt to identify one or more probes in the output of one or more processing stages.

[0040] A second embodiment of the technology described herein includes a method for operating an image processing system, the method comprising:

[0041] Take one or more images;

[0042] The image is processed using a set of one or more image processing stages to produce one or more output images; and

[0043] Use one or more of the output images;

[0044] The method further includes:

[0045] The image processing performed by one or more image processing stages from the set of one or more image processing stages is verified in the following way:

[0046] Before one or more images are processed by one or more processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images; and

[0047] Attempt to identify one or more probes in the output of one or more processing stages.

[0048] Various embodiments relate to an image processing system comprising: one or more image sensors configured to capture images of an environment; one or more image processors configured to process the images captured by the one or more image sensors using a set of one or more image processing stages to produce output images; and one or more circuits configured to use the output images produced by the one or more image processors.

[0049] The image processing system further includes a processing circuit system configured to verify image processing operations performed by one or more image processing stages in the set of stages, i.e., to check whether the one or more image processing stages accurately perform their respective image processing operations. This is accomplished by: inserting one or more probes (each of which may have a uniquely known configuration) into the one or more images before they are processed by the one or more processing stages (e.g., subjecting the one or more probes to the same image processing operations as the one or more images); and then attempting to identify some or all of the one or more probes in the output of the one or more processing stages (the resulting output images).

[0050] If one or more expected probes cannot be identified in the output (or if one or more expected probes are identified at unexpected time and / or spatial locations within the output, or if one or more unexpected probes are identified in the output (e.g., probes expected to have been removed from the output)), it can be determined that the one or more image processing stages have not (sufficiently) accurately performed their image processing operations. On the other hand, if each of the one or more expected probes can be identified in the output (e.g., at the corresponding expected time and / or spatial location) (and if no expected probe is identified in the output), it can be determined that the one or more image processing stages have (sufficiently) accurately performed their image processing operations.

[0051] Therefore, attempting to identify one or more probes in the output of one or more processing stages allows the image processing system to determine the fidelity of the image processing operations performed by the one or more image processing stages.

[0052] Since the one or more processing stages are part of a set of processing stages configured to produce output images from images captured by the image sensor, this allows the image processing system to determine whether the output images are a sufficiently accurate representation of the environment. If it is determined that the output images are not a sufficiently accurate representation of the environment, the image processing system may, for example, indicate this to the operator and / or to the circuitry configured to use the output images, in order to prevent the images from being used inappropriately (e.g., unsafely).

[0053] Therefore, the implementation schemes provide a relatively simple and robust technique for verifying image processing operations performed by one or more image processors. It should therefore be understood that the various implementation schemes provide improved image processing systems.

[0054] Various implementation schemes also extend to the image processor itself.

[0055] Therefore, another embodiment of the technology described herein includes an image processor comprising:

[0056] An image processing circuit system configured to process an image to produce an output image, wherein the image processing circuit system includes a set of one or more image processing stages, each stage being configured to process the image using a corresponding image processing operation; and

[0057] A processing circuit system configured to verify image processing performed by one or more image processing stages from the set of one or more image processing stages in the following manner:

[0058] Before one or more images are processed by one or more processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images; and

[0059] Attempt to identify one or more probes in the output of one or more processing stages.

[0060] Another embodiment of the technology described herein includes a method of operating an image processor, the method comprising:

[0061] An image is processed using a set of one or more image processing stages to produce an output image; and

[0062] The image processing performed by one or more image processing stages from the set of one or more image processing stages is verified in the following way:

[0063] Before one or more images are processed by one or more processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images; and

[0064] Attempt to identify one or more probes in the output of one or more processing stages.

[0065] The image processing system of various embodiments includes: one or more image sensors configured to capture images; one or more image processors configured to process the images captured by the one or more image sensors to produce an output image; and one or more circuits configured to use the output image produced by the one or more image processors.

[0066] Images captured by one or more image sensors may include, for example, any suitable such images of the environment.

[0067] Images may include (after processing) images intended for use by (human) operators when operating equipment (e.g., vehicles) within a control environment. Images may also include (after processing) images intended for use by other consumer circuitry (such as machine learning systems) (in addition to or in lieu of images intended for use by (human) operators.

[0068] The image can be an image of the environment surrounding the device, which can move within the environment, for example, under the control of a (human) operator (or otherwise). The image can be an image of a "natural" environment.

[0069] In various specific embodiments, the image is an image of the environment surrounding the vehicle (e.g., a motor vehicle or other item) where the vehicle can be controlled by an operator (e.g., a driver). For example, the image may include an image of the environment behind the vehicle (e.g., images that can be used by the operator (driver) in the form of a rearview mirror or side mirror). The image may additionally or alternatively include an image of the environment on the other side of the vehicle (such as in front of or to the side of the vehicle) (e.g., images that can be used by the operator (driver) when controlling the vehicle).

[0070] In various other embodiments, the image is an image of the environment surrounding the medical device (such as an endoscope) (where the endoscope can be controlled by an operator). For example, the image may include, for example, images of the medical environment inside a patient (e.g., these images are available to the operator of the medical device when controlling the medical device).

[0071] Other arrangements are possible.

[0072] Each image should (and does in various embodiments) include an array of multiple data locations (pixels), where each data location (pixel) takes a specific data (e.g., color) value. Therefore, each image may include multiple rows of data locations (pixels) and multiple columns of data locations (pixels).

[0073] The data values ​​at the data locations of the data array can be any suitable and desired data values. The data values ​​of the data array elements can be pixel and / or sampling location data (data values).

[0074] In implementations, the data values ​​represent color values, such as RGB or YUV color values. In various specific implementations, each data location (pixel) employs a specific data (e.g., color) value for each of multiple (color) channels. Therefore, for example, the data value for each data location (pixel) may include data (e.g., color) values ​​for each of three (e.g., RGB or YUV) color channels. Each data (e.g., color) value may include, for example, 8 bits.

[0075] Each of the one or more image sensors may include any suitable image sensor, such as an image sensor that is a camera.

[0076] Each image sensor can be configured to capture a (time) sequence of images (frames) of the environment, i.e., a video of the environment. Therefore, each image can be one such image (frame) from, for example, a sequence of images (frames) captured by the image sensor.

[0077] Each of one or more image sensors (one or more cameras) can be mounted on a device (e.g., a vehicle) configured to move within an environment. Thus, in various embodiments, each image sensor is an image sensor for a device (such as a vehicle) that can move within an environment, for example, under the control of a (human) operator (or otherwise). Each image sensor can be mounted in any suitable location on the device (e.g., a vehicle) so that images captured by the sensor can be used (e.g., by an operator) when controlling the device (e.g., the vehicle). For example, one or more image sensors can each be mounted externally to a device (e.g., a vehicle or an endoscope, etc.).

[0078] An image processing system may include any suitable number of image sensors, such as one, two, three or more. Each image sensor may be configured to capture its own corresponding set of images, which may be processed and used (e.g., displayed) by one or more processors.

[0079] In one specific embodiment, the device includes a vehicle, and the image processing system includes two (or more) image sensors (cameras) configured and arranged, for example, to capture images of the environment behind the vehicle as if they were rearview or side mirrors of the vehicle.

[0080] Other arrangements are possible.

[0081] An image processing system can be configured such that images captured by each image sensor are provided to one or more image processors, for example, by writing each image to a memory from which each image can be read by one or more image processors. The memory may include any suitable memory and may be configured in any suitable and desired manner. For example, it may be a memory integrated (on a chip) with the image sensor under consideration, or it may be external memory. In a particular embodiment, it resides in external memory (such as the main memory of the entire image processing system). It may be dedicated memory for this purpose, or it may be part of memory used for other data.

[0082] In the techniques described herein, images captured by one or more image sensors are processed by one or more image processors to produce an output image. The output image may be an output image for display (provided to one or more displays), and / or may be an output image to be used by some other consumer circuitry (such as a machine learning system).

[0083] Each image processor may include any suitable such image processor. Each image processor may be provided as a (single) integrated circuit (“chip”) (or other item).

[0084] Each image processor can be configured to read images to be processed from memory storing the images. (This memory can include any suitable memory, but in a particular implementation it is external memory, such as the main memory of the entire image processing system.)

[0085] One or more image processors may include a single image processor (e.g., in the form of a single integrated circuit or "chip"), or may include multiple image processors (e.g., multiple integrated circuits or "chips").

[0086] In the case where one or more image processors include multiple image processors, the image processing system can be configured such that images captured by one or more image sensors can be processed sequentially by each of the image processors to produce an output image.

[0087] In these embodiments, one or more image processors, or each image processor, may be configured to provide an intermediate processed image to another image processor, for example, by writing each intermediate processed image to a memory from which another image processor can read the intermediate processed image. This memory may include any suitable memory, but in certain embodiments it is external memory, such as the main memory of the entire image processing system.

[0088] The image processor that generates the (final) output image can also be configured to provide each output image to a consumer circuit (e.g., a display) configured to use the image, for example, by writing each output image to a memory from which the consumer circuit (such as a display controller of a display) can read each output image. This memory may include any suitable memory, but in a particular embodiment it is external memory, such as the main memory of the entire image processing system.

[0089] The one or more image processors are configured to process an image using a set of one or more image processing stages, wherein each stage is configured to process the image using a corresponding image processing operation. In other words, the one or more image processors can be configured to process an image using a set of one or more image processing operations.

[0090] In the case where the image processing system includes a single image processor, the single image processor will include all image processing stages in the set. Alternatively, in the case where the image processing system includes multiple image processors, each image processor may include one or more image processing stages from the set of image processing stages.

[0091] An image can be processed by one or more image processors in any suitable manner (e.g., by subjecting each image in the image to one or more image processing operations in turn).

[0092] In various embodiments, one or more, or each and every, of the images captured by one or more image sensors is processed independently. In various embodiments, this is done such that each output image corresponds to one (and only one) (original) image from the (original) images captured by one or more image sensors. However, processing may also be configured such that one or more, or each output image corresponds to two or more (original) images from the (original) images captured by one or more image sensors (e.g., in cases where two or more images are combined or synthesized to generate an output image).

[0093] In various implementations, one or more, or each (and every) of the images captured by the image sensor, are processed by the same set of image processing stages (by the same image processing operations). In these implementations, each stage in the set may be configured in the same manner for each (and every) image, or different images may be processed using the same set of image processing stages, but one or more of the stages in the set may be configured in different manner.

[0094] Different images may also be processed by different sets of image processing stages (image processing operations). For example, the operating mode may be changed, or an image captured by a first image sensor may be processed using a first set of image processing stages, and an image captured by a second image sensor may be processed using a different second set of image processing stages. Each image processing stage in the first set may be different from each image processing stage in the second set, or the first set and the second set may share at least some image processing stages.

[0095] Each image processing stage should be (and in various embodiments) configured to modify the image data of each image processed by that stage (and each image processing operation should have the effect of modifying each image processed by that operation). Therefore, each output image in the output images should be (and in various embodiments) a modified form of the corresponding (original) image captured by one or more image sensors (different from the corresponding (original) image captured by one or more image sensors).

[0096] The set of image processing stages may include any suitable set of one or more image processing stages (and the set of image processing operations may include any suitable set of one or more image processing operations).

[0097] In various specific embodiments, the set is configured such that each image in the set is processed to produce an image of a form suitable for display (on a display) to an operator (e.g., a driver) of a device (e.g., a vehicle), so that the operator can use the image while controlling the device (e.g., the vehicle). However, this need not be the case, for example, when the image is to be used (without being displayed) by another consumer circuit.

[0098] A set of one or more image processing stages (operations) may include a single image processing stage (operation). However, in various specific embodiments, a set of one or more image processing stages (operations) includes a set of multiple (different) image processing stages (operations). In this latter case, the set of image processing stages (operations) may be arranged in a pipeline configuration, such that each image is processed sequentially by each stage (operation) in the set to produce an output image.

[0099] Each image processing stage in the set may include any suitable such stage and may be configured to process the image using any suitable image processing operation.

[0100] In various specific implementations, one or more of the images, or each image, are scaled (e.g., enlarged and / or reduced) to generate one or more scaled images. One or more of the images, or each image, may also be cropped, to generate one or more cropped images.

[0101] In this regard, the applicant has recognized that, in the case of a vehicle (as described above) (or otherwise), it may be necessary or expected that one or more image sensors be configured to capture images at a higher resolution and / or a larger size than is possible or convenient, for example, to be displayed on a panel inside the vehicle.

[0102] For example, it may be necessary to capture images at a relatively high resolution and / or a relatively large size to allow for techniques such as automatic image recognition, correction of distorted images (e.g., those produced using a fisheye lens or similar methods to provide a wide field of view), while it may be necessary to display images at a lower resolution and / or a smaller size depending on the panel's maximum resolution and / or size. Thus, it may be necessary or desirable to reduce and / or crop one or more images, or each image, captured by one or more image sensors.

[0103] Therefore, in various implementations, a set of one or more image processing stages includes a scaler (scaling circuitry) configured to scale (e.g., zoom in and / or zoom out) an image to generate a scaled image. Similarly, a set of one or more image processing operations may include scaling, and one or more of the output images, or each output image, may include a scaled image.

[0104] In various implementations, the set of one or more image processing stages may additionally or alternatively include a cropping stage (cropping circuitry) configured to crop the image to generate a cropped image. Similarly, the set of one or more image processing operations may include cropping, and one or more of the output images or each output image may include a cropped image.

[0105] One or more of the images, or each image, may be additionally or alternatively flipped, i.e., to generate one or more flipped (mirrored) images. In this regard, the applicant has recognized that, in the case where the device is a vehicle (as described above) (or otherwise), it may be necessary or desirable to display a mirrored display of one or more images captured by the sensors on a panel, for example, inside the vehicle (e.g., in cases where the images are being displayed for use by the vehicle operator (driver) in a rearview or side mirror).

[0106] Therefore, in various embodiments, the set of one or more image processing stages additionally or alternatively includes a flipping stage (flipping circuitry) configured to flip the image to generate a flipped image. Similarly, the set of one or more image processing operations may include flipping, and one or more of the output images, or each output image, may include a flipped image. In these embodiments, the image may be flipped horizontally and / or vertically.

[0107] One or more of the images, or each image, may be additionally or alternatively subjected to image enhancement processing, i.e., to generate one or more enhanced images. Such processing may be necessary or desirable, for example, in cases where the device is a vehicle (as described above) (or other items), in order to improve the understandability of the displayed images to the operator (driver) of the device (e.g., the vehicle) (or other items).

[0108] Therefore, in various embodiments, the set of one or more image processing stages additionally or alternatively includes one or more image enhancement stages (image enhancement circuitry systems) configured to enhance an image to generate an enhanced image. Similarly, the set of one or more image processing operations may include image enhancement, and one or more, or each, of the output images may include an enhanced image. In these embodiments, one or more, or each, of the images may be enhanced in any suitable manner (e.g., by modifying the brightness, color, contrast, etc. of the image).

[0109] Other image processing operations are possible.

[0110] For example, one or more of the images, or each image, may be composited, either additionally or alternatively, to generate one or more composite images.

[0111] Therefore, in various embodiments, the combination of one or more image processing stages additionally or alternatively includes a synthesizer (compositing circuitry system) configured to synthesize two or more images to generate a composite image. Similarly, a set of one or more image processing operations may include synthesis, and one or more of the output images, or each output image, may include a composite image. In these embodiments, two or more images captured by one or more image sensors may be synthesized, and / or one or more images captured by one or more image sensors may be synthesized with one or more other images. For example, one or more other images may include, for example, a graphic overlay to be displayed with the images.

[0112] One or more images, or each image, may be additionally or alternatively filtered, i.e., to generate one or more filtered images. Therefore, in various embodiments, a set of one or more image processing stages may additionally or alternatively include filters (filtering circuitry) configured to filter images to generate filtered images. Similarly, a set of one or more image processing operations may include filtering, and one or more output images, or each output image, may include a filtered image.

[0113] One or more images, or each image, may be additionally or alternatively rotated to generate one or more rotated images. Therefore, in various embodiments, the set of one or more image processing stages may additionally or alternatively include a rotation stage (rotation circuitry system) configured to rotate the images to generate rotated images. Similarly, the set of one or more image processing operations may include rotation, and one or more output images, or each output image, may include a rotated image. In these embodiments, one or more images may be rotated by any suitable and desired amount, such as, for example, 90°, 180°, and / or 270°.

[0114] Other image processing operations include, for example, color transformation, dithering, and gamma correction.

[0115] As should be understood from the above, the set of one or more image processing stages may include one or more or all of the following: scaler, cropping stage, flipping stage, one or more image enhancement stages, compositer, filter, rotation stage, etc. Similarly, the set of one or more image processing operations may include one or more or all of the following: scaling, cropping, flipping (mirroring), image enhancement, compositing, filtering, rotation, etc.

[0116] Once the output images have been generated, each of the output images can be used in any suitable way.

[0117] In various specific implementations, each output image in the output images is displayed on one or more displays (such as one or more display panels).

[0118] For this purpose, the image processing system may include one or more display controllers operable to provide an image for display to each display. Each display controller may be configured to read the output image from a memory storing the output image. (This memory may include any suitable memory, but in a particular embodiment it is external memory, such as the main memory of the entire image processing system.)

[0119] The output image can be displayed on a single display (e.g., a panel) or multiple displays (panels). For example, in the presence of multiple image sensors, a corresponding display can be set up and used for each image sensor (captured image). However, other arrangements are also possible.

[0120] An image processing system can be configured to display an output image to the operator of the equipment (e.g., a vehicle) in a manner that allows the operator to use the displayed image while controlling the equipment (e.g., a vehicle).

[0121] Therefore, in various embodiments, one or more of the one or more displays may be disposed on or within a device (e.g., a vehicle) configured to move within an environment. Thus, for example, in the case where the device includes a vehicle, the output images may be displayed to the operator (driver) on one or more displays (e.g., display panels) disposed within the vehicle. In various specific embodiments, the output images from each of two (or more) image sensors (cameras) (e.g., configured and arranged to capture images in a manner similar to the side mirrors of the vehicle) may be displayed, for example, on two (or more) display panels disposed within the vehicle, respectively, in a manner similar to the rearview mirrors and / or side mirrors.

[0122] Additionally or alternatively, each of the one or more displays may be located remotely from the device (e.g., a vehicle), meaning the output image may be displayed to the operator on one or more displays (e.g., a display panel) remotely from the device (e.g., a vehicle). This may be the case, for example, in the case of a medical device or a remotely operated vehicle.

[0123] In various other implementations, each of the output images may be used, additionally or alternatively, by one or more consumer circuits of the image processing system in some other way. For example, a machine learning system may use the output images, for instance, to perform image recognition with respect to each image.

[0124] Other arrangements are possible.

[0125] As described above, in various embodiments, one or more processed images are displayed on, for example, a display (such as a display panel) for use by an operator when controlling equipment (such as a vehicle). Thus, image processing errors within these systems can significantly impact the safety of the equipment (such as a vehicle). Image processing errors can also be problematic in embodiments where no output image is displayed.

[0126] The image processing system therefore also includes a processing circuit system configured to verify the image processing performed by one or more image processing stages from a set of one or more image processing stages, i.e., to check whether the one or more image processing stages (sufficiently) accurately perform their respective image processing operations.

[0127] This is accomplished by inserting one or more probes (each of which may have a uniquely known configuration) into the one or more images before the images are processed by one or more image processing stages in a set of one or more image processing stages (e.g., subjecting the one or more probes to the same image processing operations as the one or more images); and then attempting to identify the one or more probes in the output of the one or more processing stages.

[0128] The image processing stage that verifies the image processing can include any one or more suitable image processing stages from the set. The image processing stage that verifies the image processing can include any one or more of the image processing stages described above.

[0129] An image processing system can be configured to verify the image processing of a single image processing stage in a set of one or more image processing stages, or it can be configured to verify the image processing of multiple different (adjacent) image processing stages in a set.

[0130] In various embodiments, the image processing system is configured to verify image processing of all stages in a set (i.e., verifying the image processing of the entire set of image processing stages) by, for example, inserting one or more probes into an image to be processed by the set, and then attempting to identify one or more probes in the output of the set of image processing stages. In these embodiments, the method may include verifying image processing of the set of one or more image processing stages by: inserting one or more probes into an image captured by one or more image sensors, processing the image captured by one or more image sensors using the set of one or more image processing stages (along with one or more probes), and attempting to identify one or more probes in the output (such as an output image).

[0131] However, in various other implementations, the image processing system may be configured to verify image processing of fewer than all stages in the set of image processing stages (i.e., verifying image processing of a subset of the set of image processing stages).

[0132] In these embodiments, the image processing system may optionally be configured to verify image processing not included in the subset of image processing stages in any suitable manner (e.g., using one or more other techniques as appropriate). This may be particularly applicable to processing stages that can be easily verified, for example, using other techniques.

[0133] In these implementations, the image processing system can be configured to process any suitable subset of the image processing stages in the validation set (e.g., adjacent ones).

[0134] For example, the verified image processing stages may or may not include the first stage from the set. If the verified image processing stages do indeed include the first stage from the set, the method may include verifying the image processing in the set of one or more image processing stages by inserting one or more probes into an image captured by one or more image sensors. However, if the verified image processing stages do not indeed include the first stage from the set, the method may include verifying the image processing in the set of one or more image processing stages by inserting one or more probes into one or more processed forms of one or more images captured by one or more image sensors.

[0135] Similarly, the verified image processing stages may or may not include the final stage in the set. If the verified image processing stages do indeed include the final stage in the set, the method may include verifying the image processing in the set of one or more image processing stages by attempting to identify one or more probes in the output (such as an output image). However, if the verified image processing stages do not indeed include the final stage in the set, the method may include verifying the image processing in the set of one or more image processing stages by attempting to identify one or more probes in a processed form of an image captured by one or more image sensors to produce an output image.

[0136] Therefore, one or more images in which one or more probes are inserted may include images captured by one or more image sensors or processed forms of those images. Similarly, the output analyzed in an attempt to identify one or more probes may include final output, such as an output image (e.g., for display), or intermediate output, such as images to undergo further processing to produce the output image.

[0137] The image processing system may also be configured to (independently) verify image processing performed by two or more different stages or groups of stages in the set, for example, verifying image processing performed by a first stage or group of stages in the set, and independently verifying image processing performed by different second stages or groups of stages in the set. This can be accomplished by the image processing system inserting one or more first probes into the image to be processed by the first stage or group and attempting to identify one or more first probes in the output of the first stage or group; and additionally (e.g., in a corresponding manner) inserting one or more second probes into the image to be processed by the second stage or group and attempting to identify one or more second probes in the output of the second stage or group.

[0138] This can be used to verify a set of image processing stages, where, for example, the image processing performed by the set of stages is relatively complex, making it impossible or highly unlikely for a probe to exist that traverses the entire set of stages.

[0139] Therefore, the image processing system can be configured to verify image processing performed by one or more first image processing stages from the set of one or more image processing stages in the following way:

[0140] Before one or more images are processed by one or more first image processing stages in the set of one or more image processing stages, one or more first probes are inserted into the one or more images; and

[0141] Attempt to identify one or more first probes in the output of one or more first processing stages.

[0142] The image processing system can also be configured to verify image processing performed by one or more second (different) image processing stages from the set of one or more image processing stages in the following way:

[0143] Before one or more images are processed by one or more second image processing stages in the set of one or more image processing stages, one or more second probes are inserted into the one or more images; and

[0144] Attempt to identify one or more second probes in the output of one or more of the second processing stages.

[0145] The image processing system can also be configured to verify, in a corresponding manner, the image processing performed by any number of one or more third (different) image processing stages in the set.

[0146] One or more probes can be inserted into the image in any suitable manner.

[0147] One or more probes should be (and in various embodiments) inserted into one or more images such that each of the one or more probes is processed by one or more image processing stages (i.e., together with one or more images) from a set of one or more image processing stages (verified image processing). In other words, one or more probes may be inserted into one or more images such that each of the one or more probes undergoes the same processing operations (sequence) as the one or more images.

[0148] This means that the output of one or more processing stages will include the processed form of one or more probes (and the processed form of one or more images), for example, one or more processed forms of one or more probes have undergone the same image processing as the processed form of one or more images. Therefore, one or more processed forms of one or more probes will represent the processing undergone by one or more images in the output.

[0149] In various specific implementations, one or more probes are directly inserted into one or more images (to be processed), i.e., using direct spatial coding. To do this, the image data of one or more images can be modified to include one or more probes, for example, in the form of a digital watermark.

[0150] In these implementations, one or more probes may be inserted into each (and every) image (to be processed by one or more image processing stages) in a sequence of images, or one or more probes may be inserted into only some (but not all) of the images (to be processed by one or more image processing stages). For example, one or more probes may be inserted into the images in a sequence of images periodically (or otherwise).

[0151] In these implementations, since the image is to be used to generate the output image, it is desirable to minimize the impact of image modifications caused by probe insertion into the image. This can be achieved in any suitable manner.

[0152] For example, one or more probes or each probe can be removed (e.g., from the output of one or more processing stages) before using (e.g., displaying) the output image.

[0153] Additionally or alternatively, one or more of the probes, or each probe, may be inserted into the image in a manner that makes their impact on the output of one or more processing stages relatively small, for example, so that the probes will be less noticeable (e.g., by operator or consumer circuitry).

[0154] In various specific implementations, one or more probes are inserted into one or more images in a manner that modifies only a small amount of image information per image data location (pixel). Specifically, one or more probes can be inserted into one or more images by using (and modifying) only one bit per image data location (pixel).

[0155] In various specific implementations, the bit used (modified) in this way can be, for example, the least significant bit of one of the color channels. Any suitable color channel can be chosen to do this, but in various implementations, the blue channel is used (modified) because the human eye is least sensitive to this color channel.

[0156] Therefore, in various implementations, one or more probes are inserted into the image by modifying the least significant bit of the data value of one (e.g., blue) color channel at one or more data locations (pixels) of the image. This has the effect of minimizing (or at least reducing) the effect of the probe insertion on the output image.

[0157] Other arrangements are possible. For example, instead of using direct spatial coding, an image can be transformed, and one or more probes can be inserted into the image in the transformed domain (and then (i.e., transformed back to the image using inverse transformation)). For example, quantization indexed modulation (QIM) can be used to insert one or more probes into the image. However, other techniques are also possible.

[0158] In these implementations, the image can be converted to any suitable domain (for probe insertion). Exemplary domains include the Fourier domain, cosine domain, wavelet domain, etc. In these implementations, one or more transformations can be used. For example, the image can first be converted to the Fourier domain, then to the wavelet domain for probe insertion (e.g., using QIM) (and then back again).

[0159] Other arrangements are possible.

[0160] In some implementations, it can be challenging to insert one or more probes directly into an image without causing noticeable artifacts in the final output image when the image (captured by an image sensor) includes regions saturated with, for example, a single color (such as white) (e.g., the region of an image showing clouds).

[0161] In this case (and others), it may be desirable to forgo inserting one or more probes into the image data (i.e., to avoid causing noticeable artifacts in the final output image).

[0162] In these embodiments, information indicating how one or more probes are inserted into the image data can be transmitted to the recognition circuitry system, for example, to allow for appropriate verification of the output image. For instance, this information may optionally be accompanied by information indicating why a probe is omitted (such as information indicating specific regional characteristics that prevent probe insertion into the area) to indicate one or more areas in the image where probes are not included. Transmitting such information to the recognition circuitry system in this way allows, for example, the recognition circuitry system to verify that the characteristics preventing probe insertion into the area have been preserved by image processing.

[0163] In various other embodiments, instead of inserting one or more probes directly into the image data of the image, one or more probes can be inserted into one or more images by inserting one or more "test" images, including one or more probes, into a sequence of images (to be processed) (e.g., between (adjacent) images in the image sequence). For this purpose, the image sequence can be modified to include, for example, one or more additional images containing one or more probes between images in the image sequence.

[0164] In these implementations, test images may be inserted between each (and every) adjacent image pairs in the sequence (to be processed by one or more image processing stages), or test images may be inserted between only some (but not all) adjacent image pairs in the sequence (to be processed by one or more image processing stages). For example, test images may be inserted between adjacent image pairs in the sequence of images periodically (or otherwise). Alternatively, test images may be inserted between adjacent image pairs in the sequence of images only occasionally (e.g., when needed or otherwise).

[0165] Each test image can be configured as needed. For example, a relatively simple image (such as an image containing a single color value) can be modified in the manner described above (and further below) to generate a test image (e.g., by inserting one or more probes into the image).

[0166] Therefore, the step of inserting one or more probes into one or more images before processing one or more images may include: inserting one or more test images into the sequence of images before the sequence of images (and test images) is processed by one or more image processing stages in the set.

[0167] In these implementations, test frames are not required (and are not used in the implementations) (e.g., displayed). Therefore, once the test image has been processed and analyzed, i.e., to attempt to identify one or more probes in the processed test image, they can be removed without being used (e.g., without being displayed).

[0168] It is also possible to combine the two methods described above (i.e., one or more probes are inserted directly into the image data, or one or more probes are inserted into the test image). For example, an image processing system may be configured to switch between these techniques depending on the situation and desired outcome (e.g., when it is determined that one of the methods will provide improved results).

[0169] For example, in cases where an image (captured by an image sensor) includes regions saturated with, for example, a single color (such as white) (e.g., an area in an image showing clouds), inserting one or more probes directly into the image without causing noticeable artifacts in the final output image can be challenging. In this case (and others), instead of inserting one or more probes directly into the image data, one or more probes can be inserted into one or more test images in the manner described above. In these embodiments, information indicating how one or more probes are inserted into the image data can be transmitted to a recognition circuitry system, for example, to allow for appropriate verification of the output image.

[0170] One or more probes inserted into one or more images can be configured in any suitable manner.

[0171] As described above, one or more probes, together with one or more images, are processed by one or more image processing stages in a set, such that the output of one or more processing stages includes the processed form of one or more probes (and the processed form of one or more images). The processed form of one or more probes in the output of one or more processing stages is then used to verify the image processing performed by the one or more image processing stages in the set. Thus, one or more probes should be (and in various embodiments) configured such that when it or they (together with one or more images) undergo image processing, image modifications to the probes represent image modifications made to the one or more images.

[0172] In various implementations, one or more probes are in the form of one or more images, i.e., each probe may be an image probe. However, one or more or each probe may alternatively have some other form, such as a transformed image probe.

[0173] In the case where each probe includes an image probe, each probe may include an array of multiple data locations (pixels), where each data location (pixel) uses a specific data value. The data value of each probe's data location can be any suitable data value.

[0174] One or more probes can have any suitable size and shape. One or more probes can (together) have a size smaller than the image into which they are inserted. Therefore, one or more probes can be inserted into one or more sub-regions of the image (less than the entire image).

[0175] However, in various specific implementations, one or more probes are configured to have the same size as the image into which they are inserted (i.e., fill the image). As further described below, configuring one or more probes to fill the image into which they are inserted enhances the effectiveness of image processing verification.

[0176] Each of the one or more probes (sets) should, and in one implementation, indeed have a suitable shape and / or features to allow identification of the one or more probes (sets) in a manner such as digital watermarking. This can be achieved in any suitable and desirable manner. For example, each of the one or more probes (sets) may be configured to have a suitable two-dimensional shape or pattern that allows identification of the one or more probes (sets).

[0177] In various specific implementations (where one or more probes are inserted into each image in a sequence of images), the configuration of one or more probes changes over time.

[0178] Therefore, inserting one or more probes into one or more images before processing one or more images by one or more image processing stages of a set of one or more image processing stages may include: inserting one or more first probes into one or more first images before processing one or more first images by one or more image processing stages of the set of one or more image processing stages; and then inserting one or more second different probes into one or more second different (subsequent) images before processing one or more second images by one or more image processing stages of the set of one or more image processing stages. Inserting one or more probes into one or more images before processing one or more images by one or more image processing stages of a set of one or more image processing stages may include: inserting one or more third different probes into one or more third different (subsequent) images before processing one or more third images by one or more image processing stages of the set of one or more image processing stages (and so on).

[0179] In various specific implementations, the configuration of one or more probes (sequentially inserted into each image in a sequence of images) is configured to change from one image to the next on a frame-by-frame basis. In other words, different (e.g., unique or quasi-unique) sets of one or more probes can be inserted into each corresponding image in a sequence of images. Similarly, different (e.g., unique or quasi-unique) watermarks can be inserted into each corresponding image in a sequence of images (where each watermark comprises a different set of one or more probes).

[0180] One or more probes inserted into an image may include a single probe. Therefore, an image processing system may be configured to insert probes into images (e.g., insert different probes into each image) and attempt to identify the probes in the output of one or more processing stages.

[0181] In these implementations, each probe can have any suitable configuration. In various implementations, the image processing system can define and use a group of multiple probes, wherein each probe in the group is distinguishable from any other probe in the group, such that each particular probe in the group can be uniquely identified within the output. Different probes from this group can be inserted into each corresponding different image in one or more images (e.g., inserted into each corresponding different image in a sequence of images).

[0182] In these embodiments, the group of multiple probes can be a finite group of probes, and thus the image processing system can be configured to cycle through the probes in the group, that is, to insert the first probe in the group into a first image, and then (e.g., by stepping through the probes in the group from one probe to the next) insert the corresponding different probes in the group into each subsequent image into which the probe is to be inserted (e.g., into each image in a sequence of images), until the last probe in the group is inserted into an image, and then the first probe in the group is inserted into the subsequent images (in the sequence of images) (and continues by stepping through the probes in the group from one probe to the next, and so on).

[0183] In various specific embodiments, one or more probes inserted into an image comprise a set of multiple probes. Therefore, in various embodiments, the image processing system is configured to insert a (different) set of multiple probes into each image and attempt to identify one or more, or each, of the set of multiple probes in the output of one or more processing stages.

[0184] In these embodiments, each probe inserted into the image may be suitably distinguishable from any other probe inserted into the image, such that each particular probe in the set of probes can be uniquely identified within the output (if present). Thus, in one embodiment, each probe in the set of multiple probes is uniquely identifiable within the set of multiple probes (and distinguishable from all other probes in the set).

[0185] In various specific embodiments, the image processing system defines and uses a group of multiple probes (where each probe in the group is distinguishable from any other probe in the group, such that each particular probe in the group can be uniquely identified, for example, within the output), and each set of multiple probes inserted into an image consists of (different) combinations (subsets) of probes from that group. Thus, in one embodiment, the image processing system defines and uses a group of multiple probes (where each probe in the group is uniquely identifiable within the group and distinguishable from all other probes in the group), and inserts (different) combinations (subsets) of probes from that group into each image.

[0186] As described above, when the group of multiple defined probes is a finite group of defined probes, inserting different combinations (subsets) of probes from that group into each image in this way allows for a much larger pool of defined and usable probes (i.e., watermarks) (e.g., when compared to inserting only a single probe into each image). This means that when the image processing system is configured to circulate through the set of probes in the pool (in a manner corresponding to the above), each unique set of probes (each unique watermark) is advantageously used at a less frequent frequency than in the original case.

[0187] As will be further described below, using a set of multiple probes can also enhance the effectiveness of image processing verification.

[0188] When a set of multiple probes is inserted into an image, each probe in the set can be positioned at a specific location within the image. In one embodiment, each probe in the set can be positioned at a location within the image different from any other probe in the set.

[0189] The size, shape, and number of multiple probes can be selected to fill less than all of the image into which the probes are inserted (only one or more sub-regions of the image into which the probes are inserted). However, in various specific embodiments, the size, shape, and number of probes are selected to fill the image. Thus, a collection of multiple probes can together have a size equal to the size of the image into which they are inserted.

[0190] In various specific implementations, the multiple probes in the set can each be distributed throughout the image, such as uniformly (spatially) throughout the image. For example, in the case of inserting two probes into the image, each probe can be inserted into each half (side) of the image. In the case of inserting four probes into the image, each probe can be inserted into the corresponding quadrant of the image, and so on.

[0191] In various specific implementations, the set of multiple probes inserted into each image includes a set of multiple probes (e.g., an array of probes) that are uniformly (spatially) distributed throughout the image and configured to fill the image.

[0192] Each probe should (and in one embodiment does) have an appropriate shape and / or features to allow probe identification. This can be achieved in any suitable and desirable manner. For example, each probe may be configured to have an appropriate two-dimensional shape or pattern that allows probe identification. Each probe may, for example, include a square or rectangular array of data locations with a two-dimensional pattern that allows probe identification.

[0193] In the techniques described herein, an attempt is made to identify one or more probes in the output of one or more processing stages.

[0194] As described above, one or more probes inserted into an image undergo the same image processing as one or more images. Therefore, attempting to identify one or more probes in the output of one or more processing stages can include one or more processed types attempting to identify one or more probes (inserted into one or more images) in the output.

[0195] As described above, the output may include one or more images, such as a sequence of images, wherein each output image may include (e.g., for display) an output image or an output image to be further processed to produce an output image. Therefore, attempting to identify one or more probes in the output of one or more processing stages may include attempting to identify one or more probes in one or more or each output image of the output of one or more processing stages.

[0196] The method used to attempt to identify one or more probes will depend on how the probes are inserted into the image. For example, if one or more probes are directly inserted into the image, the output of one or more processing stages can be directly analyzed to attempt to detect one or more probes (in their processed form). Alternatively, it may be necessary to transform the output image before attempting to identify one or more probes in the transformation domain.

[0197] The nature of image processing performed by one or more image processing stages may cause each (all) of the one or more probes inserted into the image to be expected to appear in the output. Alternatively, the nature of image processing may cause one or more of the one or more probes inserted into the image to be not expected to appear in the output, for example, where the image processing operations include cropping and / or (reducing), etc.

[0198] Therefore, attempting to identify one or more probes in the output may include attempting to identify each (all) of the one or more probes inserted into one or more images, or attempting to identify only a subset (less than all) of the one or more probes inserted into one or more images.

[0199] In various specific implementations, attempting to identify one or more probes in the output includes attempting to identify one or more expected probes in the output, wherein each expected probe is a probe that is expected to appear in the output based on one or more image processing stages.

[0200] If one or more of the expected probes cannot be identified in the output, it can be determined that one or more image processing stages have not performed their image processing operations accurately enough.

[0201] This could be due to situations where, for example, image processing has caused the output image to become excessively blurry, corrupted, and / or noisy. Such inaccuracies in image processing will be reflected in the processed form of one or more probes in the output, and may cause the image processing system to fail to recognize one or more of the probes (i.e., because they are "difficult to read").

[0202] Therefore, when one or more of the (expected) probes or each of the (expected) probes can be identified in the output, it can be determined that the output of one or more processing stages is not (different from) excessively blurry, corrupted, and / or noisy.

[0203] Similarly, the processing circuitry can be configured to verify image processing performed by one or more image processing stages from the set of one or more image processing stages in the following way:

[0204] When one or more of the (expected) probes can be identified in the output, verify that the output of one or more processing stages is not (different from) excessively blurry, corrupted, and / or noisy.

[0205] The processing circuitry can be configured to determine that the output is excessively blurry, corrupted, and / or noisy when it is not possible to identify one or more or each of the one or more (expected) probes in the output.

[0206] This could also be due to situations where, for example, image processing has incorrectly cropped and / or scaled the image. Such inaccuracies in image processing can cause unexpected probes to appear in the output and / or expected probes to not exist in the output.

[0207] Therefore, when each of the expected probes in one or more expected probes can be identified in the output (and there are no unexpected probes), it can be determined that the output of one or more processing stages has been cropped and / or scaled in the expected manner.

[0208] Similarly, the processing circuitry can be configured to verify image processing performed by one or more image processing stages from the set of one or more image processing stages in the following way:

[0209] When each of the expected probes in one or more expected probes can be identified in the output (and there are no unexpected probes), verify that the output of one or more processing stages has been cropped and / or scaled in the expected manner.

[0210] The processing circuitry can be configured to determine that the output has been cropped and / or scaled in an unintended manner when one or more expected probes cannot be identified in the output (and / or when one or more expected probes can be identified in the output).

[0211] This can also be a situation where, for example, the output from one or more processing stages does not correspond to one or more input images. This can happen, for example, where the image processing system (as described above) includes multiple different image sensors, and the same one or more image processors are configured to process images from multiple different image sensors. For example, an image from one image sensor may (unintentionally) be exchanged with an image from another image sensor.

[0212] Therefore, when one or more of the (expected) probes or each (expected) probe can be identified in the output (and there are no unexpected probes), it can be determined that the output of one or more processing stages corresponds to one or more images input to one or more processing stages.

[0213] Similarly, the processing circuitry can be configured to verify image processing performed by one or more image processing stages from the set of one or more image processing stages in the following way:

[0214] Verify that the output of one or more processing stages corresponds to one or more images input to one or more processing stages when one or more (expected) probes or each (expected) probe can be identified in the output (and there are no unexpected probes).

[0215] The processing circuitry can be configured to determine that the output does not correspond to one or more images input to one or more processing stages when it is not possible to identify one or more of the one or more (expected) probes in the output.

[0216] Therefore, the processing circuitry can be configured to verify that the output is, for example, the expected output from the expected image sensor.

[0217] As described above, in various implementations, one or more (different) probes are inserted into each image in a sequence of images, and the configuration of one or more probes can change over time.

[0218] Including time-varying watermarks in each image in a sequence of images in this way allows the image processing system to verify that the output of one or more processing stages is changing in time (e.g., not frozen). This also allows the image processing system to verify that the images (frames) output by one or more processing stages are generated at the expected (e.g., sufficient) rate (i.e., frame rate). Furthermore, this allows the image processing system to verify that the images output by one or more processing stages are generated in the correct order.

[0219] Therefore, in various implementations, the processing circuitry is configured to verify the image processing performed by one or more image processing stages from the set of one or more image processing stages in the following way:

[0220] Before the sequence of images is processed by the one or more image processing stages in the set of one or more image processing stages, corresponding differences (sets) of one or more probes are inserted into each image in the sequence of images; and

[0221] Attempt to identify each probe in a different (set) of one or more probes in the output of one or more processing stages.

[0222] When each probe in one or more (expected) probes (of a set) can be identified in the output, it can be determined that the output of one or more processing stages changes in a timely manner to be generated at the expected rate and / or in the correct order.

[0223] Similarly, the processing circuitry can be configured to verify image processing performed by one or more image processing stages from the set of one or more image processing stages in the following way:

[0224] When each of the (expected) probes in the set of one or more (expected) probes can be identified in the output, verify that the output of one or more processing stages changes in a timely manner, is generated at the expected rate and / or in the correct order.

[0225] The processing circuitry can be configured to determine that the output is not changing in a timely manner (frozen), is being produced at an unexpected (e.g., slow) rate and / or in an incorrect order when one or more of the (set of) one or more (expected) probes cannot be identified in the output (and / or when one or more of the (set of) one or more (expected) probes can be identified as appearing in the output at an unexpected (e.g., slow) rate and / or in an incorrect order).

[0226] The nature of the image processing should be such that one or more probes inserted into one or more images are expected to appear at a specific spatial location in the output (e.g., relative to the output image or relative to one or more other probes in the output image). This could be a case where, for example (as described above), the image is flipped or rotated.

[0227] Therefore, when one or more of the (expected) probes or each of the (expected) probes can be identified at the expected spatial location in the output, it can be determined that the output of one or more processing stages has been correctly flipped and / or rotated.

[0228] Similarly, the processing circuitry can be configured to verify image processing performed by one or more image processing stages from the set of one or more image processing stages in the following way:

[0229] When each of one or more (expected) probes can be identified at the expected location in the output, verify that the output of one or more processing stages has been correctly flipped and / or rotated.

[0230] The processing circuitry can be configured to determine that the output has been incorrectly flipped and / or rotated when one or more of the (expected) probes cannot be identified at the expected location in the output and / or one or more of the (expected) probes can be identified at an unexpected location in the output.

[0231] Therefore, it should be understood from the above that if one or more expected probes cannot be identified in the output (or if one or more expected probes are identified in the output at unexpected time and / or spatial locations, or if one or more unexpected probes are identified in the output), it can be determined that the one or more image processing stages have not (sufficiently) accurately performed their image processing operations. On the other hand, if each of the one or more expected probes can be identified in the output (e.g., at the corresponding expected time and / or spatial location) (and if no expected probe is identified in the output), it can be determined that the one or more image processing stages have (sufficiently) accurately performed their image processing operations.

[0232] Therefore, attempting to identify one or more probes in the output of one or more processing stages allows the image processing system to determine the fidelity of the image processing operations performed by the one or more image processing stages.

[0233] Since the one or more processing stages are part of a set of processing stages configured to produce output images (e.g., for display) from the images captured by the image sensor, this allows the image processing system to determine whether the output images are a sufficiently accurate representation of the environment. If it is determined that the output images are not a sufficiently accurate representation of the environment, the image processing system may, for example, indicate this to the operator and / or consumer circuitry to prevent (e.g., the operator and / or consumer circuitry) from using the images in an inappropriate or unsafe manner.

[0234] Therefore, the implementation provides a relatively simple and robust technique for verifying image processing operations performed by one or more image processors.

[0235] Therefore, it should be understood that various implementation schemes provide improved image processing systems.

[0236] The techniques described herein can be implemented in any suitable system, such as a properly configured microprocessor-based system. In one embodiment, the techniques described herein are implemented in a computer and / or microprocessor-based system.

[0237] The various functions of the technology described herein can be performed in any desired and suitable manner. For example, the functions of the technology described herein can be implemented in hardware or software as needed. Therefore, for example, unless otherwise specified, the various functional elements, stages, and “devices” of the technology described herein may include suitable one or more processors, one or more controllers, functional units, circuit systems, circuits, processing logic units, microprocessor arrangements, etc., which are operable to perform various functions, such as suitable dedicated hardware elements (processing circuits / circuit systems) and / or programmable hardware elements (processing circuits / circuit systems), which can be programmed to operate in a desired manner.

[0238] It should also be noted here that, as those skilled in the art will understand, the various functions of the techniques described herein can be copied and / or executed in parallel on a given processor. Similarly, various processing stages can share processing circuitry / circuits, etc., if desired.

[0239] Those skilled in the art should also understand that all embodiments of the technology described herein may, as appropriate, include any one or more or all of the features described herein in one embodiment.

[0240] The methods described herein can be implemented at least in part using software, such as computer programs. Therefore, it can be seen that, when viewed from another embodiment, the techniques described herein provide: computer software particularly suitable for performing the methods described herein when installed on a data processor; computer program elements including computer software code portions for performing the methods described herein when the program elements are run on the data processor; and a computer program including code suitable for performing all steps of one or more methods described herein when the program is run on a data processing system. The data processor may be a microprocessor system, a programmable FPGA (Field-Programmable Gate Array), etc.

[0241] The techniques described herein also extend to computer software carriers that include such software, when used to operate a display controller or a microprocessor system including a data processor, cause the controller or system to engage the data processor to perform the steps of the methods described herein. Such computer software carriers can be physical storage media, such as ROM chips, CD-ROMs, RAM, flash memory, or disks, or they can be signals, such as electronic signals transmitted through wires, optical signals, or radio signals, such as signals to satellites.

[0242] It should also be understood that not all steps of the method described herein need to be performed by computer software. Therefore, in contrast to another broad implementation, the technology described herein provides computer software and such software installed on a computer software carrier for performing at least one step of the method described herein.

[0243] Therefore, the techniques described herein can be suitably embodied as computer program products used with computer systems. Such embodiments may include a set of computer-readable instructions fixed on a tangible, non-transitory medium, such as a computer-readable medium, for example, a disk, CD-ROM, ROM, RAM, flash memory, or hard disk. It may also include a set of computer-readable instructions that can be transmitted to a computer system via a modem or other interface device through a tangible medium (including, but not limited to, optical or analog communication lines) or passively using wireless technologies (including, but not limited to, microwave, infrared, or other transmission technologies). This set of computer-readable instructions embodies all or part of the functions previously described herein.

[0244] Those skilled in the art will understand that such computer-readable instructions can be written in a variety of programming languages ​​to be used with many computer architectures or operating systems. Furthermore, such instructions can be stored using any current or future memory technology (including, but not limited to, semiconductor, magnetic, or optical technologies), or transmitted using any current or future communication technology (including, but not limited to, optical, infrared, or microwave technologies). It is conceivable that such computer program products can be distributed as removable media with accompanying printed or electronic documentation (e.g., shrink-wrapped software), pre-loaded onto a computer system (e.g., on a system ROM or fixed disk), or distributed from a server or electronic bulletin board via a network (e.g., the Internet or the World Wide Web).

[0245] Implementation schemes of the technologies described herein will now be described.

[0246] The implementation plan involves a security monitor for an image processing system.

[0247] Automotive mirror replacement systems are designed to replace or supplement vehicle mirrors (such as side mirrors) using a camera-based system. Images captured by an image sensor or camera are modified by an image processor so that they are displayed in an appropriate format to the vehicle operator on a display panel.

[0248] The applicant has determined that, in order for such a system to meet appropriate automotive safety standards, the system will need to operate, for example, with a relatively high frame rate. The system will also need to be configured to capture images using a higher resolution and / or a larger size than would be possible or convenient to display on a panel within the vehicle.

[0249] For example, it may be necessary to capture images at a relatively high resolution and / or a relatively large size, where the optical system is configured to provide a wide field of view in a manner that requires correction of the captured image (e.g., when using a fisheye lens or similar lens), while it may be necessary to display the image at a lower resolution and / or a smaller size depending on the panel's maximum resolution and / or size. Similarly, it may be necessary to capture images at a relatively high resolution and / or a relatively large size to allow for techniques such as automatic image recognition on the images, while it may be necessary to display the image at a lower resolution and / or a smaller size depending on the panel's maximum resolution and / or size. Similarly, it may be necessary to capture images at a relatively high resolution and / or a relatively large size to enable "dynamic mirror" techniques, where only a portion of the captured image (e.g., the central area of ​​the image) can be displayed in "normal" mode, but other portions (e.g., the side areas) of the captured image can be displayed at other times (e.g., when a vehicle is turning).

[0250] This may require reducing and / or cropping the images captured by the image sensor. For example, a magnification of approximately 0.19 to 0.3 may be necessary.

[0251] For example, when displaying an image as a rearview or side mirror for the driver's use, it may be necessary to flip the image, i.e., to create a mirror image. Image enhancement may also be necessary, for example, to improve the understandability of the image displayed to the driver. Other image processing operations may be required or expected.

[0252] This means that in these systems, the captured images need to be processed by one or more image processors.

[0253] Therefore, as Figure 1 As shown, images captured by an image source 10, such as a camera (e.g., the environment surrounding a vehicle), can undergo a set of one or more image transformations performed by one or more image processors 12 to produce an output image for display. The image source 10 and image processors 12 can be controlled by a controller 14, for example, to instruct the image source 10 when it should provide the next frame to one or more image processors 12 for processing, and to control the operating mode of one or more image processors (e.g., selecting a specific set or configuration of one or more image transformations to be applied to a particular frame). The output image for display can then be provided to an image destination 16 (such as a display).

[0254] The applicant has recognized that many possible processing errors can occur in such systems, and that such errors can significantly affect the safety of the vehicle.

[0255] For example, in the case of a desired set of image processing operations using multiple image processors, it will be necessary to write intermediate image data from one image processor so that it can be read in (and processed) by another image processor. That is, multiple image processors can be arranged in a pipeline configuration, where intermediate images are stored and / or cached in memory.

[0256] Similarly, in the presence of two (or more) image sensors, it may be desirable to use only a single image processor (a set of image processors) to process both image feeds. This might necessitate, for example, buffering images from one image sensor while processing an image from another.

[0257] However, (among other things) using caching implies the possibility of delayed images or portions of images. This can cause, for example, a relatively slow frame rate, which is problematic from a security perspective. Furthermore, portions of different images (frames) in an image and / or image sequence may appear out of order. This can cause distortion of the displayed image, which is also problematic from a security perspective. Other image processing errors may also occur.

[0258] Similar problems may occur when image processing is performed within software.

[0259] Therefore, for security reasons, there is a need to configure an image processing system.

[0260] The conventional approach to doing this would involve over-engineering systems, for example, through exhaustive analysis of the system to limit potential hazards, including redundancy checks. However, this would require extensive engineering and result in larger chips, which would require more power and cost.

[0261] The technology described herein provides a monitor that can verify that image transformations applied within a processing pipeline appear to be genuinely correct for security-related applications.

[0262] At the start of the image processing pipeline, one or more probes or "watermarks" are inserted into the image. The final result is then verified by observing the output from the pipeline, for example, to check the transformations applied through the pipeline.

[0263] The techniques described herein can be used to provide additional integrity checks in hazardous situations (e.g., in addition to conventional techniques) or to provide means for monitoring the entire system without additional high-integrity engineering.

[0264] Figure 2 An implementation scheme is illustrated schematically. For example... Figure 2 As shown, the image processing system includes a watermark generation stage 20, which operates under control on a controller 14 to generate a watermark for each frame generated by the image source 10.

[0265] The watermark embedding stage 22 is configured to embed each watermark into each frame generated by the image source 10. The frames with their embedded watermarks are then passed to the image processor 12 and undergo appropriate image transformations before being provided to the image destination 16 (e.g., a display) for display.

[0266] The image processing system also includes a watermark recovery and verification stage 24. The converted image (including the watermark) is provided to the watermark recovery and verification stage 24, and the watermark recovery and verification stage 24 is configured to attempt to recover and verify (i.e., identify) the watermark in the image. This can be accomplished by first attempting to identify the presence of the watermark in the image, and then attempting to identify that specific watermark (from a set of known, qualified watermarks). Figure 2 As shown, the watermark recovery and verification phase 24 uses metadata describing each watermark and information indicating the operating modes of one or more image processors 12 in its analysis.

[0267] The image processing system also includes a transformation and metadata verification stage 26, which is configured to determine whether one or more image processing stages have (sufficiently) accurately performed their respective transformations. This is accomplished by analyzing the processed watermark along with information indicating the operating modes of one or more image processors 12, i.e., determining whether each watermark appears at the expected time and spatial location.

[0268] When the watermark undergoes the same image transformation as the frame, by analyzing the transformed watermark in the output frame, the system can check whether one or more image processing stages have (sufficiently) performed their respective transformations accurately.

[0269] If a watermark cannot be identified in the output (or if it is identified at an unexpected time or spatial location within the output), it can be determined that one or more image processing stages have not (sufficiently) accurately performed their image processing transformation. Conversely, if a watermark can be identified in the output (at the expected time and / or spatial location), it can be determined that one or more image processing stages have (sufficiently) accurately performed their image processing transformation.

[0270] Therefore, attempting to identify watermarks in the output of one or more processing stages allows an image processing system to determine the fidelity of the image transformations performed by one or more image processing stages. This, in turn, allows the image processing system to determine whether the displayed image is a sufficiently accurate representation of the environment.

[0271] Therefore, the implementation provides a relatively simple and robust technique for verifying image processing transformations performed by one or more image processors 12.

[0272] Figure 3 A more detailed description is provided based on one implementation scheme. Figure 2 An image processing system. Figure 3 A sequence of image processing operations (transformations) is shown in detail, which can be applied to an image generated by a source 10 (e.g., a camera) and then provided to its destination 16 (e.g., a display).

[0273] like Figure 3 As shown, an image may undergo one or more, or all of the following: demosaicing and noise reduction 30, tone mapping and white balance 32, scaling such as downsampling 34, image cropping 36, and / or image compositing 38. Of course, other types of image processing and other pipelines are also possible.

[0274] In some implementations, the entire conversion sequence is verified. However, more generally, a monitor can operate simultaneously at a single stage, a group of stages, and / or the entire pipeline within the processing pipeline. Therefore, various implementations provide system-level monitors to track conversions applied by each IP block, group of blocks, or the entire media pipeline.

[0275] For example, such as Figure 3 As shown, a watermark can be embedded after one or more transformations in an already applied transformation.22

[0276] In cases where a particular processing pipeline is relatively complex (making it difficult for a watermark to survive sufficiently to be usable), the techniques described herein can be used to verify sub-segments of the pipeline. Therefore, this process can be performed on a first subset of the pipeline's processing stages, and on a different second subset of the pipeline's processing stages (and so on).

[0277] In this embodiment, the watermark is constructed as machine-readable information in a set of spatially distributed probes (each probe comprising a square or rectangular grid of pixels) and embedded in the image fed into the pipeline using steganography. This information is recovered at the end of the pipeline.

[0278] A finite set of square or rectangular probes can be defined, and multiple such probes can be inserted into each frame. Therefore, each watermark can consist of a specific set of multiple probes. This allows for the use of a relatively small amount of information to define a relatively large set of unique watermarks.

[0279] Each watermark may include any number of these probes. For example, probes may be inserted into each quadrant of the image. It can then be determined whether the watermark appears in each quadrant of the processed image as expected at the end of the pipeline. This will allow, for example, the system to determine whether the correct reflection has been applied, and whether the truncation (cropping) and / or resizing has been correctly applied.

[0280] However, this would not allow for the detection of errors, for example, in the center of an image. Therefore, in another implementation, multiple probes are embedded in each image, such that the probes cover most or all of the image.

[0281] Figure 4 A conceptual illustration of an image with an embedded watermark. (e.g.) Figure 4 As shown, the watermark of the image consists of an array of probes spatially distributed throughout the image.

[0282] Since only apparent authenticity is required, successful decoding of the embedded probe array allows verification, for example: (a) probes expected to have traversed the pipeline are evident in the result; (b) probes expected to have undergone transformation are positioned appropriately in the result; (c) probes originating from the left region of the input image appear to the right of the result where reflection is expected; and / or (d) the constituent pixels of the probes expected to be reflected are swapped to the left / right.

[0283] Using direct spatial coding, for example, each watermark can be inserted into an image by modifying the least significant bit (LSB) of the data value of one color channel for each pixel. For example, for RGB color values, the least significant bit of the blue channel can be modified because the human eye is least sensitive to the blue channel. This has the effect of minimizing (or at least reducing) the effect of the watermark insertion on the displayed image.

[0284] In this implementation, the embedded information can be directly extracted by analyzing the least significant bit (LSB) of the modified color channel.

[0285] Various other techniques can be used to embed watermarks into images. For example, interpolation can be used.

[0286] In some implementations, a watermark can be inserted into an image by converting the image and inserting the watermark into the converted image (and then converting the image in reverse). For example, Quantization Index Modulation (QIM) can be used to insert the watermark.

[0287] In these implementations, the image can be converted to any suitable domain for watermarking. Exemplary domains include the Fourier domain, cosine domain, wavelet domain, etc. Similarly, one or more transformations can be used. For example, the image can first be converted to the Fourier domain, then to the wavelet domain for watermarking (e.g., using QIM) (and then back again). Other arrangements are possible.

[0288] The applicant has also recognized that there may be situations where the use of watermarking techniques other than those described above is desirable. For example, it may be challenging to perform watermarking without causing noticeable artifacts in the final displayed image when the image includes areas saturated with a single color (such as white), for example, in areas of an image showing clouds.

[0289] In this (and other) case, instead of modifying the image itself to include the watermark, test frames can be inserted (e.g., interleaved) into the image stream. Each test frame can be configured such that it does not negatively affect the watermark, and the watermark can be embedded in these images in addition to or instead of the original captured image. For example, each test frame can be a composite (e.g., medium grayscale) image.

[0290] Test frames can be embedded in the image stream, allowing them to undergo the same processing as the original captured image. However, it is not necessary to display the test frames.

[0291] Alternatively or additionally, the watermark can be removed from the image before it is displayed, i.e., the watermark can be a reversible watermark.

[0292] Figure 5 An implementation scheme of an image processing system configured in this manner is shown. Figure 5 Implementation plan and Figure 2 The similarity of the implementation scheme is that the image processing system includes a watermark generation stage 20, a watermark embedding stage 22, and one or more image processors 12.

[0293] However, in Figure 5 In the implementation scheme, the watermark recovery and verification stage 24 is replaced by the watermark recovery, verification and reversal stage 28, which is also configured to recover and verify the watermark in the pipeline output and to remove the watermark before displaying the image.

[0294] For example, a watermark may not be included in frames or regions of frames where it could cause noticeable artifacts (such as saturated white areas).

[0295] like Figure 2 and Figure 5 As shown, in these implementations (and other implementations), the watermark embedding stage 22 (encoder) may provide the watermark recovery and verification stages 24, 28 (decoder) with information (e.g., in the form of metadata) instructing them on how to process a particular frame or frame region.

[0296] For example, the watermark embedding stage 22 can provide the watermark recovery and verification stages 24 and 28 with information (metadata) indicating that the watermark has not yet been included in a specific frame or frame region. The watermark recovery and verification stages 24 and 28 can then use this information in their analysis.

[0297] For example, for frames or frame regions that should not contain a watermark, watermark recovery and verification stages 24 and 28 can at least determine whether this is the case. (This can be useful because, in general, a watermark-free natural image may always be decoded as if some object were present; and it may be impossible to distinguish between the absence of a watermark and the presence of a severely damaged watermark, since entropy is high in both cases.) This determined result can then be provided to verification stage 26 in the manner described above and used in its analysis.

[0298] Metadata can be used, either additionally or alternatively, to convey the reason for omitting probes, and this can be verified in the output. For example, metadata indicating that most of the probe area is at its maximum / minimum value (per color channel) can be transmitted. Therefore, this characteristic of the image can be verified during recovery (within tolerances).

[0299] As mentioned above, there are many potential hazards in automotive mirror replacement systems.

[0300] For example, a potential danger is a blank screen, i.e., a situation where no image is displayed.

[0301] This could be considered a low-risk hazard because the vehicle operator can easily determine that the function is unavailable. However, under specific conditions of use, it could be dangerous.

[0302] Using time-varying watermarks (as described above) allows verification that frames delivered to the display contain valid watermarks.

[0303] Another potential danger is image freezing, where the displayed image does not change to reflect changes in the physical environment. For example, the display might show an empty road when a vehicle is actually approaching, or it might show a stationary vehicle that has already passed.

[0304] This can be considered a high-risk danger because the displayed images may appear realistic in some cases and therefore may be trusted by the vehicle operator.

[0305] Using time-varying watermarks (as described above) allows verification that the frames delivered to the display are different and delivered in the correct order.

[0306] Another potential danger is a blurred / corrupted image. A blurred image is one in which detail / fidelity is reduced; details that should be visible may not be. A corrupted image is one in which all or part of the image contains data unrelated to the scene; that is, corrupted pixels are random or come from other locations in the image. Corruption can occur at the level of a single pixel or a group of larger pixels.

[0307] Blurring / damage can mask the presence of another road user or obstacle. In both cases, there is a risk that the impact is minor (e.g., because it may be confined to a small area, or that the image patch may appear to have shifted realistically from another location) and that the driver will not notice the malfunction.

[0308] Using time-varying watermarks (as described above) allows verification that frames delivered to the display contain watermarks that are sufficiently legible.

[0309] Another potential danger is noisy images. Noisy images are images created by modulating spatially / temporally uncorrelated random pixels with random signals.

[0310] Here, there is a moderate risk that the loss of fidelity may obscure important details. However, this can be mitigated by the high probability that the vehicle operator will recognize the severe noise. In severe cases, this can be similar to the danger of a blank screen.

[0311] Using time-varying watermarks (as described above) allows verification that frames delivered to the display contain watermarks that are sufficiently legible.

[0312] Another potential danger is the inability to simulate mirror physics (image non-reversal). Drivers of mirror systems are accustomed to "flipping" images from left to right. Mirror replacement systems can simulate this behavior to avoid increasing the cognitive load on the driver.

[0313] The inability to simulate this behavior is considered a high-risk danger because the displayed images may appear realistic in many cases and therefore may be trusted by the vehicle operator.

[0314] Using watermarks (as described above) allows verification that frames delivered to the display have the proper spatial arrangement of watermark probes.

[0315] Another potential danger is the display of incorrect spatial regions, that is, the presentation of seemingly realistic images of incorrect spatial regions. This is considered a medium-risk danger because the system may only have one operating mode, allowing for easier detection by the operator.

[0316] Using watermarks (as described above) allows verification that the frames delivered to the display have an appropriate subset of watermark probes.

[0317] Another potential hazard occurs when an incorrect magnification factor is applied, resulting in a seemingly realistic image but the vehicle being larger / smaller than expected, thus misleading the driver about the distance to the vehicle. This is considered a moderate risk hazard due to the increased risk of misjudging distance.

[0318] Using watermarks (as described above) allows verification that the frames delivered to the display have an appropriate subset of watermark probes.

[0319] Another potential danger is the possibility of the left / right camera images being swapped, meaning an image captured on the left side of the vehicle is displayed as if it were captured on the right, and vice versa. This is considered a high-risk hazard because the driver's judgment may be impaired, potentially leading to unsafe maneuvering. Because the displayed image is often accurate, it may be trusted by the vehicle operator.

[0320] Using time-varying watermarks (as described above) allows verification that the frames delivered to the display originate from the correct source.

[0321] In embodiments of the invention, action can be taken if an image is determined to contain errors. The nature of the action will depend on the configuration of the specific system and device (e.g., a vehicle).

[0322] For example, in cases where only a relatively brief error is detected (which subsequently stops), an error log can be stored, for example, for service engineers to investigate later.

[0323] In the event of a persistent error, the car mirror replacement system can be disabled and the operator can be notified (e.g., by displaying an error message or providing other auditory, physical, or other notifications). In this case, an alternative system, such as a radar / blind spot detection system, can be used.

[0324] In this embodiment, a unique or quasi-unique watermark is used to watermark each image in the image sequence generated by source 10. However, it is possible (e.g., periodically) to watermark less than all images in the image sequence.

[0325] Although the above implementation scheme has been described in relation to an automotive mirror replacement system, the techniques of various implementation schemes can be used in other contexts.

[0326] For example, this technique can be used in medical devices such as endoscopes. In this case, fisheye lenses can introduce barrel distortion that can be corrected by an image processor.

[0327] This technique can also be used in other arrangements where the output image is not necessarily displayed. For example, the output image can be used by a machine learning system, such as being configured to perform image recognition. Such implementations can use... Figure 5 The arrangement of the image, in which the watermark is removed, for example, to avoid unintentionally providing “opposite” images to machine learning systems (or other situations).

[0328] More generally, this technique can be advantageously used in any suitable arrangement where errors in image processing may be problematic or dangerous.

[0329] As can be seen from the above, the techniques described herein, in their implementations, provide at least a relatively simple and robust skill for verifying image processing operations performed by one or more image processors. In implementations of the techniques described herein, this is achieved at least by inserting one or more probes into one or more images before processing them by one or more image processing stages from a set of one or more image processing stages, and then attempting to identify the one or more probes in the output of the one or more processing stages.

[0330] The specific embodiments described above are presented for illustrative and descriptive purposes only. They are not intended to be exhaustive or to limit the technology to the precise forms disclosed. Many modifications and variations are possible in accordance with the above teachings. The described embodiments were chosen to best explain the principles of the technology and its practical application, thereby enabling others skilled in the art to best utilize the technology in various embodiments and to have various modifications suitable for the particular intended use. The scope of the invention is intended to be defined by the appended claims.

Claims

1. An image processing system, comprising: One or more image sensors, the one or more image sensors being configured to capture images; One or more image processors are configured to process images captured by one or more image sensors to produce an output image, wherein the one or more image processors include a set of one or more image processing stages, each stage being configured to modify the image by performing a corresponding image processing operation; One or more circuits configured to use an output image generated by the one or more image processors; and A processing circuit system configured to verify one or more image processing operations performed by one or more image processing stages from a set of one or more image processing stages in the following manner: Before modifying one or more images by performing one or more image processing operations by one or more image processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images such that the one or more probes are modified together with the one or more images by the one or more image processing stages in the set of one or more image processing stages; as well as An attempt is made to identify one or more expected probes in the output of the one or more processing stages, wherein each expected probe is a probe that is expected to appear in the output based on the one or more image processing operations performed by the one or more image processing stages.

2. The image processing system of claim 1, wherein the processing circuitry is configured to insert the one or more probes into the one or more images by modifying the one or more images.

3. The image processing system according to claim 1 or 2, wherein: The one or more images include an image sequence; and The processing circuitry is configured to insert one or more probes into one or more images by inserting one or more test images into the image sequence.

4. The image processing system of claim 1 or 2, wherein the image processing system is configured to remove or discard one or more probes before using the output image.

5. The image processing system of claim 1 or 2, wherein the one or more probes comprise a set of multiple probes, and wherein the processing circuitry is configured to insert the set of multiple probes into an image such that each probe is uniformly distributed throughout the image.

6. The image processing system of claim 1 or 2, wherein the image processing system is configured to change the configuration of the one or more probes over time.

7. The image processing system of claim 1 or 2, wherein the image processing system is configured to determine that the one or more image processing stages have not been accurately performed if: One or more expected probes could not be identified in the output; Identify one or more expected probes in unexpected time and / or spatial locations within the output; and / or One or more unexpected probes are identified in the output.

8. The image processing system according to claim 1 or 2, wherein the image processing system is configured to determine that the one or more image processing stages have been accurately performed if: Able to identify one or more expected probes in the output; Able to identify one or more expected probes at the expected time and / or spatial location in the output; and / or One or more unexpected probes could not be identified in the output.

9. The image processing system according to claim 1 or 2, wherein the set of one or more image processing stages includes one or more of the following: a scaler, a cropping stage, a flipping stage, one or more image enhancement stages, a synthesizer, a filter, and a rotation stage.

10. An image processor, comprising: An image processing circuit system configured to process an image to produce an output image, wherein the image processing circuit system includes a set of one or more image processing stages, each stage being configured to modify the image by performing a corresponding image processing operation; and A processing circuit system configured to verify one or more image processing operations performed by one or more image processing stages from a set of one or more image processing stages in the following manner: Before one or more images are modified by one or more processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images such that the one or more probes are modified together with the one or more images by the one or more image processing stages in the set of one or more image processing stages; as well as An attempt is made to identify one or more expected probes in the output of the one or more processing stages, wherein each expected probe is a probe that is expected to appear in the output based on the one or more image processing operations performed by the one or more image processing stages.

11. A method of operating an image processor, the method comprising: The image is modified to produce an output image by performing corresponding image processing operations in each of one or more image processing stages in a set of image processing stages. and The following methods are used to verify one or more image processing operations performed by one or more image processing stages from the set of one or more image processing stages: Before one or more images are modified by one or more processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images such that the one or more probes are modified together with the one or more images by the one or more image processing stages in the set of one or more image processing stages; as well as An attempt is made to identify one or more expected probes in the output of the one or more processing stages, wherein each expected probe is a probe that is expected to appear in the output based on the one or more image processing operations performed by the one or more image processing stages.

12. The method of claim 11, further comprising: Use and / or display the one or more output images.

13. The method according to claim 11 or 12, comprising: The probes are inserted into the one or more images by modifying the one or more images.

14. The method of claim 11 or 12, wherein the one or more images comprise an image sequence, and the method comprises: The probes are inserted into the one or more images by inserting one or more test images into the image sequence.

15. The method according to claim 11 or 12, further comprising: The configuration of the one or more probes may be changed over time.

16. The method according to claim 11 or 12, further comprising: The image processing is determined to have not been performed accurately in the following circumstances: One or more expected probes could not be identified in the output; Identify one or more expected probes in unexpected time and / or spatial locations within the output; And / or One or more unexpected probes are identified in the output.

17. The method according to claim 11 or 12, further comprising: The image processing has been determined to have been performed accurately in the following circumstances: Able to identify one or more expected probes in the output; The ability to identify one or more expected probes at the expected time and / or spatial location in the output; And / or One or more unexpected probes could not be identified in the output.

18. A non-transitory computer-readable storage medium storing computer software code, which, when executed on a processor, performs a method of operating an image processor, the method comprising: The image is modified to produce an output image by performing corresponding image processing operations in each of one or more image processing stages in a set of image processing stages. as well as The following methods are used to verify one or more image processing operations performed by one or more image processing stages from the set of one or more image processing stages: Before modifying one or more images by performing one or more image processing operations by one or more processing stages in the set of one or more image processing stages, one or more probes are inserted into the one or more images such that the one or more probes are modified together with the one or more images by the one or more image processing stages in the set of one or more image processing stages; as well as An attempt is made to identify one or more expected probes in the output of the one or more processing stages, wherein each expected probe is a probe that is expected to appear in the output based on the one or more image processing operations performed by the one or more image processing stages.

Images