Systems and methods for error detection and correction

Abstract

The present disclosure relates to systems and methods for error detection and correction. A method for detecting and correcting transient errors includes the steps of acquiring (110) image data from a camera system and processing (120) the image data using a first image signal processor and a second image signal processor to produce first output data and second output data. At least one statistical model is generated (130) based on at least the first output data and the second output data, and the statistical model is used to identify (140) whether an error exists in the first output data. A correction value is generated (150) for a portion of the image data, wherein the correction value is an expected value based on the statistical model and is used to generate (160) updated output data. The updated output data is then output (170) to an output device.

Description

Technical Field

[0001] The present disclosure relates to detecting and correcting errors in data captured by one or more sensors of a safety critical system, particularly errors in image data captured by a camera. Background Art

[0002] Safety-critical systems (e.g., those in autonomous vehicles) typically capture data about the environment (e.g., images of the environment) to decide what actions to take. If the system detects an error in one or more portions of the data (e.g., pixels of image data), the data is typically discarded because it is classified as unreliable. If the system is required to analyze the movement of an object over time and therefore track the movement of the object in multiple portions of the data, this can lead to the introduction of further errors due to the inability to analyze the missing portions. Summary of the Invention

[0003] According to a first aspect of the present disclosure, a method for detecting and correcting transient errors is provided, the method comprising the following steps: acquiring image data from a camera system; processing the image data using a first image signal processor and a second image signal processor to generate first output data and second output data; generating at least one statistical model based on at least the first output data and the second output data; identifying whether there is an error in the first output data based on the statistical model; generating a correction value for a portion of the image data, wherein the correction value is an expected value based on the statistical model; using the correction value to generate updated output data, and outputting the updated output data to an output device.

[0004] According to a second aspect of the present disclosure, a safety-critical system is provided, comprising: a camera system for acquiring image data related to a system environment; at least two image signal processors, each for processing the image data received from the camera system and generating a first output and a second output; an error detection and correction module for detecting one or more errors in the first output and generating updated output data, wherein the updated output data includes corrected errors, which are generated based on the first output and the second output using at least a statistical model; and an output device for receiving the updated output data from the error detection and correction module.

[0005] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, comprising a set of computer-readable instructions stored thereon, which, when executed by at least one processor, enable the processor to detect and correct errors in a safety-critical system, the instructions comprising: acquiring image data from a camera system; processing the image data using a first image signal processor and a second image signal processor to generate first output data and second output data; generating at least one statistical model based on at least the first output data and the second output data; determining whether there is an error in the first output data based on the statistical model; generating a correction value for a portion of the image data, wherein the correction value is an expected value based on the statistical model; using the correction value to generate updated output data; and outputting the updated output data to an output device.

[0006] According to a fourth aspect of the present disclosure, there is provided an autonomous vehicle comprising: at least one camera system; and a processor configured to execute a method for detecting and correcting transient errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Further features and advantages of the embodiments will become apparent from the following description of preferred embodiments of the invention, which is given by way of example only and refers to the accompanying drawings in which like reference numerals are used to indicate like features.

[0008] Figure 1 is a flowchart illustrating a method according to an example;

[0009] Figure 2 schematically illustrates a system according to an example; and

[0010] Figure 3 Based on the example Figure 1 Schematic representation of the functionality of the method. DETAILED DESCRIPTION

[0011] The details of the systems and methods according to the examples will become apparent from the description below with reference to the figures. In this description, many specific details of certain examples are set forth for illustrative purposes. References in this specification to "examples" or similar language mean that the features, structures, or characteristics described in conjunction with the example are included in at least that one example, but not necessarily in other examples. It should also be noted that in order to facilitate the explanation and understanding of the concepts contained in the following examples, certain features are omitted and / or simplified as necessary in the schematic descriptions of certain examples.

[0012] In safety-critical systems, transient errors can affect the data generated and stored by system components. Such transient errors may be caused by flaws in system design or construction. Alternatively, in memory devices and processors, chip-level transient errors may occur when particles strike the component's silicon die. Chip-level transient errors result in changes to the data (e.g., electrons in the memory circuitry) rather than to the physical die itself. Particles that can cause such interactions include, but are not limited to, alpha particles produced by envelope decay, cosmic rays that produce high-energy neutrons and protons, and thermal neutrons (neutrons that lose kinetic energy before reaching thermal equilibrium with their surroundings). For example, if a particle with certain properties strikes a flip-flop in a system component, it can cause the flip-flop to change state to a different value. While this interaction is small and won't damage the physical structure of the silicon die, it can corrupt the data stored on it, affecting ongoing processing by the processor. Similarly, errors can occur if particles interact with a data bus, introducing noise that can be interpreted as data bits that lead to further errors.

[0013] In the real world, transient errors are common, and transient errors caused by particle impacts in memory are usually unavoidable, but may affect the analysis of data received by the system. Therefore, in the real world, safety-critical systems need to be able to detect and potentially correct such transient errors. An example of a system for detecting these errors is a redundant system that synchronizes operations. Some systems may be implemented as a system on a chip, so multiple components of the system may be susceptible to transient errors. Similarly, in image signal processors and other processors (e.g., graphics processing units), a large portion of the chip is dedicated to algorithmic processing of pixel data. It is necessary to ensure that these transient errors are detected and corrected throughout the processing pipeline, as they may lead to more serious errors in the data.

[0014] The determination of errors and their significance is particularly challenging for image processing applications. For example, in a camera system for an autonomous vehicle, it is necessary not only to detect errors but also to determine the best way to respond to them. Errors can be detected by comparing the output with the output of a redundant system. If the comparison indicates an error, the system can discard the data containing the error because the correct value in the data is unknown.

[0015] However, data with single bit errors, or even multiple errors (e.g., errors in a few pixels of an image), may still be useful to autonomous vehicle systems. These systems are often able to utilize inputs with minor imperfections. For example, computer vision systems may already have image correction algorithms and other methods that can be used to correct for non-optimal input image data with a certain tolerance. Most computer vision algorithms are able to analyze non-optimal image data caused, for example, by lens effects and other obstructions (e.g., dirt on the lens). Some computer vision algorithms are also able to detect partially occluded objects and / or people.

[0016] As mentioned above, safety-critical systems typically discard data when they determine that an error exists (e.g., by comparing the data with data from redundant systems that are operating in sync). This is not ideal, as many systems (especially those used in autonomous vehicles) require a continuous stream of data to make decisions, such as detecting and tracking obstacles. If an error is detected, the system is reset and / or corrupted data is discarded, interrupting the continuity of the data and impacting the vision system's ability to track moving objects. Furthermore, when the system is reset, this will result in a period of time where the system has no information or history of object movement, and the system will lose the ability to track moving objects frame by frame.

[0017] Figure 1 is a flowchart illustrating a method according to an example. At item 110, data is acquired from one or more sensors, for example, image data may be acquired from a camera. One or more sensors may be associated with a device such as an autonomous vehicle. These sensors are capable of acquiring information about the environment from many different wavelengths. For example, an image sensor may generally include an array of sensor pixels, which may be any suitable light sensor for capturing an image. Typical sensor pixels include a photosensitive element (e.g., a photodiode capable of converting incident light into an electrical signal or data). For example, the sensor pixels may be a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). Exposure generally sets the time period over which each sensor pixel of the image sensor captures light and / or the gain applied by an amplifier to the electrical signal generated by each sensor pixel. Therefore, the sensitivity of the image sensor to incident light is generally determined by the exposure. The image sensor may be configured to capture image data using a single exposure. For example, all sensor pixels of the image sensor may capture light during the same time period or time period of the same length as each other. However, in other examples, the image sensor used in the examples described herein may be a multi-exposure image sensor, which is configured to capture image data using sensor pixels having at least two different exposures.

[0018] Thus, the image data may represent at least one characteristic of the light captured by each sensor pixel, which characteristic may be proportional to the number of photons captured by each sensor pixel. The intensity may represent the brightness of the captured light, which is, for example, a measure of light intensity per unit area, rather than an absolute intensity. In other examples, the image data may represent the luminance of the captured light, which may be considered to correspond to a brightness perception, which may or may not be proportional to luminance. In general, the image data may represent any photometric quality or characteristic that can be used to represent the visual appearance of an image represented by the image data. The image data may be in any suitable format, such as a raw image format. The image data may also be streamed from the image sensor, with or without being saved to a frame buffer, and without the need for saving the raw image data to a file. However, in this case, the image data obtained after processing the raw image data may be saved to a file.

[0019] In other examples, the sensor may be an infrared camera, or a camera capable of acquiring environmental information in the visible spectrum. In some examples, the device may include other sensors (e.g., sonar or ultrasonic radiation) for receiving radio waves emitted from an output device associated with the device, rather than a camera.

[0020] Once the data is acquired, the method proceeds to item 120 where the data is processed by two or more processors configured to operate in a synchronized manner. Depending on the level of redundancy desired, there is a primary processor and one or more redundant backup processors. The outputs of the processors can be used to determine the actions that the autonomous vehicle should take. The processors do not necessarily operate on all of the data captured by the sensors, but can operate on only a portion of the data. For example, if the system determines that it needs to track the movement of an object or that only a portion of the data is needed to make a decision, the processing can operate on a portion of the image.

[0021] A statistical model is then generated at item 130. The statistical model is capable of identifying errors and generating desired correction values ​​based on the received data and the outputs of the primary and redundant processors. In addition, the statistical model can also be based on the statistical likelihood of a real-world image as a reference, from which actual correction values ​​can be calculated, and then the erroneous data can be replaced with data that matches the statistical model. The statistical model can be calculated by comparing the variance of local pixel values ​​with the expected variance given a specific noise model and the current sensor gain. For example, a flipped most significant bit is more likely to result in a high variance. The noise model can be modeled by a combination of a Poisson distribution and a normal distribution. For example, when the data is image data, the error may be in a pixel of the image data. A statistical model based on a real-world reference image can be used to generate pixel replacement values. The statistical model can also be based on the outputs of other processors operating on the same input. The statistical model can also be based on parts of the output (e.g., individual pixels in the image data) and adjacent parts, and the quality of the data can be determined based on a comparison between these parts, for example by extracting a quality or plausibility metric for the data. One or more machine learning algorithms may be used to generate a statistical model that takes into account the signal processing algorithm used, for example, the kernel size used when training the machine learning algorithm, and statistics based on the data content and previously captured data (e.g., previous image frames). It should be understood that a similar approach may be used regardless of the type of sensor device and / or data output generated, as such a statistical model may also be generated for data received by an infrared camera, an ultrasonic device, or any other sensor type suitable for use in a safety-critical system.

[0022] Once the statistical model is generated, the method proceeds to item 140 where the output of the main processor is analyzed using the statistical model to identify whether there are differences between the expected value based on the statistical model and the output. If no differences are identified, it can be assumed that the processing performed by the main processor is the same as the expected value and there are no errors. In this way, the method proceeds to item 170 and the output of the main processor is provided to further methods in the safety-critical system without adverse effects. Conversely, if one or more differences are identified, there is an error in at least one of the outputs and the severity of the error can be determined. The severity of the error can be based on the characteristics of the error in the data, for example, whether the error occurs in the most significant bit or the least significant bit of the data, whether there are multiple errors in the data, and the location of the error in the data. If more than one error is detected, it must also be ensured that the overall output is not corrupted.

[0023] If it is determined that an error exists in the output of the main processor, the method proceeds to item 150, where a correction value is generated. The correction value can be based on a statistical model and the entire output of the main processor, or alternatively, can be based on the statistical model and a portion of the output of the main processor (e.g., an individual pixel in the image data). The statistical model used to generate the correction value can be based on a variety of methods, including but not limited to machine learning algorithms trained on real-world data and other processing algorithms. The value used to replace the detected error can be a corresponding value from the output of another processor. Alternatively, the value used to replace the error can be interpolated from surrounding values, such as by bilinear interpolation or anisotropic interpolation. Thus, by using a statistical model, an optimal estimate of the error portion can be generated, thereby maximizing the amount of data that can be used by the system and reducing the need to discard data containing smaller errors. For example, when the data is image data and an error exists in the pixel value of one of the pixels of the image data, if this is the case with the surrounding pixels of the image data, the statistical model generated for the error portion and the surrounding portion can be used to generate a corrected pixel value for each of the outputs, and the most plausible value can be selected based on each of the correction values. The most plausible values ​​may be based on a real-world image used as a reference against which actual erroneous pixels are tested, thereby replacing the erroneous portions with plausible data that matches the desired statistical model.

[0024] Once the correction value is generated, the method proceeds to item 160 where a most plausible output is generated. The most plausible output is generated by replacing the erroneous data in the primary processor output with the correction value. In some examples, if the plausible data is substantially similar to one of the redundant processor outputs, the entire data may be replaced with the output of that processor. The most plausible output is then output to an output device at item 170.

[0025] Figure 2 A system 200 according to an example is schematically shown. The system 200 includes one or more sensing devices 210 for receiving data about the environment in which the system exists. For example, the sensing device 210 can be a camera capable of capturing visible light information about the environment, or optionally infrared or other wavelengths of light. The output of the sensing device 210 is processed by at least two processors 220a, 220b (e.g., image signal processors). One of the processors 220a is a main processor, while the other processors 220b are redundant processors that are configured to process the same input in synchronization with the main processor 220a. Therefore, each of the processors 220a, 220b performs the same processing and applies the same algorithm to the data. It should be understood that other types of processors can be used when the sensing device 210 obtains other information about the environment, such as from an ultrasonic sensor.

[0026] The outputs of processors 220a and 220b are then passed to the error detection and correction device 230. The error detection and correction device 230 may include memory and at least one processor (e.g., a machine learning processor for analyzing and generating output). The memory is capable of temporarily storing input data received from one or more sensing devices, as well as output received from the primary processor 220a and the redundant processor 220b. The storage device may be associated with the same device that houses the camera or other sensing device, for example, the storage device may be associated with the autonomous vehicle. The storage device may be external to the device (e.g., a remote server) or form part of the input device (e.g., the storage device of the camera system itself). Data may be stored in the device's storage device, which may be random access memory (RAM) (e.g., DDR-SDRAM (Double Data Rate Synchronous Dynamic Random Access Memory)) or other volatile memory. In other examples, the memory may be non-volatile memory or include non-volatile memory (e.g., read-only memory (ROM)) or a solid-state drive (SSD) (e.g., flash memory). The storage device in the examples may include other storage devices, such as magnetic media, optical media or tape media, compact discs (CDs), digital versatile discs (DVDs), or other data storage media. The storage device may be removable or non-removable from the device. In some examples, the data stored in the storage device represents at least a portion of an image and may be captured by an image sensor external to the device.

[0027] At least one component of the error detection and correction device 230 includes a statistical modeling module 232 for calculating various statistical models based on the outputs from the processors 220a and 220b. The statistical modeling module 232 may include a machine learning algorithm that analyzes each of the outputs and generates various statistical models to determine expected values ​​for the data. Such statistical models may include comparing the variance of local pixel values ​​to the expected variance given a particular noise model and the current sensor gain.

[0028] Once the statistical models are calculated, they are used to determine whether there are errors in the output of the main processor 220a in the error determination module 234. The error determination module 234 compares the data to the statistical model and determines whether the value is the expected value based on the model. For example, if the data is image data (e.g., image data from a camera), the error determination module 234 checks whether the pixel values ​​correspond to the expected values ​​based on the statistical model. If the pixel values ​​correspond to the expected values, the output of the main processor can be provided to the output device 240. Optionally, when the pixel values ​​do not correspond to the expected values ​​provided by the statistical model, a corrected output may be generated.

[0029] In some examples, when an error is detected, the output generation module 236 can compare the output of each of the processors 220a, 220b with a statistical model to help determine which of the outputs is most suitable for output, because the selected output can be a higher quality output. In other examples, the statistical model can be used to generate an expected value for the error. For example, when the data is image data, if it is determined that the data includes an erroneous pixel, the statistical model can be used to generate the expected pixel value. In further examples, it should be understood that a combination of these methods can be used. This may be appropriate when there are multiple errors (e.g., multiple erroneous pixels) in the data.

[0030] Output generation module 236 outputs the corrected data to output device 240. Output device 240 may be a main processor of the autonomous vehicle, where the data is used to make decisions about actions the vehicle should take.

[0031] Although Figure 2 The example shown in FIG2 shows the error detection and correction module 230 at the end of the processing pipeline, but it should be understood that the error detection and correction module 230 can be located between the operations performed by the processor 220 a. In such an example, the output of the error detection and correction module 230 is passed back to the processor for further processing before being passed to the output device 240.

[0032] Figure 3 Based on the example Figure 1 Schematic representation of the functionality of the method, in this example, the data is image data received by a camera. The image data captured by the camera is provided to multiple image signal processors for processing, thereby generating multiple outputs 310a, 310b. During the processing operation, transient errors corrupt the output of flip-flop 315f in the image signal processor due to the corruption of pixel values ​​in the image data. When the image data is further processed by the image signal processor and other signal processing algorithms (e.g., noise reduction algorithms) are applied to the output, the errors are blurred between adjacent pixels, so that the impact of the errors is reduced but diffused.

[0033] The output data is then passed to the Figure 2An error detection and correction module is described, wherein a statistical model 320 is generated based on output data 310a, 310b of a primary image signal processor (and other image signal processors arranged to process image data in a synchronous manner). The output 310b of the redundant image signal processor is less likely to be affected by the same transient error that affected the output 310a of the primary image signal processor. The error detection and correction module is able to determine whether an error exists and the severity of the error. If the statistical model 320 indicates that only a minor error exists in the output data, such as in the least significant bit of pixel value 330f at item 330, the error detection and correction module may still determine that the data is usable by the output device. In some examples, even if the error is determined to be non-significant, the error detection and correction module may determine that one of the outputs 310b of one of the redundant image signal processors is a higher quality output and may provide that output to the output device instead. For example, the statistical model may indicate that the output should generally correspond to the output 310b of one of the redundant image signal processors and, therefore, provide that output 310b to the output device.

[0034] Conversely, if a transient error occurs at the end of the processing pipeline, or if the erroneous pixel value is determined to be a severe error because the error is unlikely to propagate among surrounding pixels, as indicated by item 330, the error detection and correction module may determine that the error is severe enough to warrant generating a correction value 340. If it is determined that calculation of correction value 340 is necessary, the calculation may be performed based on an expected value (based on statistical model 320) and other data 330 associated with primary output 310a. Primary output 310a and correction value 340 may then be combined to produce a corrected output 350, which is delivered to an output device. Alternatively, if one of the outputs 310b of one of the redundant image signal processors sufficiently matches the expected output indicated by the statistical model, the output data of that redundant processor may be used as the corrected output and delivered to the output device.

[0035] It should be understood that the replacement and / or calculation of corrected pixels can occur at any point in the processing pipeline, such as before the noise reduction algorithm described above. This ensures that errors do not accumulate or spread spatially because the error detection and correction module enforces equality between the multiple outputs of the processor at multiple points throughout the processing pipeline.

[0036] The above embodiments are to be understood as illustrative examples of the present invention. Further embodiments of the present invention are contemplated. It should be understood that any feature described with respect to any one embodiment may be used alone or in combination with other features described, or in combination with one or more features of any other embodiment, or in combination with any combination of any other embodiments. Furthermore, equivalents and modifications not described above may also be employed without departing from the scope of the present invention as defined in the appended claims.

Claims

1. A method for detecting and correcting transient errors, the method comprising the following steps: acquiring image data having at least two different exposures from a camera system; processing the image data using a first image signal processor and a second image signal processor to generate first output data representing a first of the at least two differently exposed image data and second output data representing a second of the at least two differently exposed image data; generating, using at least one machine learning algorithm, at least one statistical model based on at least the first output data, the second output data, previous image data acquired by the camera system, and at least one real-world reference image; identifying whether there is an error in the first output data based on the statistical model; generating a correction value for a portion of the image data by applying the at least one statistical model to the first output data to generate an expected value for the error; generating updated output data using the correction value; as well as The updated output data is output to an output device.

2. The method according to claim 1, wherein The step of processing the image data includes processing a portion of the image data acquired by the camera system.

3. The method according to claim 1, wherein The steps to generate updated output data include: combining the correction value with the first output data; or A portion of the second output data corresponding to the error in the first output data is compared with the correction value, and if the portion substantially corresponds to the correction value, the updated output data is set as the second output data.

4. A safety-critical system comprising: a camera system for acquiring image data having at least two different exposures associated with a system environment; at least two image signal processors, each for processing the image data received from the camera system and generating first output data representing a first of the at least two differently exposed image data and second output data representing a second of the at least two differently exposed image data; an error detection and correction module, configured to detect one or more errors in the first output data, the error detection and correction module comprising: a statistical model unit for generating at least one statistical model based on at least the first output data, the second output data, previous image data acquired by the camera system, and at least one real-world reference image using at least one machine learning algorithm; an error determining unit, configured to determine whether there is an error in the first output data based on the at least one statistical model; an output generating unit for generating a correction value for a portion of the image data by applying the at least one statistical model to the first output data to generate an expected value for the error, and generating updated output data, wherein the updated output data includes the corrected error generated using the correction value; and An output device is configured to receive the updated output data from the error detection and correction module.

5. The safety-critical system according to claim 4, wherein: The error detection and correction module also includes: a combining unit, configured to combine the expected value with the first output data; and A comparing unit is configured to compare the expected value with a portion of the second output corresponding to the error in the first output data, and output the second output data if the portion of the second output data is substantially similar to the expected value.

6. A non-transitory computer-readable storage medium comprising a set of computer-readable instructions stored thereon, the instructions, when executed by at least one processor, causing the processor to detect and correct errors in a safety-critical system, the instructions comprising: acquiring image data having at least two different exposures from a camera system; processing the image data using a first image signal processor and a second image signal processor to generate first output data representing a first of the at least two differently exposed image data and second output data representing a second of the at least two differently exposed image data; generating, using at least one machine learning algorithm, at least one statistical model based on at least the first output data, the second output data, previous image data acquired by the camera system, and at least one real-world reference image; determining whether there is an error in the first output data based on the statistical model; generating a correction value for a portion of the image data by applying the at least one statistical model to the first output data to generate an expected value for the error; generating updated output data using the correction value; as well as The updated output data is output to an output device.

7. An autonomous vehicle comprising: at least one camera system; as well as A processor configured to execute the method according to claim 1.

Images