Safety monitor for image misclassification

Abstract

A system, apparatus, and method for implementing a safety monitor framework for a safety-critical inference application are disclosed. The system includes a safety-critical inference application, a safety monitor, and an inference accelerator engine. The safety monitor receives an input image, test data, and neural network specifications from the safety-critical inference application. The safety monitor generates a modified image by adding additional objects outside the input image. The safety monitor provides the modified image and the neural network specifications to an inference accelerator engine that processes the modified image and provides output to the safety monitor. The safety monitor determines the probability of incorrect processing of the original input image by comparing the output for the additional objects with known good results. The safety monitor complements the full fault range of the inference accelerator engine and covers faults observable only at the network level.

Description

Technology Field

[0001] A new field of technology is machine learning, and neural networks are a type of machine learning model. Neural networks have demonstrated excellent performance in tasks such as handwritten digit classification and face detection. Furthermore, neural networks have shown the potential to perform well in other more challenging visual classification tasks. Other applications of neural networks include speech recognition, language modeling, sentiment analysis, and text prediction.

[0002] In the typical deployment of machine learning algorithms, software applications provide neural networks to an inference accelerator hardware engine. When an inference accelerator operates in a safety-critical environment, it is desirable to monitor the accelerator to identify abnormal behavior. A common implementation for monitoring the inference accelerator involves embedding monitoring logic into the inference accelerator processing hardware subblocks. For example, a machine inspection architecture is a mechanism where monitoring logic within the processing hardware checks for abnormal behavior. However, while such inspections provide monitoring at a low-level hardware level, they may overlook errors observable only at the output of the inference accelerator. This approach also fails to detect random intermittent faults at the neural network level. For instance, the inference accelerator may experience misclassification errors, such as failure to detect objects of interest. Brief explanation of the drawing

[0003] The advantages of the methods and mechanisms described in this specification can be better understood by referring to the following description together with the accompanying drawings. Figure 1 is a block diagram of one implementation example of a computing system. FIG. 2 is a block diagram of one implementation example of a safety-critical reasoning application running in a safety-critical system. Figure 3 is a block diagram of one implementation example of an inference defect detection framework. FIG. 4 is an example of an image transmitted from a safety-critical application to a safety monitor framework according to one embodiment. FIG. 5 is a modified image generated by a safety monitor framework from an original image according to one embodiment. Figure 6 is a generalized flowchart illustrating one implementation example of a method for operating a safety monitor framework for an inference accelerator. Figure 7 is a generalized flowchart illustrating one implementation example of a method for implementing a safety monitor framework. FIG. 8 is a generalized flowchart illustrating one implementation example of a method for generating a confidence indicator for use by a safety-critical application. FIG. 9 is a generalized flowchart illustrating one implementation example of a method for determining an object to be used to modify an image. FIG. 10 is a generalized flowchart illustrating one implementation example of a method for a safety monitor framework that changes the operating mode. Specific details for implementing the invention

[0004] In the following description, numerous specific details are described to provide an overall understanding of the methods and mechanisms presented herein. However, those skilled in the art should recognize that various embodiments may be implemented without these specific details. In some examples, known structures, components, signals, computer program instructions, and techniques are not shown in detail to avoid obscuring the approach described herein. For the sake of simplicity and clarity of the examples, it will be understood that the elements depicted in the drawings are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated compared to others.

[0005] Systems, apparatus, and methods for implementing a safety monitor framework for a safety-critical neural network application are disclosed herein. In one embodiment, the system comprises a safety-critical neural network application, a safety monitor, and an inference accelerator engine. The safety monitor receives an input image, test data (e.g., test vectors), and neural network specifications (e.g., layers and weights) from the safety-critical neural network application. In one embodiment, the test data includes a list of known good objects / images and images having corresponding classifications of such objects / images. In one embodiment, the safety monitor modifies the input image to add additional objects outside the boundaries of the input image. In this embodiment, the modified image is larger than the input image to accommodate additional objects outside the boundaries of the original image. In one embodiment, the configuration of the modified image (i.e., where the extra space and additional objects are inserted) is stored in a data structure (i.e., metadata) that is passed through the system flow along with the modified image.

[0006] In one embodiment, the additional objects include one or more duplicate objects identical to those found in the previous input image. In another embodiment, the additional objects include one or more objects provided in the test vector. The safety monitor provides the modified image to an inference accelerator. The inference accelerator processes the modified image and provides the output back to the safety monitor. Based on the output generated by the inference accelerator, the safety monitor determines the misclassification probability of the original input image. In one embodiment, the safety monitor determines the misclassification probability of the original input image by comparing known good results with the output associated with the modification. The safety monitor provides an indicator of misclassification probability to safety-critical applications. Through this approach, the safety monitor compensates for the full error range and protects against errors observable only in the neural network output.

[0007] Now, referring to FIG. 1, a block diagram of one embodiment of a computing system (100) is illustrated. In one embodiment, the computing system (100) includes at least an inference accelerator engine (105), processor(s) (110A-B), an input / output (I / O) interface (120), a bus (125), and a memory subsystem (130). In other embodiments, the computing system (100) may include other components and / or the computing system (100) may be arranged differently. In one embodiment, the inference accelerator engine (105) is implemented on the processor(s) (110B). The inference accelerator engine (105) represents any combination of software, firmware, and / or hardware for implementing various machine learning algorithms or machine learning models on the processor(s) (110B).

[0008] In one embodiment, the inference accelerator engine (105) implements one or more layers of a convolutional neural network. For example, in this embodiment, the inference accelerator engine (105) implements one or more convolutional layers and / or one or more fully connected layers. In another embodiment, the inference accelerator engine (105) implements one or more layers of a recurrent neural network. Generally, an "inference engine" or "inference accelerator engine" is defined as hardware and / or software that receives image data and generates one or more label probabilities for the image data. In some cases, the "inference engine" or "inference accelerator engine" is referred to as a "classification engine" or "classifier."

[0009] The inference accelerator engine (105) is utilized in various different applications that vary depending on the embodiment. For example, in one embodiment, the inference accelerator engine (105) analyzes an image or video frame to generate one or more label probabilities for the frame. For example, potential use cases include at least eye tracking, object recognition, point cloud estimation, ray tracing, light field modeling, depth tracking, etc. In the case of an eye tracking use case, the probabilities generated by the inference accelerator engine (105) are based on learned patterns, dwell times, transition angles, blinking, etc. In another embodiment, the inference accelerator engine (105) is trained and customized for different types of use cases.

[0010] The inference accelerator engine (105) may be used by various different safety-critical applications depending on the embodiment. For example, in one embodiment, the inference accelerator engine (105) is used in automotive applications. For example, the inference accelerator engine (105) controls one or more functions of an autonomous vehicle (i.e., an autonomous vehicle), a driver assistance vehicle, or an advanced driver assistance system. In another embodiment, the inference accelerator engine (105) is trained and customized for different types of use cases. Depending on the embodiment, the inference accelerator engine (105) generates probabilities of classification results for various objects detected in an input image or video frame.

[0011] Processor(s) (110A-B) represent any number and type of processing units (e.g., Central Processing Unit (CPU), Graphics Processing Unit (GPU), Digital Signal Processor (DSP), Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC)). In one embodiment, part of the processing associated with the inference accelerator engine (105) is performed by the processor(s) (110B). Additionally, the inference accelerator engine (105) is implemented using these types of processing units and / or other types of processing elements. The memory subsystem (130) includes any number and type of memory devices. For example, memory types of the memory subsystem (130) include High-Bandwidth Memory (HBM), Non-Volatile Memory (NVM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND Flash Memory, NOR Flash Memory, Ferroelectric Random Access Memory (FeRAM), etc. The memory subsystem (130) is accessible by the inference accelerator engine (105) and processor(s) (110A-B).The I / O interface (120) represents any number and type of I / O interface (e.g., Peripheral Component Interconnect (PCI) bus, PCI-Extended (PCI-X), PCI Express (PCIE) bus, Gigabit Ethernet (GBE) bus, Universal Serial Bus (USB)). Various types of peripheral devices may be coupled to the I / O interface (120). These peripheral devices include, but are not limited to, displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, etc.

[0012] In various embodiments, the computing system (100) is a computer, laptop, mobile device, game console, server, streaming device, wearable device, or any other various type of computing system or device. In some embodiments, all or one or more parts of the computing system (100) are integrated within a robotic system, an autonomous vehicle, an autonomous drone, a surgical tool, or other type of mechanical device or system. Note that the number of components of the computing system (100) varies from embodiment to embodiment. For example, in other embodiments, there may be more or fewer of each component than the number shown in FIG. 1. Also note that in other embodiments, the computing system (100) includes other components not shown in FIG. 1. Additionally, in other embodiments, the computing system (100) is configured in a different way than shown in FIG. 1.

[0013] Now, referring to FIG. 2, a block diagram of one embodiment of a safety-critical reasoning application (220) running in a safety-critical system (200) is shown. In one embodiment, the safety-critical system (200) includes at least processing unit(s) (205 and 210) representing any number and type of processing units. It should also be noted that the safety-critical system (200) may include any number of other components not shown to avoid obscuring the drawing. In one embodiment, the processing unit(s) (205) include one or more central processing units (CPUs). In another embodiment, the processing unit(s) (205) may include other types of processing units. In one embodiment, the processing unit(s) (210) include one or more graphics processing units (GPUs). In another embodiment, the processing unit(s) (210) may include other types of processing units (e.g., digital signal processor (DSP), field programmable gate array (FPGA), application-specific integrated circuit (ASIC)).

[0014] In one embodiment, the safety-critical reasoning application (220) is executed on the processing unit(s) (205). The safety-critical reasoning application (220) represents any type of software application executed in a dangerous environment where safety is critical. For example, in one embodiment, the safety-critical reasoning application (220) controls a self-driving or driver-assisted vehicle or other vehicle. In another embodiment, the safety-critical reasoning application (220) operates within a robot, as an autopilot control mechanism for an airplane, or as part of another system in various challenging and high-risk environments.

[0015] In one embodiment, a first part of the safety monitor framework (230) (i.e., safety monitor framework (230A)) is executed on the processing unit(s) (205) and a second part of the safety monitor framework (230) (i.e., safety monitor framework (230B)) is executed on the processing unit(s) (210). In another embodiment, the safety monitor framework (230) is executed entirely on the processing unit(s) (205) or entirely on the processing unit(s) (210). The inference accelerator (240) represents any combination of software and / or hardware used to implement one or more machine learning inference algorithms and / or machine learning inference models. In one embodiment, the inference accelerator (240) is implemented using dedicated hardware (e.g., FPGA, ASIC, IP core). In another embodiment, the inference accelerator (240) includes software instructions designed to be executed on the processing unit(s) (210). In another embodiment, the inference accelerator (240) may be any suitable combination of software and / or hardware. In one embodiment, the inference accelerator (240) operates according to the topology and weights / biases provided by the safety-critical inference application (220).

[0016] The safety-critical inference application (220) provides an image to be processed by the inference accelerator (240). In one embodiment, the safety monitor framework (230) receives inputs being transmitted from the safety-critical inference application (220) to the inference accelerator (240), and the safety monitor framework (230) modifies one or more of these inputs and then transmits the modified inputs to the safety-critical inference application (220). After the inference accelerator (240) processes the modified inputs, the inference accelerator (240) transmits the processing results to the safety-critical inference application (220) through the safety monitor framework (230). The safety monitor framework (230) analyzes the results to determine whether the inference accelerator (240) is malfunctioning. In one embodiment, the safety monitor framework (230) generates a reliability indicator that the safety monitor framework (230) specifies that the inference accelerator (240) has correctly processed the inputs generated by the safety-critical inference application (220). The safety monitor framework (230) transmits a modified version of the reliability indicator and processing result to the safety essential inference application (220). In one embodiment, if the reliability indicator is below a threshold, the safety essential inference application (220) takes one or more corrective actions (e.g., system shutdown, system reboot, retry the same image, generate a warning signal to the user, reduce vehicle speed, change operating mode).

[0017] Now, referring to FIG. 3, a block diagram of one embodiment of an inference fault detection framework (300) is illustrated. In one embodiment, a computing system (310) includes a safety monitor framework (315) and inference accelerator hardware (350). In one embodiment, the safety monitor framework (315) includes a test generation unit (320), a scaler unit (325), a result filter (330), a test verification unit (335), an inference stack (340), and an inference driver (345). In another embodiment, the safety monitor framework (315) includes other components and / or is arranged in other appropriate ways. In one embodiment, the safety monitor framework (315) is designed to comply with an Automatic Safety Integrity Level (ASIL) risk classification scheme. In another embodiment, the safety monitor framework (315) may be designed to comply with a different risk classification scheme in a different type of environment.

[0018] In one embodiment, the safety-critical inference application (305) transmits the image, test vector, and inference network specification to the safety monitor framework (315). The test vector is received and used by the test generation unit (320) to determine which object is to be added to the original image. The test vector is also used to determine whether the result generated by the inference accelerator hardware (350) is accurate. In one embodiment, a duplicate object is added to the original image, and the duplicate object is identical to an object present in the previous image. In another embodiment, a given object that was not present in the previous image is added to the original image. For example, in one embodiment, the given object is an object identified in the test vector as a known good object in one or more test images. As used herein, "known good object" is defined as an object that is likely to be accurately identified or processed in a consistent manner by the inference accelerator hardware (350). In one embodiment, the given object to be added to the original image is selected based at least partially on what is likely to occur in the image being processed. The test generation unit (320) generates a modified image by adding one or more extra given object(s) to the original image. The inference stack (340) transmits the modified image and the inference network specification to the inference accelerator hardware (350). In one embodiment, the modified image and the inference network specification are transmitted to the inference accelerator hardware (350) via an inference driver (345).

[0019] After the inference accelerator hardware (350) processes the modified image according to the provided inference network specifications, the inference accelerator hardware (350) returns the processing result to the inference stack (340). The result filter (330) analyzes the result and filters the result to remove any extra objects that were added to the original image. The result filter (330) delivers the object and classification result to the application (305). The object and classification result may contain all types of data, the type of which varies depending on the implementation. For example, in one implementation, if the application (305) is looking for the number of objects in the original image, the result will include the number and location of the objects within the original image. In another implementation, different data may be delivered to the object and classification result depending on what the application (305) is looking for in the original image.

[0020] In one embodiment, the result from the inference accelerator hardware (350) is analyzed by a test verification unit (335). The test verification unit (335) determines whether the extra object(s) added to the original image were correctly processed and / or classified by the inference accelerator hardware (350). In another embodiment, the test verification unit (335) may use other suitable techniques to verify the result generated by the inference accelerator hardware (350). If the test verification unit (335) determines that the inference accelerator hardware (350) correctly processed a known good object, the test verification unit (335) returns a pass test result indicator to the application (305). Otherwise, if the known good object was incorrectly processed, the test verification unit (335) returns a failed test result indicator to the application (305). In response to receiving a failed test result indicator, the application (305) may take one of various corrective actions (e.g., rebooting, generating an error, playing the same image).

[0021] Now, referring to FIG. 4, an example of an image (400A) transmitted from a safety-essential application to a safety monitor framework according to one embodiment is illustrated. In the example illustrated in FIG. 4, the image (400A) includes a truck (402), a person (404), a dog (406), a horse (408), and another person (410). It should be understood that this example of an image (400A) containing these objects represents only one specific embodiment. The probabilities of detection and accurate identification are also indicated at the top of the dotted box surrounding each identified object. In one embodiment, these probabilities are determined during the training phase of the target inference accelerator engine. In one embodiment, these probabilities are provided to the safety monitor framework for the target inference accelerator engine that the safety-essential application provides to the specific neural network and / or image processing target inference accelerator engine.

[0022] Now, referring to FIG. 5, a modified image (400B) generated by a safety monitor framework from an original image (400A) according to one embodiment is shown. The modified image (400B) is intended to represent the image (400A) (of FIG. 4) after being modified by a safety monitor framework (e.g., the safety monitor framework (315) of FIG. 3). As shown, the modified image (400B) includes the original image (400A) along with a known good object area (505) added to the modified image (400B) outside the boundary (right) of the original image (400A). The area of ​​the original image (400A) on the left side of the modified image (400B) remains unchanged. Thus, the portion of the original image (400A) in the modified image (400B) includes all the original objects shown in the image (400A) (of FIG. 4).

[0023] In one embodiment, the known good object area (505) includes a person (510) representing a known good object found in one of the test vector images. In another embodiment, the known good object area (505) may include a different number and / or type of objects. Depending on the embodiment, the objects displayed in the known good object area (505) may include duplicate objects that are exact replicas of objects in the original image and / or new objects that were not included as part of the original image (400A). Additionally, it should be understood that although the known good object area (505) is shown to the right of the modified image (400B), this is merely to represent one specific embodiment. In another embodiment, the known good object area (505) may be added to the top, left, and / or bottom of the original image (400A). In some embodiments, a part of the original image is used as the known good object area (505). For example, if a part of an image is considered unimportant for some reason (e.g., if it is determined that the image content of a specific area does not have a functional impact on the processing result), that specific area can be used as a known good object area.

[0024] After generating a modified image (400B), the safety monitor framework passes the modified image (400B) to an inference accelerator engine (e.g., the inference accelerator hardware (350) of FIG. 3). The inference accelerator engine processes the modified image (400B) and then passes the output based on this processing to the safety monitor framework. The safety monitor framework determines whether an object added to a known good object area (505) has been identified and / or processed according to previously provided test data. If this added object has been correctly identified and / or appropriately processed, the safety monitor framework provides a pass indicator to the safety-essential application. The safety monitor framework also filters the processing results related to the original image (400A) by excluding objects from the known good object area (505). Then, the safety monitor framework provides these filtered results to the safety-essential application.

[0025] On the other hand, if an added object is misidentified and / or mishandled, the safety monitor framework provides a failure indicator to the safety-critical application. In response to receiving the failure indicator, the safety-critical application takes one or more corrective actions. For example, in one embodiment, the safety-critical application terminates in response to receiving the failure indicator. In another embodiment, in response to receiving the failure indicator, the safety-critical application generates the same frame to be reprocessed by the inference accelerator engine. In another embodiment, the safety-critical application performs other actions in response to receiving the failure indicator.

[0026] Now, referring to FIG. 6, one embodiment of a method (600) for operating a safety monitor framework for an inference accelerator is illustrated. For the sake of discussion, the steps of this embodiment and the steps of FIGS. 7 through 9 are illustrated sequentially. However, it should be noted that in various embodiments of the described method, one or more of the described elements are performed simultaneously in a different order than illustrated or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or devices described herein are configured to implement the method (600).

[0027] The safety monitor framework receives test data and neural network specifications from a safety-essential inference application (block (605)). The test data may include training information, test vectors, and / or other metadata. In some cases, the test data is provided or received by the safety monitor framework in advance before starting the method (600). In one embodiment, the test vectors include previously identified objects and / or objects likely to be identified in the image being processed. Additionally, the safety monitor framework receives an input image from the safety-essential inference application (block (610)). In one embodiment, the input image is a real image that needs to be processed for a real-time application. Next, the safety monitor framework generates a modified image from the input image based on the test data (block (615)). In one embodiment, the modified image includes the input image having one or more extra objects added outside the boundaries of the input image. For example, in one embodiment, the safety monitor framework detects a first object within the previous input image. In this implementation example, if the probability that the first object in the test data will be identified by the inference accelerator engine is greater than or equal to a threshold, the safety monitor framework adds the first object to a space outside the original input image. In this example, the modified image contains the original input image and the first object. Note that any number of objects can be added to the area outside the original input image. The modified image is generated as a combination of the original input image and the added area.

[0028] Next, the safety monitor framework passes the modified image to the inference accelerator engine (block (620)). Next, the inference accelerator engine processes the modified image (block (625)). Note that the inference accelerator engine does not recognize that the original image has been modified. Therefore, the inference accelerator engine performs normal processing as if the image had been received directly from the safety-critical application without modification. Next, the inference accelerator engine passes the output from the processing of the modified image to the safety monitor framework (block (630)). The safety monitor framework checks the output to determine whether the inference accelerator engine is operating normally or malfunctioning (block (635)). For example, if the extra object added to the modified image is processed in the expected manner, the safety monitor framework concludes that the inference accelerator engine is operating normally. Otherwise, if the result of the inference accelerator engine processing the extra object is unexpected and / or does not match the result of the same object in the original image portion of the modified image, the safety monitor framework concludes that the inference accelerator engine is malfunctioning.

[0029] If the safety monitor framework determines that the inference accelerator engine is malfunctioning (conditional block (640), "Yes"), the safety monitor framework generates an error message and delivers it to the safety-essential inference application (block (645)). In block (645), the safety monitor framework may also provide other outputs from the inference accelerator engine to the safety-essential application. In response to receiving the error message, the safety-essential application performs one or more corrective actions (block (650)). Depending on the type of error detected and / or the current state of the safety-essential application, the safety-essential application may optionally decide to continue sending subsequent images to the inference accelerator engine for processing in some cases. Alternatively, the safety-essential application may decide to terminate in response to receiving the error message. If the safety monitor framework determines that the inference accelerator engine is functioning normally (conditional block (640), "No"), the safety monitor framework generates a non-error message and delivers it to the safety-essential application (block (655)). The safety monitor framework may also provide another output (e.g., a filtered result) from the inference accelerator engine to the safety-essential application in block (655). In response to receiving a non-error message, the safety-essential application provides another image to the safety monitor framework, and method (600) returns to block (610).

[0030] Now, referring to FIG. 7, an exemplary embodiment of a method (700) for implementing a safety monitor framework is illustrated. The safety monitor framework receives an input image and test data from a safety-essential application (block (705)). The safety monitor framework determines whether any object identified in the test data existed in the previous image (block (710)). For this discussion, it is assumed that the input image is part of a continuous sequence of images, such as a camera input stream. If one or more objects identified by the test data existed in the previous image (conditional block (715), "Yes"), the safety monitor framework modifies the input image by adding one or more duplicate copies of these identified objects outside the boundaries of the original image (block (720)). Then, the safety monitor framework passes the modified image to the inference accelerator engine (block (725)). If the object identified by the test data did not exist in the previous image (conditional block (715), "No"), the safety monitor framework passes the input image to the inference accelerator engine with processing performed in the conventional manner (block (730)). After block (730), method (700) is terminated.

[0031] After block (725), the inference accelerator engine processes the modified image and returns a first set of processing results to the safety monitor framework (block (735)). The safety monitor framework analyzes the first set of processing results and generates an error status message (block (740)). Note that the error status message indicates the error state of the inference accelerator engine or the probability thereof. In one embodiment, the error status message is a single bit indicating that the inference accelerator engine is functioning normally or is malfunctioning. The safety monitor framework also converts the first set of processing results into a second set of processing results (block (745)). In one embodiment, the safety monitor framework converts the first set of processing results into a second set of processing results by removing any result data associated with the modification of the original image from the first set of processing results. That is, the second set of processing results indicates how the processing result would look if the original image, rather than the modified image, were processed by the inference accelerator engine. Next, the safety monitor framework passes a second set of error status messages and processing results to the safety-essential application (block (750)). After block (750), the method (700) is terminated.

[0032] Now, referring to FIG. 8, one embodiment of a method (800) for generating a reliability indicator for use in a safety-critical application is illustrated. A safety monitor framework intercepts an image transmitted from a safety-critical application to an inference accelerator engine (block (805)). In one embodiment, the safety monitor framework is inserted between the safety-critical application and the inference accelerator engine. In one embodiment, neither the safety-critical application nor the inference accelerator engine is aware that a safety monitor framework is inserted between them. However, in another embodiment, the safety-critical application and / or the inference accelerator engine are aware that a safety monitor framework is located between them. After performing one or more modifications to the image, the safety monitor framework transmits the modified image to the inference accelerator engine (block (810)). In one embodiment, one or more modifications include adding one or more extra objects to a space outside the original image. In the case of video streams, one way to improve reliability for object identification is to insert one similar object and one dissimilar object into the spare space of the next video frame (assuming the object is still in the next capture frame at a close temporal distance).

[0033] After the inference accelerator engine processes the modified image, the safety monitor framework intercepts the result transmitted from the inference accelerator engine to the safety-essential application (block (815)). The safety monitor framework analyzes the result to determine the probability that the inference accelerator engine processed the image correctly (block (820)). For example, in one embodiment, the safety monitor framework determines whether the result indicates that a misclassification occurred. If the result of the image modification processing does not match the result provided with the test data, the safety monitor framework will conclude that there is a relatively high probability that the inference accelerator misprocessed the original input image. In one embodiment, a relatively high probability that the inference accelerator misprocessed the image is represented by a relatively low (i.e., close to zero) confidence indicator.

[0034] Next, the safety monitor framework generates a confidence indicator that specifies the probability that the inference accelerator engine will correctly process the image (block (825)). Additionally, the safety monitor framework filters the results related to the modifications made to the original image and modifies the results (block (830)). Then, the safety monitor framework passes the confidence indicator and the modification results back to the safety-essential application (block (835)). After block (835), the method (800) ends.

[0035] Now, referring to FIG. 9, an embodiment of a method (900) for determining an object to be used to modify an image is illustrated. A safety monitor receives test data containing known good objects and their probability of occurrence in the actual image (block (905)). In one embodiment, a safety essential application transmits the test data to the safety monitor. Additionally, the safety monitor analyzes the detected output of the previous image to track the detection frequency of various objects in the previous image (block (910)).

[0036] The safety monitor determines whether any object has both an occurrence probability greater than a first threshold and a detection frequency in a previous image greater than a second threshold (conditional block (915)). Note that the values ​​of the first threshold and the second threshold may vary depending on the implementation. If any object has both an occurrence probability greater than the first threshold and a detection frequency in a previous image greater than the second threshold (conditional block (915), "Yes"), the safety monitor adds one or more of these objects to the next image being delivered to one or more inference accelerator engines (block (920)). Otherwise, if the object does not have both an occurrence probability greater than the first threshold and a detection frequency in a previous image greater than the second threshold (conditional block (915), "No"), the safety monitor adds at least one object having an occurrence probability greater than the first threshold or a detection frequency in a previous image greater than the second threshold to the next image (block (925)). If the object does not have a probability of occurrence greater than the first threshold or a detection frequency in the previous image greater than the second threshold, the safety monitor may choose to add a different type of object to the next image or not add any object. If the object does not have both a probability of occurrence greater than the first threshold and a detection frequency in the previous image greater than the second threshold, the safety monitor may decrease the value of the first threshold and / or the value of the second threshold. After blocks (920 and 925), the method (900) ends.

[0037] Now, referring to FIG. 10, one embodiment of a method (1000) for a safety monitor framework that changes the operating mode is illustrated. A safety monitor of a safety-critical system having a safety-critical application and an inference accelerator engine is started by operating in a first mode (block (1005)). In one embodiment, while operating in the first mode, the safety monitor transmits actual image data to the inference accelerator engine for classification. When the safety monitor detects a first condition (conditional block (1010), "Yes"), the safety monitor switches to a second mode for monitoring the inference accelerator engine (block 1015). Note that the "second mode" may also be referred to herein as "safety mode". In one embodiment, while operating in the second mode, the safety monitor generates experimental image data and transmits the experimental image data to the inference accelerator engine for classification. Note that the "experimental image data" may also be referred to herein as "known good object image data". In this embodiment, after the inference accelerator engine generates a classification result for the experimental image data, the safety monitor generates a reliability indicator based on the analysis of the classification result, and the reliability indicator indicates the probability that the result is accurate.

[0038] In one embodiment, the first condition is receiving a signal from a safety-critical application that enhanced monitoring should be performed. For example, a particularly critical section of code within the application may be detected, and the application may determine that a close examination of the inference accelerator engine is required during the period in which this section of code is executed. In another embodiment, the first condition is detecting abnormal behavior related to the inference accelerator engine. In a further embodiment, the first condition is a timer expiration. In another embodiment, the first condition may be a different type of condition.

[0039] In one embodiment, when operating in the first mode, the safety monitor modifies one of all N images, where N is a positive integer greater than 1. In this embodiment, when operating in the second mode, the safety monitor modifies one of all M images, where M is a positive integer less than N. If the safety monitor does not detect the first condition (conditional block (1010), "No"), the safety monitor remains in the first mode (block (1020)). After block (1020), the method (1000) returns to the conditional block (1010).

[0040] After block (1015), if the safety monitor detects a second condition (conditional block (1020), "Yes"), the safety monitor returns to the first mode (block (1025)). In one embodiment, the second condition is receiving a signal from a safety-essential application to re-enter the first mode. In another embodiment, the second condition is detecting a return to normal operation by the inference accelerator engine. In an additional embodiment, the second condition is a value of a timer within a specific range. In another embodiment, the second condition may be a different type of condition. After block (1025), the method (1000) returns to the conditional block (1010). Otherwise, if the safety monitor does not detect the second condition (conditional block (1020), "No"), the safety monitor remains in the second mode (block (1030)). After block (1030), the method (1000) returns to the conditional block (1020). In other embodiments, it should be understood that the safety monitor operates in two or more different types of modes. In these embodiments, the inference accelerator engine may be adjusted in finer increments depending on the application and / or the state of the inference accelerator engine. For example, the safety monitor may change from modification every 10th frame, every 9th frame, every 8th frame, every 7th frame, etc. Alternatively, the safety monitor may switch between experimental image data generation every different frame, every 3rd frame, every 4th frame, etc.

[0041] In various embodiments, program instructions of a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions executable by a general-purpose or special-purpose processor are considered. In various embodiments, these program instructions may be expressed in a high-level programming language. In other embodiments, program instructions may be compiled from a high-level programming language into a binary, intermediate, or other form. Alternatively, program instructions describing the behavior or design of hardware may be written. These program instructions may be expressed in a high-level programming language such as C. Alternatively, a Hardware Design Language (HDL) such as Verilog may be used. In various embodiments, program instructions are stored in any of various non-transient computer-readable storage media. The storage medium is accessible by the computing system while it is being used to provide program instructions to the computing system for program execution. Generally speaking, such a computing system includes one or more memories and one or more processors configured to execute program instructions.

[0042] It should be emphasized that the embodiments described above are merely non-limiting examples. Once the above disclosure is fully understood, numerous variations and modifications will be apparent to those skilled in the art. The following claims should be interpreted to include all such variations and modifications.

Claims

Claim 1 A system comprising a safety monitor including a circuit, wherein the circuit tracks the detection frequency of one or more objects in image data; and is configured to add at least one of the one or more objects to current image data for processing, wherein the circuit is configured to add the at least one object to current image data in response to the at least one object having a detection frequency exceeding a first threshold in previously processed image data. Claim 2 A system according to claim 1, wherein the circuit is configured to add the at least one object outside the boundary of the image corresponding to the current image data in order to generate modified image data. Claim 3 A system according to paragraph 2, wherein the circuit is configured to store a configuration of modified image data that identifies the location of at least one object in the modified image data. Claim 4 A system according to paragraph 2, wherein the circuit further transmits modified image data to an inference accelerator; receives from the inference accelerator at least one output representing the processing of the modified image data by the inference accelerator; and is configured to calculate the possibility of misclassification of the current image data based at least partially on the at least one output. Claim 5 A system according to claim 4, characterized in that the current image data is considered to be misclassified in response to a predetermined test vector data corresponding to the current image data and at least one output not matching. Claim 6 A system according to claim 1, characterized in that at least one object is selected based on at least a portion of predetermined test vector data corresponding to previously processed image data. Claim 7 A system according to claim 6, characterized in that, in the above-mentioned predetermined test vector data, the at least one object is identified as an object having a calculated probability of being correctly identified by an inference accelerator, which exceeds a second threshold in previously processed image data. Claim 8 A method comprising: tracking the detection frequency of one or more objects in image data by means of a safety monitor; and adding at least one of the one or more objects to current image data for processing by means of the safety monitor, wherein the at least one object is added to current image data in response to the detection frequency of the at least one object having a first threshold value in previously processed image data. Claim 9 A method according to claim 8, further comprising the step of adding the at least one object outside the boundary of the image corresponding to the current image data to generate modified image data by the safety monitor. Claim 10 A method according to claim 9, further comprising the step of storing a configuration of the modified image data that identifies the location of at least one object in the modified image data by the safety monitor. Claim 11 A method according to claim 9, further comprising: a step of transmitting modified image data to an inference accelerator by means of the safety monitor; a step of receiving from the inference accelerator by means of the safety monitor at least one output indicating processing of the modified image data by the inference accelerator; and a step of calculating the probability of misclassification of the current image data based at least partially on the at least one output by means of the safety monitor. Claim 12 A method according to claim 11, characterized in that the current image data is considered to be misclassified in response to a predetermined test vector data corresponding to the current image data and at least one output not matching. Claim 13 A method according to claim 8, wherein at least one object is selected based on at least a portion of a predetermined test vector data corresponding to previously processed image data. Claim 14 A method according to claim 13, characterized in that, in the above-mentioned predetermined test vector data, at least one object is identified as an object having a calculated probability of being correctly identified by an inference accelerator, which exceeds a second threshold in previously processed image data. Claim 15 A system comprising: an inference accelerator; and a safety monitor including a circuit, wherein the circuit tracks the detection frequency of one or more objects in image data; and is configured to add at least one of the one or more objects to current image data for processing, wherein the circuit is configured to add the at least one object to current image data in response to the at least one object having a detection frequency exceeding a first threshold in previously processed image data. Claim 16 A system according to claim 15, wherein the circuit is configured to add the at least one object outside the boundary of the image corresponding to the current image data in order to generate modified image data. Claim 17 A system according to claim 16, characterized in that the circuit is configured to store a configuration of modified image data that identifies the location of at least one object in the modified image data. Claim 18 A system according to claim 16, wherein the circuit further transmits modified image data to the inference accelerator; receives from the inference accelerator at least one output indicating processing of the modified image data by the inference accelerator; and is configured to calculate the possibility of misclassification of the current image data based at least partially on the at least one output. Claim 19 A system according to claim 15, characterized in that at least one object is selected based on at least a portion of predetermined test vector data corresponding to previously processed image data. Claim 20 A system according to claim 19, characterized in that, in the above-mentioned predetermined test vector data, the at least one object is identified as an object having a calculated probability of being correctly identified by an inference accelerator, which exceeds a second threshold in previously processed image data.

Images