A safe detector for image misclassification

Abstract

A system, apparatus, and method for implementing a safety monitoring framework for safety-critical reasoning applications are disclosed. The system includes a safety-critical reasoning application, a safety monitor, and an inference accelerator engine. The safety monitor receives an input image, test data, and a neural network specification from the safety-critical reasoning application. The safety monitor generates a modified image by adding additional objects outside the input image. The safety monitor provides the modified image and the neural network specification to the inference accelerator engine, which processes the modified image and provides an output to the safety monitor. The safety monitor determines the probability of incorrect processing of the original input image by comparing the output for the additional objects with known good results. The safety monitor supplements the overall fault coverage of the inference accelerator engine and covers faults that can only be observed at the network level.

Description

Background Art

[0001] Related technical description

[0002] One emerging area of ​​technology is machine learning, where neural networks are a type of machine learning model. Neural networks have shown excellent performance in tasks such as handwritten digit classification and face detection. In addition, neural networks have shown the potential to perform well in other more challenging visual classification tasks. Other applications of neural networks include speech recognition, language modeling, sentiment analysis, text prediction, and others.

[0003] In a typical deployment of machine learning algorithms, a software application supplies a neural network to an inference accelerator hardware engine. When an inference accelerator operates in a safety-critical environment, the inference accelerator needs to be monitored to check for abnormal behavior. A typical implementation for monitoring an inference accelerator is to insert monitoring logic into the inference accelerator processing hardware sub-box. For example, the machine check architecture is a mechanism in which monitoring logic in the processing hardware checks for abnormal behavior. However, while these checks provide monitoring at a lower hardware level, they may overlook faults that are only observable at the output of the inference accelerator. This approach also does not detect random intermittent faults at the neural network level. For example, an inference accelerator may have misclassification errors, such as failure to detect an object of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0005] Figure 1 is a block diagram of one implementation of a computing system.

[0006] Figure 2 is a block diagram of one implementation of a safety-critical reasoning application executed in a safety-critical system.

[0007] Figure 3 is a block diagram of one implementation of the inference fault detection framework.

[0008] Figure 4 is an example of an image sent from a safety-critical application to a safety monitoring framework according to one implementation.

[0009] Figure 5 is a modified image that has been generated from an original image by a security monitoring framework according to one implementation.

[0010] Figure 6 is a generalized flow chart illustrating one implementation of a method for operating a security monitoring framework for an inference accelerator.

[0011] Figure 7 is a generalized flow chart illustrating one implementation of a method for implementing a security monitoring framework.

[0012] Figure 8 is a generalized flow chart illustrating one implementation of a method for generating a confidence indicator for use by a safety-critical application.

[0013] Fig. 9 is a generalized flow chart illustrating one implementation of a method for determining which object to use to modify an image.

[0014] Fig.10 is a generalized flow chart illustrating one implementation of a method for a security monitoring framework to change operating modes. DETAILED DESCRIPTION

[0015] In the following description, many specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, it will be appreciated by those of ordinary skill in the art that various implementations may be practiced without these specific details. In some cases, well-known structures, components, signals, computer program instructions, and techniques are not shown in detail to avoid confusion with the methods described herein. It should be understood that for the sake of simplicity and clarity of description, the elements shown in the figures are not necessarily drawn to scale. For example, the size of some elements may be enlarged relative to other elements.

[0016] Disclosed herein are systems, apparatus, and methods for implementing a safety monitoring framework for safety-critical neural network applications. In one implementation, the system includes a safety-critical neural network application, a safety monitor, and an inference accelerator engine. The safety monitor receives an input image, test data (e.g., test vectors), and a neural network specification (e.g., layers and weights) from the safety-critical neural network application. In one implementation, the test data includes a list of images with known good objects / images and corresponding classifications of these objects / images. In one implementation, the safety monitor modifies the input image to add additional objects outside the boundaries of the input image. In such an embodiment, the modified image is larger than the input image to accommodate the additional objects outside the boundaries of the original image. In one implementation, the configuration of the modified image (i.e., where the additional space and additional objects are inserted) is stored in a data structure (i.e., metadata), which is passed along with the modified image via a system stream.

[0017] In one implementation, the additional objects include one or more redundant objects that are the same as objects found in a previous input image. In another implementation, the additional objects include one or more objects provided in a test vector. The safety monitor provides the modified image to an inference accelerator. The inference accelerator processes the modified image and provides an output back to the safety monitor. The safety monitor determines the probability of misclassifying the original input image based on the output generated by the inference accelerator. In one implementation, the safety monitor compares a known good result with the output associated with the modification to determine the probability of misclassifying the original input image. The safety monitor provides an indicator of the probability of misclassification to a safety-critical application. In this way, the safety monitor supplements the overall fault coverage and prevents faults that are only observable at the output of the neural network.

[0018] Reference now Figure 1 , a block diagram of one implementation of a computing system 100 is shown. In one implementation, the computing system 100 includes at least an inference accelerator engine 105, one or more processors 110A-B, an input / output (I / O) interface 120, a bus 125, and one or more memory subsystems 130. In other implementations, the computing system 100 may include other components and / or the computing system 100 may be arranged differently. In one implementation, the inference accelerator engine 105 is implemented on one or more processors 110B. The inference accelerator engine 105 represents any combination of software, firmware, and / or hardware for implementing various machine learning algorithms or machine learning models on one or more processors 110B.

[0019] In one implementation, the inference accelerator engine 105 implements one or more layers of a convolutional neural network. For example, in this implementation, the inference accelerator engine 105 implements one or more convolutional layers and / or one or more fully connected layers. In another implementation, the inference accelerator engine 105 implements one or more layers of a recurrent neural network. Generally speaking, an "inference engine" or "inference accelerator engine" is defined as hardware and / or software that receives image data and generates one or more label probabilities for the image data. In some cases, an "inference engine" or "inference accelerator engine" is referred to as a "classification engine" or "classifier."

[0020] The inference accelerator engine 105 is utilized in any of a variety of different applications that may vary depending on the implementation. For example, in one implementation, the inference accelerator engine 105 analyzes an image or video frame to generate one or more label probabilities for the frame. For example, possible use cases include at least eye tracking, object recognition, point cloud estimation, ray tracing, light field modeling, depth tracking, etc. For the eye tracking use case, the probabilities generated by the inference accelerator engine 105 are based on learning patterns, dwells, turn angles, blinks, etc. In other implementations, the inference accelerator engine 105 is trained and customized for other types of use cases.

[0021] The inference accelerator engine 105 can be used in any of a variety of different safety-critical applications that vary depending on the implementation. For example, in one implementation, the inference accelerator engine 105 is used in an automotive application. For example, the inference accelerator engine 105 controls one or more functions of an autonomous vehicle (i.e., an autonomous vehicle), a driver-assist vehicle, or an advanced driver-assistance system. In other implementations, the inference accelerator engine 105 is trained and customized for other types of use cases. Depending on the implementation, the inference accelerator engine 105 generates classification result probabilities for various objects detected in an input image or video frame.

[0022] The one or more processors 110A-B represent any number and type of processing units (e.g., central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC)). In one implementation, some of the processing associated with the inference accelerator engine 105 is performed by one or more processors 110B. In addition, the inference accelerator engine 105 is implemented using any of these types of processing units and / or other types of processing elements. The memory subsystem 130 includes any number and type of memory devices. For example, the memory types in the memory subsystem 130 may include high bandwidth memory (HBM), non-volatile memory (NVM), dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), or others. The memory subsystem 130 can be accessed by the inference accelerator engine 105 and one or more processors 110A-B. The I / O interface 120 represents any number and type of I / O interfaces (e.g., a peripheral component interconnect (PCI) bus, a PCI expansion (PCI-X), a PCIE (PCI Express) bus, a Gigabit Ethernet (GBE) bus, a universal serial bus (USB)). Various types of peripheral devices can be connected to the I / O interface 120. Such peripheral devices include (but are not limited to) a display, a keyboard, a mouse, a printer, a scanner, a joystick or other type of game controller, a media recording device, an external storage device, a network interface card, and the like.

[0023] In various implementations, computing system 100 is any of a computer, laptop, mobile device, gaming console, server, streaming device, wearable device, or various other types of computing systems or devices. In some implementations, the entire computing system 100 or one or more portions thereof are integrated into a robotic system, self-driving vehicle, autonomous drone, surgical tool, or other type of mechanical device or system. It should be noted that the number of components of computing system 100 varies from implementation to implementation. For example, in other implementations, there are Figure 1 It should also be noted that in other implementations, the computing system 100 includes Figure 1 Other components not shown in FIG. In addition, in other implementations, the computing system 100 is used with Figure 1 The different ways of structuring are shown in .

[0024] Now turn Figure 2, a block diagram of one implementation of a safety-critical reasoning application 220 executed in a safety-critical system 200 is shown. In one implementation, the safety-critical system 200 includes at least one or more processing units 205 and 210, which represent any number and type of processing units. It should be noted that the safety-critical system 200 may also include any number of other components, which are not shown to avoid obscuring the drawings. In one implementation, the one or more processing units 205 include one or more central processing units (CPUs). In other implementations, the one or more processing units 205 may include other types of processing units. In one implementation, the one or more processing units 210 include one or more graphics processing units (GPUs). In other implementations, the one or more processing units 210 may include other types of processing units (e.g., digital signal processors (DSPs), field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs)).

[0025] In one implementation, the safety-critical reasoning application 220 is executed on one or more processing units 205. The safety-critical reasoning application 220 represents any type of software application that is executed in a hazardous environment where safety is of high importance. For example, in one implementation, the safety-critical reasoning application 220 controls an autonomous or driver-assist vehicle or other vehicle. In other implementations, the safety-critical reasoning application 220 operates within a robot, as an automatic navigation control mechanism in an aircraft, or as part of other systems in a variety of challenging, high-risk environments.

[0026] In one implementation, the first portion of the safety monitoring framework 230 (i.e., the safety monitoring framework 230A) is executed on one or more processing units 205, and the second portion of the safety monitoring framework 230 (i.e., the safety monitoring framework 230B) is executed on one or more processing units 210. In other implementations, the safety monitoring framework 230 is executed entirely on one or more processing units 205 or entirely on one or more processing units 210. The inference accelerator 240 represents any combination of software and / or hardware for implementing one or more machine learning inference algorithms and / or machine learning inference models. In one implementation, the inference accelerator 240 is implemented using dedicated hardware (e.g., FPGA, ASIC, IP core). In another implementation, the inference accelerator 240 includes software instructions designed to be executed on one or more processing units 210. In other implementations, the inference accelerator 240 can be any suitable combination of software and / or hardware. In one implementation, the inference accelerator 240 operates according to the topology and weights / biases provided by the safety-critical inference application 220.

[0027] The safety-critical reasoning application 220 provides an image for processing by the inference accelerator 240. In one implementation, the safety monitoring framework 230 receives inputs being sent from the safety-critical reasoning application 220 to the inference accelerator 240, and the safety monitoring framework 230 modifies one or more of the inputs and then sends the modified inputs to the safety-critical reasoning application 220. After the inference accelerator 240 has processed the modified inputs, the inference accelerator 240 sends the processing results to the safety-critical reasoning application 220 via the safety monitoring framework 230. The safety monitoring framework 230 analyzes the results to determine whether the inference accelerator 240 has failed. In one implementation, the safety monitoring framework 230 generates a confidence indicator that specifies the degree of confidence that the safety monitoring framework 230 has in that the inference accelerator 240 accurately processed the inputs generated by the safety-critical reasoning application 220. The safety monitoring framework 230 transmits the confidence indicator and the modified version of the processing results to the safety-critical reasoning application 220. In one implementation, if the confidence indicator does not meet the threshold, the safety-critical reasoning application 220 takes one or more corrective actions (e.g., shut down, restart the system, retry the same image, generate a warning signal for the user, reduce vehicle speed, change operating mode).

[0028] Reference now Figure 3 , a block diagram of an implementation of an inference fault detection framework 300 is shown. In one implementation, a computing system 310 includes a safety monitoring framework 315 and inference accelerator hardware 350. In one implementation, the safety monitoring framework 315 includes a test generation unit 320, a scaler unit 325, a result filter 330, a test verification unit 335, an inference stack 340, and an inference driver 345. In other implementations, the safety monitoring framework 315 includes other components and / or is arranged in other suitable manners. In one implementation, the safety monitoring framework 315 is designed to be compatible with an automatic safety integrity level (ASIL) risk classification scheme. In other implementations, the safety monitoring framework 315 can be designed to comply with other risk classification schemes in other types of environments.

[0029] In one implementation, the safety-critical reasoning application 305 transmits the image, the test vector, and the reasoning network specification to the safety monitoring framework 315. In one implementation, the test generation unit 320 receives and uses the test vector to determine which objects to add to the original image. The test vector is also used to determine whether the results generated by the reasoning accelerator hardware 350 are accurate. In one implementation, a redundant object is added to the original image, wherein the redundant object is the same as an object present in a previous image. In another implementation, a given object that does not exist in the previous image is added to the original image. For example, in one implementation, the given object is an object that is identified in the test vector as a known good object in one or more test images. As used herein, a "known good object" is defined as an object that has a high probability of being correctly identified or processed in a consistent manner by the reasoning accelerator hardware 350. In one implementation, the selection of the given object added to the original image is based at least in part on a high probability of appearing in the image being processed. The test generation unit 320 adds one or more additional given objects to the original image to create a modified image. The reasoning stack 340 transmits the modified image and the reasoning network specification to the reasoning accelerator hardware 350. In one implementation, the modified image and inference network specification are transmitted to the inference accelerator hardware 350 via the inference driver 345 .

[0030] The inference accelerator hardware 350 processes the modified image according to the provided inference network specification, and then the inference accelerator hardware 350 returns the processing results to the inference stack 340. The result filter 330 analyzes the results and also filters the results to remove any additional objects added to the original image. The result filter 330 transmits the object and classification results to the application 305. The object and classification results can include any type of data, where the type varies depending on the implementation. For example, in one implementation, if the application 305 is looking for the number of objects in the original image, the results will include the number of objects and their locations within the original image. In other implementations, other data can be transmitted in the object and classification results, depending on what the application 305 is searching for in the original image.

[0031] In one implementation, the results from the inference accelerator hardware 350 are analyzed by the test verification unit 335. The test verification unit 335 determines whether one or more additional objects added to the original image are correctly processed and / or classified by the inference accelerator hardware 350. In other implementations, the test verification unit 335 may use other suitable techniques to verify the results generated by the inference accelerator hardware 350. If the test verification unit 335 determines that the inference accelerator hardware 350 correctly processes the known good object, the test verification unit 335 returns a qualified test result indicator to the application 305. Otherwise, if the known good object is incorrectly processed, the test verification unit 335 returns a failed test result indicator to the application 305. In response to receiving the failed test result indicator, the application 305 can take any of a variety of corrective actions (e.g., restart, generate an error, replay the same image).

[0032] Reference now Figure 4 , shows an example of an image 400A sent from a safety-critical application to a safety monitoring framework according to one implementation. Figure 4 In the example shown, image 400A includes a truck 402, a person 404, a dog 406, a horse 408, and another person 410. It should be understood that this example of image 400A including these objects is merely indicative of one particular implementation. The probabilities of detection and correct recognition are also shown at the top of the dashed box around each recognized object. In one implementation, these probabilities are determined during a training phase of a target reasoning accelerator engine. In one implementation, these probabilities are provided in test data that a safety-critical application provides to a safety monitoring framework for a particular neural network and / or for a target reasoning accelerator engine that is processing the image.

[0033] Reference now Figure 5 , shows a modified image 400B that has been generated from an original image 400A by a security monitoring framework according to one implementation. The modified image 400B is intended to represent the modified image 400B generated by a security monitoring framework (e.g., Figure 3 The image 400A ( Figure 4 As shown, the modified image 400B includes the original image 400A and a known good object region 505 that has been added to the modified image 400B outside the border (right side) of the original image 400A. The region of the original image 400A to the left of the modified image 400B remains unchanged. Thus, the portion of the original image 400A of the modified image 400B includes all of the original objects as they appear in ( Figure 4 ) in image 400A.

[0034] In one implementation, the known good object region 505 includes a person 510, which represents a known good object found in one of the test vector images. In other implementations, the known good object region 505 may include other numbers and / or types of objects. Depending on the implementation, the objects shown in the known good object region 505 may include redundant objects that are exact replicas of objects in the original image, and / or new objects that are not included as part of the original image 400A. In addition, although the known good object region 505 is shown on the right side of the modified image 400B, it should be understood that this is intended to represent only one specific implementation. In other implementations, the known good object region 505 can be added to the top, left, and / or bottom of the original image 400A. In some implementations, a portion of the original image is used as the known good object region 505. For example, if a portion of the image is considered non-essential for some reason (e.g., determining that the image content in a particular region has no functional impact on the processing results), then the particular region can be used as a known good object region.

[0035] After generating the modified image 400B, the security monitoring framework transmits the modified image 400B to the inference accelerator engine (e.g., Figure 3 The inference accelerator hardware 350 of the inference accelerator engine 350). The inference accelerator engine processes the modified image 400B and then transmits the output to the safety monitoring framework based on this processing. The safety monitoring framework determines whether the objects added to the known good object area 505 are recognized and / or processed based on the previously provided test data. If these added objects are correctly recognized and / or properly processed, the safety monitoring framework provides a qualified indicator to the safety-critical application. The safety monitoring framework also filters the processing results related to the original image 400A by excluding objects in the known good object area 505. The safety monitoring framework then provides these filtered results to the safety-critical application.

[0036] On the other hand, if the added object is incorrectly identified and / or erroneously processed, the safety monitoring framework provides a failure indicator to the safety-critical application. In response to receiving the failure indicator, the safety-critical application takes one or more corrective actions. For example, in one implementation, the safety-critical application terminates in response to receiving the failure indicator. In another implementation, in response to receiving the failure indicator, the safety-critical application generates the same frame for reprocessing by the inference accelerator engine. In other implementations, the safety-critical application performs other actions in response to receiving the failure indicator.

[0037] Reference now Figure 6, shows one implementation of a method 600 for operating a security monitoring framework for an inference accelerator. For discussion purposes, the steps and steps in this implementation are shown in sequential order. Figures 7 to 9 However, it should be noted that in various implementations of the described method, one or more of the described elements may be performed simultaneously, in a different order than shown, or omitted entirely. Other additional elements may also be performed as needed. Any of the various systems or devices described herein may be configured to implement method 600.

[0038] The safety monitoring framework receives test data and a neural network specification from a safety-critical reasoning application (block 605). The test data may include training information, test vectors, and / or other metadata. In some cases, the test data is supplied or received in advance by the safety monitoring framework before the method 600 begins. In one implementation, the test vectors include objects that have been previously identified and / or have a high probability of being identified in the processed image. In addition, the safety monitoring framework receives an input image from the safety-critical reasoning application (block 610). In one implementation, the input image is a real image that needs to be processed for a real-time application. Next, the safety monitoring framework generates a modified image from the input image based on the test data (block 615). In one implementation, the modified image includes the input image, wherein one or more additional objects are added outside the boundary of the input image. For example, in one implementation, the safety monitoring framework detects a first object within a previously input image. In this implementation, if the first object has been identified in the test data as having a probability exceeding a threshold to be identified by the reasoning accelerator engine, the safety monitoring framework adds the first object to a space outside the original input image. In this instance, the modified image includes the original input image and the first object. It should be noted that any number of objects may be added to the region outside the original input image.The modified image is created from a combination of the original input image and the additional region.

[0039] The safety monitoring framework then transmits the modified image to the inference accelerator engine (box 620). Next, the inference accelerator engine processes the modified image (box 625). It should be noted that the inference accelerator engine does not know that the original image has been modified. Therefore, the inference accelerator engine performs normal processing as if the image is received directly from the safety-critical application and has not been modified. The inference accelerator engine then transmits the output from processing the modified image to the safety monitoring framework (box 630). The safety monitoring framework checks the output to determine whether the inference accelerator engine is operating normally or has failed (box 635). For example, if one or more additional objects added to the modified image are processed in an expected manner, the safety monitoring framework concludes that the inference accelerator engine is operating normally. Otherwise, if the result from the inference accelerator engine processing one or more additional objects is unexpected and / or does not match the result of one or more of the same objects in the original image portion of the modified image, the safety monitoring framework concludes that the inference accelerator engine has failed.

[0040] If the safety monitoring framework determines that the inference accelerator engine has failed (conditional box 640, "yes" branch), the safety monitoring framework generates an error message and transmits the error message to the safety-critical inference application (box 645). In box 645, the safety monitoring framework may also provide other outputs from the inference accelerator engine to the safety-critical application. In response to receiving the error message, the safety-critical application performs one or more corrective actions (box 650). It should be noted that in some cases, the safety-critical application may optionally decide to continue to send subsequent images to the inference accelerator engine for processing, depending on the type of error detected and / or the current state of the safety-critical application. Alternatively, in response to receiving the error message, the safety-critical application may decide to terminate. If the safety monitoring framework determines that the inference accelerator engine is operating normally (conditional box 640, "no" branch), the safety monitoring framework generates a non-error message and transmits the non-error message to the safety-critical application (box 655). In box 655, the safety monitoring framework may also provide other outputs from the inference accelerator engine (e.g., filtered results) to the safety-critical application. In response to receiving the non-error message, the safety-critical application provides another image to the safety monitoring framework and the method 600 returns to block 610 .

[0041] Reference now Figure 7, showing one implementation of a method 700 for implementing a safety monitoring framework. The safety monitoring framework receives an input image and test data from a safety-critical application (box 705). The safety monitoring framework determines whether any object identified in the test data exists in a previous image (box 710). For the purposes of this discussion, it is assumed that the input image is part of a continuous sequence of images, such as a camera input stream. If any of the one or more objects identified by the test data exists in a previous image (conditional box 715, "yes" branch), the safety monitoring framework modifies the input image by adding redundant copies of one or more of these identified objects outside the original image boundary (box 720). The safety monitoring framework then transmits the modified image to the inference accelerator engine (box 725). If any object identified by the test data does not exist in the previous image (conditional box 715, "no" branch), the safety monitoring framework passes the input image to the inference accelerator engine and performs processing in a conventional manner (box 730). After box 730, the method 700 ends.

[0042] After box 725, the inference accelerator engine processes the modified image and returns the first set of processing results to the safety monitoring framework (box 735). The safety monitoring framework analyzes the first set of processing results to generate an error status message (box 740). It should be noted that the error status message refers to the error status of the inference accelerator engine or its probability. In one implementation, the error status message is a single bit indicating that the inference accelerator engine is operating normally or has failed. The safety monitoring framework also converts the first set of processing results into a second set of processing results (box 745). In one implementation, the safety monitoring framework converts the first set of processing results into the second set of processing results by removing any result data associated with the modification of the original image from the first set of processing results. In other words, the second set of processing results is what the processing results would look like if the original image, rather than the modified image, had been processed by the inference accelerator engine. Next, the safety monitoring framework transmits the error status message and the second set of processing results to the safety-critical application (box 750). After box 750, method 700 ends.

[0043] Reference now Figure 8, showing one implementation of a method 800 for generating a confidence indicator used by a safety-critical application. A safety monitoring framework intercepts an image being sent from a safety-critical application to an inference accelerator engine (block 805). In one implementation, the safety monitoring framework is placed between the safety-critical application and the inference accelerator engine. In one implementation, neither the safety-critical application nor the inference accelerator engine is aware that the safety monitoring framework is placed between them. However, in other implementations, the safety-critical application and / or the inference accelerator engine is aware that the safety monitoring framework is located between them. The safety monitoring framework modifies the image one or more times and then transmits the modified image to the inference accelerator engine (block 810). In one implementation, the one or more modifications include adding one or more additional objects to a space outside the original image. For a video stream, in the case of object recognition, one method of increasing the confidence level is to embed a similar object and a different object into additional space in the next video frame (assuming that the objects are still in the next captured frame at a close temporal distance).

[0044] After the inference accelerator engine processes the modified image, the safety monitoring framework intercepts the results being sent from the inference accelerator engine to the safety-critical application (box 815). The safety monitoring framework analyzes the results to determine the probability that the inference accelerator engine correctly processed the image (box 820). For example, in one implementation, the safety monitoring framework determines whether the results indicate that a misclassification has occurred. If the results from the processing modification of the image are inconsistent with the results provided by the test data, the safety monitoring framework will conclude that the probability that the inference accelerator incorrectly processed the original input image is relatively high. In one implementation, the relatively high probability that the inference accelerator incorrectly processes the image is represented by a relatively low (i.e., close to zero) confidence indicator.

[0045] Next, the safety monitoring framework generates a confidence indicator to specify the probability that the inference accelerator engine correctly processed the image (block 825). In addition, the safety monitoring framework modifies the result by filtering out any results related to modifications made to the original image (block 830). The safety monitoring framework then transmits the confidence indicator and the modified result back to the safety-critical application (block 835). After block 835, the method 800 ends.

[0046] Now refer to Fig. 9, shows one implementation of a method 900 for determining which object is used to modify an image. A safety monitor receives test data including known good objects and their probabilities of appearing in a real image (block 905). In one implementation, a safety-critical application transmits the test data to the safety monitor. In addition, the safety monitor analyzes the detected output of previous images to track the detection frequency of various objects in previous images (block 910).

[0047] The safety monitor determines whether any object has both a probability of occurrence greater than a first threshold and a detection frequency greater than a second threshold in a previous image (conditional box 915). It should be noted that the values ​​of the first threshold and the second threshold may vary depending on the implementation. If any object has both a probability of occurrence greater than the first threshold and a detection frequency greater than the second threshold in a previous image (conditional box 915, "yes" branch), the safety monitor adds one or more of these objects to the next image being transmitted to the inference accelerator engine (box 920). Otherwise, if no object has both a probability of occurrence greater than the first threshold and a detection frequency greater than the second threshold in a previous image (conditional box 915, "no" branch), the safety monitor adds at least one object with a probability of occurrence greater than the first threshold or a detection frequency greater than the second threshold in a previous image to the next image (box 925). If no object has a probability of occurrence greater than the first threshold or a detection frequency greater than the second threshold in a previous image, the safety monitor may choose to add another type of object to the next image or choose not to add any object. Note that if no object has both a probability of occurrence greater than the first threshold and a detection frequency in previous images greater than the second threshold, the safety monitor may decrease the value of the first threshold and / or the value of the second threshold. After blocks 920 and 925, method 900 ends.

[0048] Reference now Fig.10, showing an implementation of a method 1000 for a safety monitoring framework to change an operating mode. A safety monitor in a safety-critical system having a safety-critical application and an inference accelerator engine is started by operating in a first mode (block 1005). In one implementation, when operating in the first mode, the safety monitor sends real image data to the inference accelerator engine for classification. If the safety monitor detects a first condition (conditional block 1010, "yes" branch), the safety monitor switches to a second mode for monitoring the inference accelerator engine (block 1015). It should be noted that the "second mode" may also be referred to as a "safe mode" herein. In one implementation, when operating in the second mode, the safety monitor generates experimental image data and transmits the experimental image data to the inference accelerator engine for classification. It should be noted that the "experimental image data" may also be referred to as "known good object image data" herein. In this implementation, after the inference accelerator engine generates a classification result for the experimental image data, the safety monitor generates a confidence indicator based on an analysis of the classification result, wherein the confidence indicator represents the probability that the result is accurate.

[0049] In one implementation, the first condition is receiving a signal from a safety-critical application that enhanced monitoring should be performed. For example, a particularly important code segment within an application may be detected, and the application may determine that an increased review of the inference accelerator engine is required during the time period when this code segment is being executed. In another implementation, the first condition is detection of abnormal behavior associated with the inference accelerator engine. In another implementation, the first condition is expiration of a timer. In other implementations, the first condition may be other types of conditions.

[0050] In one implementation, when operating in the first mode, the safety monitor modifies one of every N images, where N is a positive integer greater than one. In this implementation, when operating in the second mode, the safety monitor modifies one of every M images, where M is a positive integer less than N. If the safety monitor does not detect the first condition (conditional block 1010, "no" branch), the safety monitor remains in the first mode (block 1020). After block 1020, method 1000 returns to conditional block 1010.

[0051] After box 1015, if the safety monitor detects a second condition (conditional box 1020, "yes" branch), the safety monitor returns to the first mode (box 1025). In one implementation, the second condition is receiving a signal from the safety-critical application to re-enter the first mode. In another implementation, the second condition is detecting that the inference accelerator engine returns to normal behavior. In another implementation, the second condition is that the value of the timer is within a specific range. In other implementations, the second condition can be other types of conditions. After box 1025, method 1000 returns to conditional box 1010. Otherwise, if the safety monitor does not detect the second condition (conditional box 1020, "no" branch), the safety monitor remains in the second mode (box 1030). After box 1030, method 1000 returns to conditional box 1020. It should be understood that in other implementations, the safety monitor operates in more than two different types of modes. In these implementations, the frequency of the inference accelerator can be adjusted with finer granularity according to the state of the application and / or the inference accelerator engine. For example, the safety monitor may change from modifying every tenth frame, every nine frames, every eight frames, every seven frames, etc. Alternatively, the safety monitor may switch between generating experimental image data every other frame, every third frame, every fourth frame, etc.

[0052] In various implementations, the method and / or mechanism described herein is implemented using program instructions of a software application. For example, it is contemplated that program instructions that can be executed by a general-purpose processor or a special-purpose processor are implemented. In various implementations, such program instructions can be represented by a high-level programming language. In other implementations, program instructions can be compiled from a high-level programming language into a binary form, an intermediate form, or other forms. Alternatively, program instructions describing the behavior or design of hardware can be written. Such program instructions can be represented by a high-level programming language (such as C). Alternatively, a hardware design language (HDL), such as Verilog, can be used. In various implementations, program instructions are stored on any of a variety of non-temporary computer-readable storage media. The storage medium can be accessed by a computing system during use to provide program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

[0053] It should be emphasized that the above implementations are only non-limiting examples of implementations. Once the above disclosure is fully understood, various changes and modifications will become apparent to those skilled in the art. The appended claims are intended to be interpreted as covering all such variations and modifications.

Claims

1. A system comprising: an inference accelerator engine configured to classify objects detected in the image data; as well as A safety monitor configured to: transmitting real image data from a safety-critical application to the inference accelerator engine; determining which known good object image data to generate based at least in part on a probability of occurrence in real image data and / or a frequency of detection in previous images, wherein a known good object is an object that is known to have been correctly classified by the inference accelerator engine with a relatively high probability; generating the known good object image data and transmitting the known good object image data to the inference accelerator engine; and generating a confidence indicator based on an analysis of results produced by the inference accelerator engine classifying the known good object image data, wherein the confidence indicator represents a probability that the real image data was correctly classified by the inference accelerator engine; Wherein the system is configured to perform one or more corrective actions in response to the confidence indicator failing to satisfy a threshold.

2. The system of claim 1, wherein the safety monitor is further configured to: When operating in a first mode, only real image data from a safety-critical application is transmitted to the inference accelerator engine; entering a second mode in response to detecting the first condition; and The known good object image data is generated in response to entering the second mode and the known good object image data is transmitted to the inference accelerator engine.

3. The system of claim 2, wherein the first condition comprises receiving a signal from the safety-critical application to enter the second mode, and wherein a first corrective action is to terminate the safety-critical application.

4. The system of claim 1, wherein the safety monitor is further configured to: generating a modified image by combining the known good object image data with the real image data without altering the real image data; and The modified image is transmitted to the inference accelerator engine.

5. The system of claim 4, wherein the safety monitor is further configured to: analyzing detected outputs of previous images to track detection frequencies of various objects in the previous images; determining whether any object has both a probability of occurrence greater than a first threshold and a frequency of detection in a previous image greater than a second threshold; and The one or more first objects are added to the next image in response to determining that the one or more first objects have both a probability of occurrence greater than the first threshold and a frequency of detection in a previous image greater than the second threshold.

6. The system of claim 4, wherein the safety monitor is further configured to: detecting at least one known good object in the test vector data; adding the at least one known good object to additional space outside the original boundaries of the input image; and A modified image is created from the input image and the additional space.

7. The system of claim 1, wherein the safety monitor is configured to receive test data from the safety-critical application, the test data indicating how the inference accelerator engine should classify the known good object image data.

8. A method for implementing a security monitoring framework, the method comprising: transmitting, by the safety monitor, real image data from the safety-critical application to an inference accelerator engine configured to classify objects detected in the image data; determining, by the safety monitor, which known good object image data to generate based at least in part on a probability of occurrence in real image data and / or a frequency of detection in previous images, wherein a known good object is an object known to have been correctly classified by the inference accelerator engine with a relatively high probability; generating, by the safety monitor, the known good object image data and transmitting the known good object image data to the inference accelerator engine; generating, by the safety monitor, a confidence indicator based on an analysis of results produced by the inference accelerator engine classifying the known good object image data, wherein the confidence indicator represents a probability that the real image data was correctly classified by the inference accelerator engine; as well as One or more corrective actions are performed by the safety-critical application in response to the confidence indicator failing to satisfy a threshold.

9. The method of claim 8, further comprising the safety monitor: When operating in a first mode, only real image data from a safety-critical application is transmitted to the inference accelerator engine; entering a second mode in response to detecting a first condition; and The known good object image data is generated in response to entering the second mode and the known good object image data is transmitted to the inference accelerator engine.

10. The method of claim 9, wherein the first condition comprises receiving a signal from the safety-critical application to enter the second mode, and wherein a first corrective action is to terminate the safety-critical application.

11. The method of claim 8, further comprising: generating, by the safety monitor, a modified image by combining the known good object image data with the real image data without altering the real image data; as well as The modified image is transmitted to the inference accelerator engine.

12. The method of claim 11, further comprising: analyzing, by the safety monitor, detected outputs of previous images to track detection frequencies of various objects in the previous images; determining, by the safety monitor, whether any object has both a probability of occurrence greater than a first threshold and a frequency of detection in a previous image greater than a second threshold; as well as The one or more first objects are added, by the safety monitor, to the next image in response to determining that the one or more first objects have both a probability of occurrence greater than the first threshold and a frequency of detection in a previous image greater than the second threshold.

13. The method of claim 11, further comprising: detecting, by the safety monitor, at least one known good object in the test vector data; adding, by the safety monitor, the at least one known good object to additional space outside of an original boundary of the input image; as well as A modified image is created by the security monitor from the input image and the additional space.

14. The method of claim 8, further comprising: Test data is received by the safety monitor from the safety-critical application, the test data indicating how the inference accelerator engine should classify the known good object image data.

15. An apparatus for implementing a security monitoring framework, the apparatus comprising: a memory storing program instructions; as well as at least one processor coupled to the memory, wherein the program instructions are executable by the at least one processor to: transmitting real image data from a safety-critical application to an inference accelerator engine configured to classify objects detected in the image data; determining which known good object image data to generate based at least in part on a probability of occurrence in real image data and / or a frequency of detection in previous images, wherein a known good object is an object that is known to have been correctly classified by the inference accelerator engine with a relatively high probability; generating the known good object image data and transmitting the known good object image data to the inference accelerator engine; generating a confidence indicator based on an analysis of results produced by the inference accelerator engine classifying the known good object image data, wherein the confidence indicator represents a probability that the real image data was correctly classified by the inference accelerator engine; and One or more corrective actions are performed in response to the confidence indicator not satisfying a threshold.

16. The apparatus of claim 15, wherein the program instructions are further executable by the at least one processor to: When operating in a first mode, only real image data from a safety-critical application is transmitted to the inference accelerator engine; entering a second mode in response to detecting a first condition; and The known good object image data is generated in response to entering the second mode and the known good object image data is transmitted to the inference accelerator engine.

17. The apparatus of claim 16, wherein the first condition comprises receiving a signal from the safety-critical application to enter the second mode, and wherein a first corrective action is to terminate the safety-critical application.

18. The apparatus of claim 15, wherein the program instructions are further executable by the at least one processor to: generating a modified image by combining the known good object image data with the real image data without altering the real image data; and The modified image is transmitted to the inference accelerator engine.

19. The apparatus of claim 18, wherein the program instructions are further executable by the at least one processor to: analyzing detected outputs of previous images to track detection frequencies of various objects in the previous images; determining whether any object has both a probability of occurrence greater than a first threshold and a frequency of detection in a previous image greater than a second threshold; and The one or more first objects are added to the next image in response to determining that the one or more first objects have both a probability of occurrence greater than the first threshold and a frequency of detection in a previous image greater than the second threshold.

20. The apparatus of claim 18, wherein the program instructions are further executable by the at least one processor to: detecting at least one known good object in the test vector data; adding the at least one known good object to additional space outside the original boundaries of the input image; and A modified image is created from the input image and the additional space.

Images