System, method and non-temporary computer readable medium for detecting abnormality of artificial intelligence system

Abstract

To provide a system, a method and a non-temporary computer readable medium for detecting an abnormality that may occur in an AI algorithm.SOLUTION: A system for detecting an abnormality of an AI system, receives at least mobile device position data and artificial intelligence (AI) output data from a mobile device, determines a position corresponding to the mobile device position data, accesses past AI output data stored in a data structure and associated with the position on the basis of at least the position, compares the received AI output data with the past AI output data stored in the data structure with resect to the position, and transmits a command to allow the mobile device to execute at least one action to the mobile device on the basis of at least the comparison between the received AI output data and the past AI output data associated with the position.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a system, method and non-transitory computer-readable medium for detecting anomalies in an artificial intelligence system. [Background technology]

[0002] Semi-autonomous vehicle systems or other Advanced Driver Assistance Systems (ADAS), and Autonomous Driving (AD) or other autonomous driving systems are systems that automate or otherwise enhance vehicle control to improve safety, autonomous navigation, etc. Such systems may use embedded vehicle computing devices called “Electronic Control Units” (ECUs), which may include an AD / ADAS ECU, a powertrain ECU, a chassis ECU, etc. For example, an AD / ADAS ECU may be configured to perform domain-specific functions, such as perception, localization, navigation, vehicle control, etc. To perform these functions, artificial intelligence (AI) algorithms (e.g., machine learning models) may be implemented in the AD / ADAS ECU or other vehicle ECUs. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-128457 Summary of the Invention [Problem to be solved by the invention]

[0004] AI algorithms may include deep neural networks (DNNs), and / or various other types of artificial neural networks, and / or various other types of machine learning models. For example, AI algorithms may be used in ECUs of AD / ADAS vehicles to achieve a highly accurate and safe autonomous driving experience. For example, AI applications may be used in various roles, such as object detection and mission-critical control, for example, emergency braking, lane changes, and evasive maneuvers. Malfunctioning of AI algorithms on which vehicles rely can lead to serious accidents. Therefore, while the use of AI algorithms to enable a vehicle's AD / ADAS is expected to be beneficial, ensuring these systems operate properly and as intended can improve vehicle safety.

[0005] The present invention was made in light of these problems, and aims to provide a system, method, and non-transitory computer-readable medium for detecting anomalies that may occur in AI algorithms. [Means for solving the problem]

[0006] The present invention includes multiple means for solving at least some of the above-mentioned problems, one example of which is as follows: That is, a system for detecting an anomaly in an AI system receives at least mobile device location data and artificial intelligence (AI) output data from a mobile device, determines a location corresponding to the mobile device location data, accesses past AI output data stored in a data structure and associated with the location based at least on the location, compares the received AI output data with past AI output data stored in the data structure for the location, and transmits an instruction to the mobile device to cause the mobile device to perform at least one action based at least on a comparison of the received AI output data and the past AI output data associated with the location. [Effects of the Invention]

[0007] The present invention provides a system, method, and non-transitory computer-readable medium for detecting anomalies that may occur in AI algorithms.

[0008] Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a logical configuration of a system that performs efficient monitoring of AI algorithms in an embodiment. [Figure 2] FIG. 1 illustrates an example of a hardware configuration of a system according to an embodiment. [Figure 3] FIG. 10 is a diagram showing an example of diagnostic levels in an embodiment. [Figure 4] FIG. 1 is a flow diagram illustrating an example process for performing diagnostics on an AI algorithm in an embodiment. [Figure 5] FIG. 10 is a diagram showing an example of identifying an AI algorithm that is likely to have an AI abnormality from among multiple AI algorithms in an embodiment. [Figure 6] FIG. 10 illustrates an example of reducing the number of false positives for detecting AI anomalies in an embodiment. [Figure 7] FIG. 10 is a diagram showing an example of determining whether to add AI application output data to a historical AI application output database in an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A detailed description of embodiments of the present invention will now be described with reference to the drawings, in which the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings refers to similar or identical items or features. [Example]

[0011] Some embodiments herein are directed to methods and apparatus configurations for detecting anomalies that may occur in AI algorithms, such as those used by AI applications executed for ADAS and / or AD systems used in vehicles and / or computing devices of other types of mobile devices that may use AI algorithms (e.g., aerial drones and other types of aircraft, marine drones and other types of marine vehicles, underwater drones and other types of underwater vehicles, mobile autonomous and semi-autonomous robots, etc.). For example, an AI application may include an application that executes an AI algorithm to generate an output or result. Embodiments herein can detect indicators of AI algorithm anomalies for various AI embedded systems. Embodiments herein include detecting indicators of AI algorithm anomalies on the cloud side based on current and / or past AI application outputs, and further based on on-board sensor configuration information for the vehicle. Examples of AI application outputs may include detected object identifiers, detected object locations, corresponding confidence scores, etc. For example, if a current AI application output does not match a historical AI application output at the same location and the current AI application output has a high confidence level, this discrepancy may indicate the presence of an AI algorithm anomaly. As an example, an AI application output may be determined to indicate an AI algorithm anomaly if the AI application recognizes a human on a highway with a high level of confidence, despite the historical probability of a human being being observed on the highway at that location being very low.

[0012] In some embodiments, historical data of AI application outputs is stored in a database or other type of data structure for each road segment traversed by the vehicle. A predictive determination of the AI application output may be performed for each road segment that the vehicle sensors can cover at the current vehicle position, which may include taking into account sensor configuration information for each of the current and past vehicles. As a result, the number of diagnostic processes performed on the vehicle side can be reduced by dynamically changing the diagnostic level (e.g., corresponding to the frequency of diagnostics) according to the determination result of the indicators of AI algorithm abnormality.

[0013] Additionally or alternatively, rather than storing historical AI application outputs in association with road segments, such as in embodiments herein directed to other types of mobile devices, the AI application outputs may be stored in association with corresponding location information, such as longitude and latitude coordinates (and possibly altitude / depth). Clustering methods or other thresholds may be applied to group together AI application outputs that are within a threshold distance of one another. When retrieving historical AI application outputs from a database or other data structure, a similar threshold may be applied to location data received from a mobile device to determine that historical AI application outputs that are sufficiently close to the mobile device's received location are relevant to the mobile device's AI algorithm outputs.

[0014] Conventional methods for implementing AI algorithms generally cannot detect indicators of AI algorithm anomalies. On the other hand, embodiments herein can efficiently (i.e., with little computational overhead) detect various indicators of AI algorithm anomalies, which may be caused by, for example, hardware issues, software issues, and / or security issues. For example, hardware issues may include faulty (permanently failed) memory locations, faulty sensors, faulty network connections, hardware intrusions, etc. Furthermore, software issues may be caused by poor training of the AI algorithm, bugs in the programming code, or security issues, such as those due to malicious attacks, all of which may cause unexpected AI behavior. Therefore, embodiments herein are intended to ensure that AI algorithms behave correctly during their operation.

[0015] During the design and testing phase of an AI algorithm, the proper operation of the AI algorithm is typically verified using a large amount of data as an AI algorithm validation dataset. However, the size of the AI validation dataset is too large to implement on a vehicle computing device. Furthermore, the available computing power of a vehicle computing device is usually limited, and validation of an AI algorithm using such limited computing resources cannot be achieved in a realistic amount of time. Therefore, embodiments herein are configured to efficiently use the computing resources of a vehicle computing device to validate an AI algorithm. As a result, embodiments herein substantially reduce the number of computationally intensive diagnostic functions for detecting AI anomalies, and improve the reliability of embedded AI systems, such as for AD / ADAS ECUs or other vehicle computing devices.

[0016] Examples herein include AI diagnostic methods for diagnosing anomalies in AI algorithms implemented in an AD / ADAS ECU or other vehicle computing device. For example, a vehicle herein may include one or more machine learning models (MLMs), such as artificial neural networks or other types of AI algorithms, for performing various functions. In some cases, the AI algorithms may be included in or otherwise used by an AI application executing on the AD / ADAS ECU or other vehicle computing device onboard the vehicle.

[0017] Additionally, some embodiments herein may include AI diagnostic techniques that can efficiently improve the reliability of vehicle AI algorithms and corresponding AI applications, and may be implemented to efficiently detect improperly operating AI algorithms, such as those that may be caused by hardware malfunctions, other hardware anomalies, security breaches, software errors, etc., as described above.

[0018] For example, hardware malfunctions can cause an AI algorithm to begin behaving differently over time compared to when the AI algorithm is run immediately after training and deployment. There are various factors, such as permanent hardware failures and / or hardware intrusions, that can affect the output of an AI algorithm and potentially cause a serious accident. Thus, embodiments herein can determine whether an AI algorithm behavior has changed and / or whether the AI algorithm is operating normally.

[0019] The AI diagnostic methods described herein do not require hardware redundancy (e.g., adding other AI execution hardware) and are not dependent on any particular hardware. The AI diagnostic methods described herein are cost-effective and also improve vehicle safety by efficiently examining the operation of AI algorithms. As an example, if a hardware anomaly occurs in an ECU system, which may include an AD / ADAS ECU, a powertrain ECU, a chassis ECU, etc., embodiments herein may efficiently identify the cause of the anomaly, such as which hardware element is associated with the AI algorithm that is not operating properly. Consequently, embodiments herein also improve the maintainability and repair efficiency of the ECU system.

[0020] For purposes of explanation, some exemplary embodiments are described in the context of an AD / ADAS ECU running one or more AI algorithms onboard a vehicle. However, the embodiments herein are not limited to the specific embodiments provided and may be extended to other types of computing devices, other types of vehicles, other types of AI algorithms, other types of hardware configurations, other types of local conditions, etc., as will be apparent to those skilled in the art in light of the disclosure herein. For example, the solutions herein are scalable and may be applied to ubiquitous systems, such as construction, agricultural, and mining equipment, ships and other marine vehicles, in addition to ground vehicles. The embodiments herein may also be scaled to smaller applications, such as autonomous robots.

[0021] 1 illustrates an example logical configuration of a system 100 for implementing efficient monitoring of AI algorithms, according to some embodiments. System 100 includes a vehicle 102, which is an example of a ground vehicle mobile device to which the methods described herein may be applied. As noted above, in other examples, the methods and apparatus configurations herein also apply to other types of mobile devices that may use AI algorithms, such as aerial drones and other types of aircraft, marine drones and other types of marine vehicles, underwater drones and other types of underwater vehicles, and mobile autonomous and semi-autonomous robots. Consequently, the description herein applies to all of these and other types of mobile devices that use AI algorithms.

[0022] In this example, vehicle 102 includes one or more vehicle computing devices 104 that can communicate with one or more service computing devices 108 over one or more networks 106. Vehicle 102 may also include one or more onboard sensors 112 and one or more vehicle systems 114 that communicate with vehicle computing device 104, such as via a Controller Area Network (CAN) bus (not shown in FIG. 1 ).

[0023] The one or more networks 106 may include any suitable network, including a wireless network such as a cellular network, a wide area network such as the Internet, a local area network such as an intranet, a local wireless network such as Wi-Fi, a short-range wireless communication such as BLUETOOTH®, a wired network including fiber optics and Ethernet, any combination thereof, or any other suitable communication network. The components used for such communication technologies may depend, at least in part, on the type of network, the selected environment, or both. Protocols for communicating over such networks are well known and will not be described in detail herein.

[0024] In this example, a vehicle AI diagnostic program (not shown in FIG. 1 ) executing on the service computing device 108 may access services provided by the service computing device 108 to detect indicators of AI algorithm anomalies and may maintain or otherwise access a sensor configuration database 120, which may include information about the sensor configuration of each individual vehicle 102. The service computing device 108 may further maintain or otherwise access a historical AI application output database 122, which may include the output of multiple AI applications of multiple other vehicles 102 in the past at specific locations, such as those that may be associated with road segments, or other location information. For example, the location may be determined based on satellite positioning system information received from satellite receivers onboard each vehicle, or other location sensing methods, and may include specific geolocation coordinates (e.g., longitude and latitude) corresponding to the recognition of previously identified landmarks, geographic features, traffic signs / signals / intersections, other vehicles, pedestrians, or any of a number of other recognized objects, road features and conditions, etc.

[0025] In some cases, road segments may be determined for some or all of the roads in an area. Numerous methods for defining road segments based on mapping information are known in the art. As one non-limiting example, the service computing device 108 may obtain mapping information from a map-providing service server or the like (not shown in FIG. 1 ) and identify routes along the roads shown in the mapping information. The service computing device 108 may divide each identified route into a number of waypoints. As an example, the waypoints may be identified on a high-precision map or a standard map, which may correspond to specific longitude and latitude coordinates. The service computing device 108 may then identify respective road segments between each successive pair of waypoints. The length of each road segment between each pair of waypoints may range from a few centimeters to hundreds of meters, depending in part on the road configuration and the desired level of granularity. Based at least on the waypoint information, each road segment may be associated with geolocation information. Furthermore, while one method for defining road segments is described herein, numerous variations will be apparent to those skilled in the art having the benefit of this disclosure.

[0026] Additionally, while road segments are described in examples herein as one method of associating AI output data with geolocation coordinates, in other examples, road segments may not be used. Instead, actual geolocation coordinates may be associated with each set of AI output data, such as in examples herein directed to other types of mobile devices, and various methods, such as clustering methods, may be used to associate particular AI output data with particular vehicle geolocation coordinates. For example, clustering or other thresholding methods may be applied to group together AI application outputs that are within a threshold distance of one another. Subsequently, when historical AI application outputs are retrieved from historical AI application output database 122, the same or a different threshold may be applied to the location data received from the mobile device to determine that historical AI application outputs that are sufficiently close to the mobile device's received location are associated with the mobile device's AI algorithm output.

[0027] In this example, a vehicle AI diagnostic program (not shown in FIG. 1 ) may send a vehicle identifier (ID) for a particular vehicle 102 and the vehicle's 102's current location 124 to the service computing device 108. The service computing device 108 may use the vehicle ID to extract the vehicle's 102's sensor configuration by accessing the sensor configuration database 120. If the vehicle sensor configuration is not already included in the sensor configuration database 120, the service computing device 108 may request the sensor configuration from the vehicle computing device 104. Additionally, the vehicle computing device 104 may send AI application output data 126 to the service computing device 108 to enable the service computing device 108 to detect whether there may be an anomaly in the AI algorithm 128 used by the AI application 130 running on the vehicle computing device 104.

[0028] As an example, AI algorithms 128, such as convolutional neural networks, deep learning neural networks, or other types of machine learning models, may be used for object detection and recognition (or to perform any of a variety of other AI functions) by AI applications 130 onboard the vehicle 102. The service computing device 108 may receive the AI application output data 126 from the vehicle computing device 104 to perform cloud-side AI anomaly indicator detection 134 by comparing the AI application output data 126 with historical AI application outputs in the historical AI application output database 122 for the same approximate vehicle position and the same or similar sensor configuration. Based on the analysis result of whether any indicators of AI anomalies are detected on the cloud side, the service computing device 108 may send a diagnostic level 136 to the vehicle computing device 104.

[0029] The diagnostic level 136 sent to the vehicle computing device 104 may instruct the vehicle computing device 104 as to which diagnostic level to apply to a particular AI application 130 based on the analysis results for detecting anomalies in the AI algorithm. The diagnostic level may be dynamically changed at the vehicle computing device 104 as appropriate, which helps reduce the computational load at the vehicle computing device 104. For example, as described further below, the vehicle computing device 104 may execute a vehicle AI diagnostic program according to different diagnostic levels.

[0030] In the service computing device 108, the data used to detect indicators of AI anomalies may include historical AI application outputs extracted from database information in the historical AI application output database 122, the sensor configuration of the vehicle 102 extracted from the sensor configuration database 120, vehicle position determined from a vehicle GPS receiver (not shown in FIG. 1), vehicle ID, and current AI application output data 126 being output by the AI application 130 running on the vehicle computing device 104.

[0031] In some examples, historical AI application output database 122 may store information about objects recognized by the AI application for each road segment of a plurality of previously identified road segments that correspond to mapping information about the region in which vehicle 102 is traveling. As one specific example, assume that historical AI application output database 122 indicates that 1,050 traffic lights were recognized in 850 camera images for a selected road segment n. This information may represent, for example, a historical probability of object appearance, indicating that AI application 130 is likely to recognize a traffic light when vehicle 102's position corresponds to road segment n. Additionally or alternatively, as described above, location information (e.g., longitude, latitude, altitude, depth, etc.) and clustering may be used in place of or in addition to road segments in other examples.

[0032] The sensor configuration database 120 may include information on the configuration and specifications of onboard sensors installed on the vehicle 102. Additionally, the sensor configuration information may include the direction and range used by the sensors to identify the road segment being covered (e.g., road segments n, n+1, and n+2). The vehicle position data may include the vehicle's current longitude and latitude information, which is used to identify the road segment being covered and to determine the corresponding global coordinates of any detected objects. As mentioned above, the vehicle position data may be determined by satellite positioning system information, such as by the output of the vehicle's GPS (Global Positioning System) receiver. Additionally or alternatively, other types of location sensors may be used to determine or verify the vehicle's geographic location, such as WiFi signals, cell phone towers, traffic cameras, and communications from other vehicles. The vehicle ID may be an individually distinguishable value that identifies a particular vehicle 102 (e.g., unique within the system 100) and may be used by the service computing device 108 to access the corresponding sensor configuration information (or at least the vehicle's make and model) for the particular vehicle 102 from the sensor configuration database 120.

[0033] In some examples, current AI application output data 126 may include information such as detected object name (e.g., at least the type of object recognized), confidence score, and object location, which is compared to historical AI application output from historical AI application output database 122. Based on the comparison and other information described above, service computing device 108 determines whether there are any indicators of an AI algorithm anomaly. For example, if the AI algorithm recognizes a human with a high confidence score on a highway where the probability of observing a human is very low, this is determined to be an indicator of an AI anomaly.

[0034] Depending on the analysis results of detecting one or more indicators of an AI anomaly by the service computing device 108, the level of diagnosis may be dynamically changed on the vehicle side, which may help reduce the computational load caused by the performance of diagnostic operations by the vehicle computing device 104. For example, when performing a first diagnostic level 140 (Level 1), the AI application output data 126 may be periodically transmitted from the vehicle computing device 104 to the service computing device 108, such as at period 142, and when Level 1 is in effect, no other actual diagnosis of the AI algorithm 128 is performed by the vehicle computing device 104, and the execution of the AI application 130 is not affected.

[0035] If the service computing device 108 detects minor indicators of an AI algorithm anomaly based at least in part on the received AI application output data 126, the service computing device 108 may instruct the vehicle computing device 104 to increase the applied diagnostic level to a second diagnostic level (Level 2). When the second diagnostic level 146 (Level 2) is enabled, periodic AI diagnostics 148 are performed on the AI algorithm 128 by the vehicle computing device 104, and the AI application output data 126 is also periodically transmitted to the service computing device 108. In some embodiments, the frequency of the diagnostics is determined at least in part based on the availability of computational resources of the vehicle computing device 104.

[0036] Additionally, if the service computing device 108 detects significant indicators of an AI algorithm anomaly in the AI application output data 126, the service computing device 108 may instruct the vehicle computing device 104 to change the diagnostic level to a third diagnostic level (Level 3). Once the diagnostic level is set to the third diagnostic level 150 (Level 3), dedicated diagnostics (referred to herein as cause identification and recovery actions 152) may be performed. By way of example, the cause identification and recovery actions 152 may include rebooting the associated vehicle computing device 104 and performing other diagnostic procedures, such as checking for viruses and malware, permanent memory failures, etc. In some embodiments, the vehicle may be pulled over to the side of the road to perform the cause identification and recovery actions 152, may proceed to a maintenance facility, etc. Once the cause identification and recovery actions 152 are complete, the vehicle computing device 104 may again run the AI application 130 and may periodically send the AI application output data 126 to the service computing device 108 to check whether the AI algorithm anomaly has been resolved. If the anomaly remains after multiple iterations, the vehicle may stop, continue at a slow speed, go to a maintenance facility, etc., depending on the severity of the anomaly.

[0037] In addition to detecting an indicator of an AI anomaly (normal, minor, or significant), the diagnostic level may also be changed based on other considerations. As one example, vehicle control information may indicate unexpected behavior, such as hard braking, sudden acceleration, or unwanted lane changes. As another example, if the total amount of time used by the vehicle computing device 104 to perform periodic AI diagnostics 148 exceeds a threshold time, this may also be an indicator that the level should be increased to Level 3.

[0038] 2 illustrates an example hardware configuration of the system 100 shown in FIG. 1 , according to some embodiments. For example, each vehicle computing device 104 may include one or more processors 216, one or more computer-readable media 218, one or more communication interfaces (I / Fs) 220, and, in some embodiments, one or more vehicle human-machine interfaces (HMIs) 222. In some cases, the vehicle computing device 104 may include one or more ECUs (electronic control units) or any of various other types of computing devices. For example, the vehicle computing device 104 may include one or more ADAS / AD ECUs that control at least a portion of the vehicle systems 114, such as performing ADAS and / or AD tasks, such as navigation, braking, steering, acceleration, and deceleration. The vehicle computing device 104 may also include one or more other ECUs, such as those that control other systems of the vehicle systems 114, all of which collectively may be included in the vehicle computing device 104 described herein.

[0039] An ECU is a generic term for any embedded processing system that controls one or more of a vehicle's systems, subsystems, or components. Software such as vehicle AI diagnostic program 224, vehicle control program 228, and / or AI application 130 may be executed by one or more ECUs and stored in a portion of computer-readable media 218 (e.g., program ROM, solid-state storage, etc., as described below) associated with each ECU, enabling the ECU to operate as an embedded system. ECUs may typically communicate with each other over a vehicle bus, such as the CAN bus described above, according to a vehicle bus protocol.

[0040] Each ECU or other vehicle computing device 104 may include one or more processors 216, which may include one or more of a central processing unit (CPU), graphics processing unit (GPU), microprocessor, microcomputer, microcontroller, digital signal processor, state machine, logic circuit, system-on-chip processor, and / or any device that manipulates signals based on operational instructions. By way of example, processor 216 may include one or more hardware processors and / or logic circuits of any suitable type that are specifically programmed or configured to execute the algorithms and other processes described herein. Processor 216 may be configured to fetch and execute computer-readable instructions (computer-executable instructions) stored on computer-readable medium 218 and programmed to perform the functions described herein.

[0041] The computer-readable medium 218 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storing information such as computer-readable instructions, data structures, programs, program modules, and other code or data. For example, the computer-readable medium 218 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid-state storage, magnetic disks, cloud storage, or any other medium that can be used to store desired information and that can be accessed by a computing device. Depending on the configuration of the vehicle computing device 104, the computer-readable medium 218 may be tangible, non-transitory media, insofar as it is a non-transitory computer-readable medium, when referred to, does not include media such as energy, carrier signals, electromagnetic waves, and / or the signals themselves. In some cases, the computer-readable medium 218 may be co-located with the vehicle computing device 104; in other examples, portions of the computer-readable medium 218 may be remote from the vehicle computing device 104.

[0042] The computer-readable medium 218 may be used to store any functional components executable by the processor 216. In many embodiments, these functional components include instructions or programs executable by the processor 216 that, when executed, cause the processor 216 to perform actions such that the vehicle computing device 104 performs the functions described herein. The functional components stored on the computer-readable medium 218 may include one or more of a vehicle AI diagnostic program 224, a vehicle control program 228, and an AI application 130, each of which may include one or more computer programs, applications, executable code, or portions thereof. Furthermore, while these programs and applications are illustrated together in this example, during use, some or all of these programs and applications may execute on separate vehicle computing devices 104. Alternatively, in some embodiments, these programs and applications 130, 224, and 228 may each be part of a single program or application. For example, in some cases, the vehicle control program 228 may also be an AI application 130 that uses one or more AI algorithms 128 to determine at least one value.

[0043] Additionally, the computer-readable medium 218 may store data, data structures, machine learning models and other AI algorithms 128, as well as other information used to implement the functions and services described herein. For example, the computer-readable medium 218 may store the AI algorithms 128, which may be one or more machine learning models. Examples of AI algorithms 128 that may be used in some embodiments herein may include artificial neural networks, such as self-organizing neural networks, recurrent neural networks, convolutional neural networks, modular neural networks, and deep learning neural networks, as well as predictive models, decision trees, classifiers, regression models, such as linear regression models, support vector machines, probabilistic models, such as Markov models and hidden Markov models, and the like. Embodiments herein are not limited to any particular type of AI algorithm.

[0044] Additionally, the computer-readable medium 218 may store sensor data 226 received from on-board sensors 112, which may be used by the AI application 130 to determine AI application output data 126, which may then be used by the vehicle control program 228 to control one or more vehicle systems 114, subsystems, components, etc. Furthermore, while the data, data structures, and AI algorithms 128 are illustrated together in this example, during use, some or all of these elements may be stored by a separate vehicle computing device 104. The vehicle computing device 104 may also include or hold other functional components and data, which may include programs, drivers, etc., and data used or generated by the functional components. Furthermore, the vehicle computing device 104 may include many other logical, programmatic, and physical components, of which the foregoing are merely examples relevant to the discussion herein.

[0045] The one or more communications interfaces 220 may include one or more software and hardware components that enable communication with various other devices, such as through a CAN bus and / or through one or more networks 106. For example, the communications interface 220 may enable communication through one or more of a LAN, the Internet, a cable network, a cellular network, a wireless network (e.g., Wi-Fi) and a wired network (e.g., CAN, Fiber Channel, Fiber Optic, Ethernet), a direct connection, and a near-field communication such as BLUETOOTH®, etc., as additionally listed elsewhere herein.

[0046] The sensor data 226 may include sensor data received from the onboard sensors 112. For example, the onboard sensors 112 may include any of several different types of sensors, such as a camera system, radar, LIDAR, ultrasound, a satellite positioning system receiver such as a Global Navigation Satellite System (GNSS) receiver (hereinafter referred to by the generic name “GPS,” which is also intended to encompass any other satellite positioning system), an accelerometer, a compass, etc. Additionally, the sensor data 226 used by the vehicle control program 228 may include information received from or associated with various vehicle systems 114, such as, for example, a suspension controller associated with a suspension system, a steering controller associated with a steering system, a vehicle speed controller associated with a braking and acceleration system (not shown in FIG. 1 ), etc.

[0047] For example, vehicle control program 228 may use rule-based and / or artificial intelligence-based control algorithms (including one or more of AI algorithms 128) to determine parameters for vehicle control, such as based on the output (e.g., perception output) of AI application 130. For example, vehicle control program 228 may determine an appropriate action, such as braking, steering, or accelerating, and may send one or more control signals to one or more vehicle systems 114 based on the determined action. For example, vehicle control program 228 may send control signals to a suspension controller, a steering controller, and / or a vehicle speed controller to control or partially control the vehicle in some applications.

[0048] The service computing device 108 may include one or more servers or other types of computing devices that may be embodied in any manner. For example, in the case of a server, the programs, other functional components, and data may be implemented on a single server, a cluster of servers, a server farm or data center, a cloud-hosted computing device, etc., although other computer architectures may additionally or alternatively be used.

[0049] Furthermore, while the figures illustrate the functional components and data of the service computing device 108 as residing in a single location, these functional components and data may alternatively be distributed across different computing devices and different locations in any manner. Consequently, functionality may be implemented by one or more service computing devices 108, and the various functions described herein may be distributed in various ways across different computing devices. Multiple service computing devices 108 may be located together or separately and organized, for example, as a virtual server, a server bank, and / or a server farm. The described functionality may be provided by the servers of a single entity or company, or by the servers and / or services of multiple different entities or companies.

[0050] In the illustrated example, each service computing device 108 may include one or more processors 240, one or more computer-readable media 242, and one or more communication interfaces 244. Each processor 240 may be a single processing unit or multiple processing units and may include single or multiple computing units or multiple processing cores. Processor 240 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, system-on-chip processors, state machines, logic circuits, and / or any device that manipulates signals based on operational instructions. For example, processor 240 may be one or more hardware processors and / or logic circuits of any suitable type specifically programmed or configured to execute the algorithms and processes described herein. Processor 240 may be configured to fetch and execute computer-readable instructions stored on computer-readable media 242 and programmed to perform the functions described herein.

[0051] The computer-readable medium 242 may include volatile and nonvolatile memory and / or removable and non-removable media implemented in any type of technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Such computer-readable medium 242 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, optical storage, solid-state storage, magnetic tape, magnetic disk storage, storage arrays, network-attached storage, storage area networks, cloud storage, or any other medium that can be used to store desired information and that can be accessed by a computing device. Depending on the configuration of the service computing device 108, the computer-readable medium 242 may be any type of computer-readable medium and / or may be tangible non-transitory medium, insofar as, as referred to herein, it is a non-transitory computer-readable medium that does not include media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0052] The computer-readable medium 242 may be used to store any functional components executable by the processor(s) 240. In many embodiments, these functional components include instructions or programs executable by the processor(s) 240 that, when executed, cause one or more processors 240 to perform actions such that the functionality described herein resides in the service computing device 108. The functional components stored on the computer-readable medium 242 may include a server AI diagnostic program 246 that may be executed to detect indicators of AI algorithm anomalies based on receiving and analyzing the vehicle ID, vehicle location, and AI application output data 126, and to provide instructions to the diagnostic level 136 based on the analysis.

[0053] In addition, computer-readable medium 242 may store data used to perform the operations described herein. Thus, computer-readable medium 242 may include sensor configuration database 120 and historical AI application output database 122, as described above. Additionally, computer-readable medium 242 may, at least temporarily, store AI application output data 126. Additionally, if no anomaly is detected, server AI diagnostic program 246 may add received AI application output data 126 to historical AI application output database 122 for the corresponding road segment (or other location data) and sensor configuration. However, if an anomaly is detected, received AI application output data 126 is not added to historical AI application output database 122.

[0054] Further, the service computing device 108 may also include or retain other functional components and data not specifically shown in Figure 2, which may include programs, drivers, etc., and data used or generated by the functional components. In addition, the service computing device 108 may include many other logical, programmatic, and physical components, of which the foregoing are merely exemplary in connection with the description herein.

[0055] Communications interface 244 may include one or more interfaces and hardware components that allow for communication with various other devices, such as through network 106. For example, communications interface 244 may allow for communication through one or more of the Internet, cable networks, cellular networks, wireless networks (e.g., Wi-Fi) and wired networks (e.g., fiber optics and Ethernet), and short-range wireless communications such as BLUETOOTH®, BLUETOOTH® low energy, etc., as additionally listed elsewhere herein.

[0056] Figure 3 illustrates an example of diagnostic levels 300, according to some embodiments. In the example of Figure 3, a first diagnostic level 302 (Level 1) corresponds only to sending AI application output data to the service computing device 108, a second diagnostic level 304 (Level 2) further includes performing limited diagnostics on the AI algorithms (e.g., periodically) in addition to periodically sending the AI application output data to the service computing device 108, and a third diagnostic level 306 (Level 3) includes performing dedicated diagnostics on the AI algorithms, such as by identifying causes of anomalies and implementing recovery.

[0057] Additionally, as shown at 308, if an indicator of an AI anomaly classified as a minor indicator is detected, the vehicle computing device 104 may be instructed to operate at the second diagnostic level 304 (Level 2), i.e., to perform periodic, limited diagnostics on the AI algorithm. Additionally, as shown at 310, if there has been no recent detection of an AI anomaly, the vehicle computing device 104 may be instructed to return to the first diagnostic level 302 (Level 1) and no longer perform any periodic diagnostics of the AI algorithm. Alternatively, as shown at 312, if the vehicle computing device 104 is currently operating at the second diagnostic level 304 (Level 2) and detects an indicator of abnormal vehicle behavior, such as hard braking or abrupt lane changes, classified as an AI anomaly or a significant indicator of an AI anomaly, the vehicle computing device 104 may increase the level of diagnostics to the third diagnostic level 306 (Level 3) to perform dedicated diagnostics of the corresponding AI algorithm, such as identifying the cause of the anomaly and implementing recovery.

[0058] Alternatively, as shown at 314, if the service computing device 108 detects an indicator of an AI anomaly that is classified as a significant indicator, the service computing device 108 (i.e., the server AI diagnostic program 246) may cause the vehicle computing device 104 to transition the diagnostic level for the vehicle computing device 104 to a third diagnostic level 306 (Level 3) to initiate dedicated diagnostics of the AI algorithm to identify the cause of the AI anomaly and restore normal operation of the AI algorithm. Further, as shown at 316, once cause identification and restoration are complete, the vehicle computing device 104 may return to the first diagnostic level 302 (Level 1) to periodically send AI application output data 126 to the service computing device 108.

[0059] FIG. 4 is a flow diagram illustrating an example of a process 400 for performing diagnostics on an AI algorithm, according to some embodiments. The processes described herein are illustrated in logical flow diagrams as a collection of blocks representing sequences of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In the case of software, the blocks may represent computer-readable instructions stored on one or more computer-readable media that are programmed to perform the recited operations and executed by one or more processors. Generally, computer-readable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the blocks are described should not be construed as limiting. Any number of the described blocks may be combined in any order and / or in parallel to implement a process, or alternative processes, and not all blocks need be executed. For illustrative purposes, the processes are described with reference to the environments, systems, and devices described in the examples herein, although the processes may be implemented in a wide variety of other environments, systems, and devices.

[0060] In some embodiments, process 400 may be performed by system 100 described above. For example, in the illustrated example, process 400 may be performed by service computing device 108, such as by execution of server AI diagnostic program 246. Alternatively, vehicle computing device 104 may perform some or all of process 400, in some cases, if vehicle computing device 104 has access to sufficient database information and sufficient processing power.

[0061] In the example process 400 of FIG. 4, one or more indicators of an anomaly in an AI algorithm may be detected based on the output of an AI application. Process 400 illustrates a server-side process for detecting indicators of an AI anomaly for a single AI algorithm, for a single sensor in some embodiments, or for multiple sensors in other embodiments. As described above, first, road segments covered by the sensors are identified. In addition to historical AI output information, historical AI application output database 122 may include names and locations of stationary objects, such as landmarks, traffic lights, stop signs, buildings, geographic features, utility poles, and trees. In some cases, the process of detecting AI anomalies may be divided into detecting moving and stationary objects. For example, for moving objects such as humans or vehicles, if the current AI application output includes a human with a high confidence score and the actual presence of a human is unlikely based on historical data, this may be classified as a minor indicator of an AI anomaly, and periodic diagnostics (Level 2) may be performed. Thus, in some embodiments, a discrepancy detected regarding a moving object may be classified as a minor indication of an AI anomaly.

[0062] Additionally, for stationary objects such as traffic lights, if the current AI application output does not include the traffic light that should have been detected and there are no obstacles, this may be classified as a significant indicator of an AI anomaly, and dedicated diagnostics (Level 3) may be performed. Stationary object information (object name (e.g., traffic light), longitude, latitude, height) stored in historical AI application output database 122 may also be used to detect significant indicators of an AI anomaly, for example, based on the difference between the actual height of the object and the height estimated by the AI application. Furthermore, if there is a failure to detect the expected number of landmarks or other stationary objects at the current GPS location, taking into account the sensor configuration when there are no obstacles, this may also be classified as a significant indicator of an AI anomaly, and dedicated diagnostics (Level 3) may be performed. Thus, in some embodiments, inconsistencies regarding stationary objects as well as unexpected vehicle behavior may be classified as significant indicators of an AI anomaly.

[0063] At step 402, the service computing device 108 may receive vehicle location data, vehicle ID, and past and present AI application outputs from the vehicle computing device 104 for use in performing diagnostics to detect any indicators of AI algorithm anomalies. As described above, for example, with reference to FIG. 1, past AI application outputs may be obtained by accessing the historical AI application output database 122.

[0064] At step 404, the service computing device 108 may receive sensor configuration information for the vehicle. For example, based on the vehicle ID received at step 402, the service computing device 108 may access the sensor configuration database 120 to obtain the sensor configuration for the particular vehicle. As one example, the sensor configuration may have already been provided to the service computing device 108 by the vehicle computing device 104, such as when registering with an AI anomaly monitoring service provided by the service computing device 108. As another example, the sensor configuration may be a standard sensor configuration for the make and model of the vehicle corresponding to the vehicle ID received at step 402. Other variations will be apparent to those skilled in the art having the benefit of this disclosure.

[0065] At step 406, the service computing device 108 may determine road segments covered by the vehicle sensors based on at least the vehicle position data and sensor configuration data for the vehicle. Additionally or alternatively, rather than determining road segments, such as in examples herein directed to other types of mobile devices other than ground vehicles, the AI application output may be determined based on the received position data and stored in association with corresponding location information, such as longitude and latitude coordinates (and possibly altitude / depth).

[0066] In step 408, the service computing device 108 may determine whether the vehicle AI system to be monitored is currently operating. If so, the process proceeds to step 410. If not, the process returns to step 402 to await the next receipt of vehicle information. As an example, the service computing device 108 may send a communication to the vehicle computing device 104 to determine whether the AI system is currently operating.

[0067] At step 410, based on a determination that at least the vehicle AI system is operational, the service computing device 108 may send instructions to the vehicle computing device 104 to periodically, i.e., in accordance with the Level 1 diagnostics described above with respect to Figures 1-3, send AI application output to the service computing device 108.

[0068] In step 412, the service computing device 108 may determine whether there are any discrepancies between past AI model outputs and current AI application outputs for stationary objects on the currently considered road segment (or, in other embodiments, currently determined locations). If there are discrepancies, this may be classified as a significant indicator of an AI anomaly, and the process proceeds to step 420. If there are no discrepancies, the process proceeds to step 414.

[0069] At step 414, the service computing device 108 may determine whether there are any discrepancies between the past and current AI application outputs for the moving object associated with the currently considered road segment (or other currently determined location). If there are discrepancies, this may be classified as a minor indicator of an AI anomaly, and the process proceeds to step 416. If there are no discrepancies, the process returns to step 406 to select the next road segment (or next location) for consideration of the corresponding past and current AI application outputs.

[0070] In step 416, based on the determination of at least minor indicators of AI algorithm anomaly, the service computing device 108 may send instructions to the vehicle computing device 104 instructing the vehicle computing device 104 to perform periodic diagnostics (Level 2).

[0071] In step 418, the service computing device 108 may determine whether an AI algorithm anomaly is detected by the level 2 periodic diagnostics. If detected, the process proceeds to step 420. If not detected, after a threshold number of iterations, the process may return to step 406.

[0072] At step 420, based on detecting an AI algorithm anomaly via the Level 2 periodic diagnostics at step 418 or based on determining that there is a discrepancy between past and current AI application output for stationary objects, as determined at step 412, the service computing device 108 may send instructions to the vehicle computing device 104 to perform dedicated diagnostics according to Level 3. In some embodiments, sending the instructions may include at least one of notifying a vehicle occupant of the AI algorithm anomaly and recommending immediate service of the vehicle, sending a notification to a device associated with maintenance on the vehicle, changing the vehicle's operating speed or stopping the vehicle on the side of the road to perform a process to identify the cause of the AI anomaly and implement a recovery, or directing the vehicle to a maintenance location to identify the cause of the anomaly and implement a recovery.

[0073] At step 422, the service computing device 108 may receive notification of the completion of the level 3 diagnosis, including cause identification and / or recovery of the AI algorithm. For example, in response to receiving the instruction from the service computing device 108 at step 420, the vehicle computing device 104 may perform level 3 diagnosis on the AI algorithm, such as to determine the cause of the anomaly and / or to implement recovery, such as by rebooting the computing device executing the AI algorithm. In some examples, if the vehicle is currently operating on the road, the vehicle control program 228 may cause the vehicle to slow down, stop, or visit a maintenance facility, as described above, to implement cause identification diagnosis and recovery. Once the level 3 diagnosis is complete, the vehicle computing device 104 may periodically transmit AI application output and current vehicle position data to the service computing device 108, allowing the service computing device 108 to continue monitoring the AI systems on the vehicle. Numerous other variations will be apparent to those skilled in the art having the benefit of this disclosure.

[0074] 5 illustrates an example 500 of identifying an AI algorithm that is likely to have an AI anomaly from among multiple AI algorithms, according to some embodiments. In this example, assume there are three AI algorithms: a first AI algorithm 502 that receives sensor data from a first sensor 504 (sensor 1), a second AI algorithm 506 that receives sensor data from a second sensor 508 (sensor 2), and a third AI algorithm 510 that performs sensor fusion on the results of the first sensor 504 (sensor 1) and the second sensor 508 (sensor 2), as shown at 512.

[0075] Additionally, in this example, the sensor detection areas of the first sensor 504 (sensor 1) and the second sensor 508 (sensor 2) overlap, as shown at 514. For example, the first sensor 504 (sensor 1) may have a wide-angle field of view, and the second sensor 508 (sensor 2) may have a narrower, long-range field of view. Because the sensor detection areas (i.e., fields of view) overlap, as shown at 514, the first AI algorithm 502 and the second AI algorithm 506 may detect the same object within their respective fields of view, such as a human 516.

[0076] Additionally, when identifying a malfunctioning AI algorithm from among the multiple AI algorithms 502, 506, and 510, embodiments herein may consider both short-term and long-term perspectives. From a short-term perspective, if all three AI algorithms show high confidence scores and the names of the identified objects are the same (e.g., "human"), the service computing device 108 (not shown in FIG. 5) may determine that no indicators of an AI anomaly exist. In particular, as shown at 518, in a first case, assume that the first AI algorithm 502 identifies a human with 80% confidence, the second AI algorithm 506 identifies a human with 70% confidence, and the third AI algorithm 510 identifies a human with 85% confidence. Therefore, the results of these AI algorithm outputs indicate that no AI anomaly is detected among the multiple AI algorithms 502, 506, and 510.

[0077] On the other hand, as shown at 520, in a second case, assume that the second AI algorithm 506 exhibits a relatively low confidence score (e.g., 20 percent) for identifying a human (or a different object name), the first AI algorithm 502 identifies the human with 70 percent confidence, and the third AI algorithm 510 that performs sensor fusion identifies the human with 60 percent confidence. In this example, the service computing device may determine that the results of the AI algorithm output include indicators of an AI anomaly. Furthermore, from a short-term perspective, the second AI algorithm 506 may appear to be an algorithm that includes an AI anomaly.

[0078] To keep track of AI anomalies over the long term, a data structure 522 may be maintained that includes an ID 524 for each AI algorithm and an anomaly counter 526 for each listed AI algorithm. Thus, over the long term, if the output of any AI algorithm is inconsistent with the outputs of other AI algorithms, the value of the anomaly counter 526 may be incremented for that AI algorithm. Subsequently, if the value of the anomaly counter 526 meets a threshold, the corresponding AI algorithm may be flagged as having an indication of an AI anomaly. Based on such an indication of an AI anomaly for each AI algorithm, the diagnostic computing load can be reduced by increasing the diagnostic level only for that AI algorithm whose anomaly counter 526 has reached its threshold limit. Consequently, in the second case, the anomaly counter 526 for the second AI algorithm 506 may be incremented from "410" to "411," but the value of the anomaly counter 526 for the second AI algorithm 506 remains below the anomaly indicator threshold, while the anomaly counter 526 for the first AI algorithm 502 has already exceeded the threshold and may be subject to, for example, level 2 periodic diagnostics.

[0079] 6 illustrates an example 600 of reducing the number of false positives for detecting AI anomalies, according to some embodiments. As an example, to reduce the number of false positives in detected indicators of AI anomalies, the diagnostic level is not changed for some exceptions that may occur in special scenarios. For example, as shown in 602, when the vehicle 102 enters an area under construction, the ongoing construction often causes the current AI algorithm output to differ from past AI outputs, which may lead to a false indication of an AI anomaly.

[0080] In this example, the vehicle 102 detects one or more traffic cones 604 as it passes through a particular road segment indicating a construction zone. Accordingly, in this situation, the service computing device 108 may determine that the vehicle has entered a construction zone and may stop storing AI application output data received from the vehicle 102. For example, such a special scenario may be treated as an exception, and the diagnostic level for the corresponding AI is not changed, even though there may be a significant difference between the current recognition result and the recognition result stored in the historical AI application output database 122 for the same road segment. Examples of when such an exception applies may include when the AI application output data includes the recognition of one or more traffic cones, construction barricades, construction equipment, etc. indicating a construction zone, as well as various other special scenarios, such as the detection of the presence of an ambulance, fire engine, police vehicle, etc. Numerous other variations will be apparent to those skilled in the art having the benefit of this disclosure.

[0081] 7 shows an example 700 of determining whether to add AI application output data 126 to the historical AI application output database 122, according to some embodiments. In this example, the historical AI application output database 122 is populated based on a comparison to a confidence score threshold. When adding new data to the historical AI application output database 122, the AI application output data 126 received from each vehicle 102 may be stored for each road segment traversed by each vehicle 102 using an AI algorithm monitoring service provided by the service computing device 108.

[0082] Generally, if the confidence score of an AI algorithm output is relatively low, the output is more likely to be substantially incorrect. Therefore, in some embodiments herein, a threshold for a minimum confidence score (confidence score threshold) is set, and only AI output data having a confidence score above the confidence score threshold is stored in the historical AI application output database 122. Setting the confidence score threshold improves the accuracy of detecting indicators of AI anomalies and reduces the number of false positives.

[0083] In the example of FIG. 7 , the vehicle 102 may be able to recognize a human 702 on a road 704. As shown at 706, in a first case, assume that the recognized object name is a human and the confidence score provided by the AI algorithm is 85 percent. Further, assume that the confidence score threshold for adding data to the historical AI application output database 122 is 30 percent. Thus, in the first case, the AI application output data 126 is added to the historical AI application output database 122 for the current road segment (or location) of the vehicle 102. On the other hand, as shown at 708, in a second case, assume that the AI application output object name is a human and the AI algorithm's confidence score is 29 percent. Because the confidence score of 29 percent is lower than the confidence score threshold of 30 percent, the AI application output data 126 is not added to the historical AI application output database 122. Further, although a confidence score threshold of 30 percent is used in this example, in other embodiments, the confidence score threshold may vary depending on the particular usage scenario, system configuration, etc.

[0084] The example processes described herein are provided for illustrative purposes only. Numerous other variations will be apparent to those skilled in the art in light of the disclosure herein. Furthermore, while the disclosure herein describes some examples of suitable frameworks, architectures, and environments for implementing the processes, the examples herein are not limited to the specific examples shown and described. Furthermore, the disclosure provides various example examples as described and illustrated in the drawings. However, the disclosure is not limited to the examples described and illustrated herein, but may extend to other examples known or that will become known to those skilled in the art.

[0085] The various instructions, processes, and methods described herein may be executed by processors herein as computer-readable instructions, such as computer programs and applications stored on a computer-readable medium. In general, the terms program and application may be used interchangeably and may include instructions, routines, modules, objects, components, data structures, executable code, etc. for performing particular tasks or implementing particular data types. These programs, applications, etc. may be executed as native code or may be downloaded and executed, such as in a virtual machine or other runtime compilation execution environment. In general, the functionality of the programs and applications may be combined or distributed as desired in various embodiments. These program, application, and method embodiments may be stored on a computer-readable medium or transmitted across some form of communication medium.

[0086] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Instead, the specific features and acts are disclosed as example forms of implementing the claims. [Explanation of symbols]

[0087] 104...Vehicle computing device 106…Network 108...Service Computing Device 112...In-vehicle sensor 114...Vehicle Systems 120...Sensor configuration DB 122...History AI application output DB 126…AI application output data 128…AI algorithm 130…AI Applications 216...processor 218...Computer-readable medium 220...Communication I / F 222...HMI (Human Machine Interface) for vehicles 224...Vehicle AI diagnostic program 226...Sensor data 228...Vehicle control program 240...processor 242...Computer-readable medium 244...Communication I / F 246...Server AI diagnostic program 504...Sensor 1 508...Sensor 2 510…Sensor fusion AI 524…ID of AI algorithm 526...Abnormal counter

Claims

1. A system for detecting an abnormality in an artificial intelligence (AI) system installed in a vehicle, one or more processors configured to perform processes according to executable instructions, said processes comprising: receiving at least vehicle position data and AI output data from the vehicle; determining a road segment corresponding to the vehicle position data; accessing, based on at least the road segment, historical AI output data stored in a data structure and associated with the road segment; comparing the received AI output data with the past AI output data stored in the data structure for the road segment; transmitting instructions to the vehicle that cause the vehicle to perform at least one action based at least on a comparison of the received AI output data and the past AI output data associated with the road segment; and Including, system.

2. 10. The system of claim 1, The process receiving a vehicle identifier from the vehicle; determining a sensor configuration for the vehicle based at least on the received vehicle identifier; taking into account the sensor configuration for the vehicle in the comparison of the received AI output data with the past AI output data for the road segment. system.

3. 10. The system of claim 1, comparing the received AI output data with the past AI output data includes detecting at least one difference between a moving object identified in the received AI output data and a moving object identified in the past AI output data. system.

4. 4. The system of claim 3, and wherein the instructions transmitted to the vehicle based at least on detecting the at least one difference include instructions for the vehicle to perform periodic diagnostics of an AI algorithm corresponding to the AI output data to check for AI algorithm anomalies. system.

5. 10. The system of claim 1, comparing the received AI output data with the past AI output data includes detecting at least one difference between a stationary object identified in the received AI output data and a stationary object identified in the past AI output data. system.

6. 6. The system of claim 5, and wherein the instructions sent to the vehicle based at least on detecting the at least one difference include instructions for the vehicle to perform at least one of a cause identification or recovery process for an AI algorithm corresponding to the AI output data to check for an AI algorithm anomaly. system.

7. 10. The system of claim 1, comparing the received AI output data with the past AI output data includes not identifying indicators of AI algorithm anomalies; and wherein the instructions transmitted to the vehicle based at least on not identifying the indicator of the AI algorithm anomaly include instructions to cause the vehicle to periodically transmit additional AI output data to continue monitoring the AI algorithm anomaly. system.

8. 10. The system of claim 1, The process comparing a confidence score associated with the received AI output data to a confidence score threshold; storing the received AI output data in the data structure associated with the road segment based at least on the confidence score associated with the received AI output data that meets the confidence score threshold; and Further comprising: system.

9. 1. A method for detecting an anomaly in an artificial intelligence (AI) system on a mobile device, comprising: receiving, by one or more processors, at least mobile device position data and AI output data from the mobile device; determining a location corresponding to the mobile device location data; accessing, based at least on the location, past AI output data stored in a data structure and associated with the location; comparing the received AI output data with the past AI output data stored in the data structure for the location; transmitting instructions to the mobile device that cause the mobile device to perform at least one action based at least on a comparison of the received AI output data and the past AI output data associated with the location; method.

10. 10. The method of claim 9, further comprising: receiving a mobile device identifier from the mobile device; determining a sensor configuration for the mobile device based at least on the received mobile device identifier; taking into account the sensor configuration for the mobile device when comparing the received AI output data with the past AI output data for the location. method.

11. 10. The method of claim 9, comparing the received AI output data with the past AI output data includes detecting at least one difference between a moving object identified in the received AI output data and a moving object identified in the past AI output data. method.

12. 12. The method of claim 11, and wherein the instructions transmitted to the mobile device based at least on detecting the at least one difference include instructions for the mobile device to perform periodic diagnostics of an AI algorithm corresponding to the AI output data to check for AI algorithm anomalies. method.

13. 10. The method of claim 9, comparing the received AI output data with the past AI output data includes detecting at least one difference between a stationary object identified in the received AI output data and a stationary object identified in the past AI output data. method.

14. 14. The method of claim 13, and wherein the instructions transmitted to the mobile device based at least on detecting the at least one difference include instructions for the mobile device to perform at least one of a cause identification or recovery process for an AI algorithm corresponding to the AI output data to check for AI algorithm anomalies. method.

15. 10. The method of claim 9, comparing the received AI output data with the past AI output data includes not identifying indicators of AI algorithm anomalies; and wherein the instructions transmitted to the mobile device based at least on not identifying an indicator of the AI algorithm anomaly include instructions for causing the mobile device to periodically transmit additional AI output data to continue monitoring the AI algorithm anomaly. method.

16. 10. The method of claim 9, further comprising: comparing the confidence score associated with the received AI output data to a confidence score threshold; storing the received AI output data in the data structure associated with the location based at least on the confidence score associated with the received AI output data that satisfies the confidence score threshold; method.

17. One or more non-transitory computer-readable media storing instructions executable by one or more processors to cause the one or more processors to perform a process, the process comprising: receiving at least mobile device location data and artificial intelligence (AI) output data from the mobile device; determining a location corresponding to the mobile device location data; accessing, based at least on the location, past AI output data stored in a data structure and associated with the location; comparing the received AI output data with the past AI output data stored in the data structure for the location; transmitting instructions to the mobile device that cause the mobile device to perform at least one action based at least on a comparison of the received AI output data and the past AI output data associated with the location; Including, Non-transitory computer-readable medium.

18. 20. One or more non-transitory computer-readable media as recited in claim 17, The process receiving a mobile device identifier from the mobile device; determining a sensor configuration for the mobile device based at least on the received mobile device identifier; taking into account the sensor configuration for the mobile device when comparing the received AI output data with the past AI output data for the location. Non-transitory computer-readable medium.

19. 20. One or more non-transitory computer-readable media as recited in claim 17, comparing the received AI output data with the past AI output data includes detecting at least one difference between a moving object identified in the received AI output data and a moving object identified in the past AI output data; and wherein the instructions transmitted to the mobile device based at least on detecting the at least one difference include instructions for the mobile device to perform periodic diagnostics of an AI algorithm corresponding to the AI output data to check for AI algorithm anomalies. Non-transitory computer-readable medium.

20. 20. One or more non-transitory computer-readable media as recited in claim 17, comparing the received AI output data with the past AI output data includes detecting at least one difference between a stationary object identified in the received AI output data and a stationary object identified in the past AI output data; and wherein the instructions transmitted to the mobile device based at least on detecting the at least one difference include instructions for the mobile device to perform at least one of a cause identification or recovery process for an AI algorithm corresponding to the AI output data to check for AI algorithm anomalies. Non-transitory computer-readable medium.

Images