A system, method, and non-temporary computer-readable medium for detecting anomalies in artificial intelligence systems.

Abstract

To provide a system, a method and a non-temporary computer readable medium for detecting an abnormality that may occur in an AI algorithm.SOLUTION: A system for detecting an abnormality of an AI system, receives at least mobile device position data and artificial intelligence (AI) output data from a mobile device, determines a position corresponding to the mobile device position data, accesses past AI output data stored in a data structure and associated with the position on the basis of at least the position, compares the received AI output data with the past AI output data stored in the data structure with resect to the position, and transmits a command to allow the mobile device to execute at least one action to the mobile device on the basis of at least the comparison between the received AI output data and the past AI output data associated with the position.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a system, method, and non - transient computer - readable medium for detecting anomalies in an artificial intelligence system.

Background Art

[0002] Semi - autonomous vehicle systems or other advanced driver assistance systems (ADAS), and autonomous driving (AD) systems or other autonomous driving systems are systems that automate vehicle control or otherwise improve it to enhance safety, autonomous navigation, etc. Such systems may use an embedded vehicle computing device called an "electronic control unit" (ECU), which may include an AD / ADAS ECU, a power train ECU, a chassis ECU, etc. For example, an AD / ADAS ECU may be configured to perform domain - specific functions such as perception, positioning, navigation, vehicle control, etc. To implement these functions, an artificial intelligence (AI) algorithm (e.g., a machine learning model) may be implemented in the AD / ADAS ECU or other vehicle ECUs.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] AI algorithms may include deep neural networks (DNNs) and / or various other types of artificial neural networks and / or various other types of machine learning models. For example, AI algorithms may be used in the ECUs of AD / ADAS vehicles to achieve a highly accurate and safe autonomous driving experience. For example, AI applications may be used in various roles in object detection and mission-critical control, such as for emergency braking, lane changes, and evasive maneuvers. Malfunctions of AI algorithms on which a vehicle depends can lead to serious accidents. Therefore, while the use of AI algorithms to enable vehicle AD / ADAS is expected to be beneficial, vehicle safety can be improved by ensuring that these systems operate properly and as intended.

[0005] This invention has been made in view of these problems and aims to provide a system, method, and non-temporary computer-readable medium for detecting anomalies that may occur in AI algorithms. [Means for solving the problem]

[0006] The present invention includes several means for solving at least some of the above-mentioned problems, one example being as follows: The system for detecting anomalies in an AI system receives at least mobile device position data and artificial intelligence (AI) output data from a mobile device, determines a position corresponding to the mobile device position data, accesses past AI output data stored in a data structure and associated with the position based at least on the position, compares the received AI output data with past AI output data stored in the data structure with respect to that position, and transmits a command to the mobile device to perform at least one action based at least on the comparison between the received AI output data and past AI output data associated with the position. [Effects of the Invention]

[0007] The present invention makes it possible to provide a system, method, and non-temporary computer-readable medium for detecting anomalies that may occur in AI algorithms.

[0008] Other issues, configurations, and effects not mentioned above will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the logical configuration of a system for efficiently monitoring AI algorithms in the embodiment. [Figure 2] This figure shows an example of the system's hardware configuration in the embodiment. [Figure 3] This figure shows an example of the diagnostic level in the embodiment. [Figure 4] This flowchart shows an example of a process for performing a diagnosis on the AI ​​algorithm in the embodiment. [Figure 5] This figure shows an example of identifying an AI algorithm that is highly likely to have an AI anomaly from among multiple AI algorithms in the embodiment. [Figure 6] This figure shows an example of reducing the number of false positives in order to detect AI anomalies in the examples. [Figure 7] This figure shows an example of how to determine whether or not to add the AI ​​application output data in the embodiment to the historical AI application output database. [Modes for carrying out the invention]

[0010] A detailed description of embodiments of the present invention will be given below with reference to the drawings. In the drawings, the leftmost digit of the reference number identifies the drawing in which the reference number first appears. Where the same reference number is used in different drawings, it refers to similar or identical items or features. [Examples]

[0011] Some embodiments of this specification relate to methods and apparatus configurations for detecting anomalies that may occur in AI algorithms, such as those used in vehicles and / or other types of mobile devices that may use AI algorithms (e.g., aerial drones or other types of aircraft, marine drones and other types of marine vehicles, underwater drones and other types of underwater vehicles, mobile autonomous robots and semi-autonomous robots), and those used by AI applications running against ADAS and / or AD systems. For example, an AI application may include an application that runs an AI algorithm to produce an output or result. Embodiments of this specification can detect indicators of AI algorithm anomalies for various AI embedded systems. Embodiments of this specification include detecting indicators of AI algorithm anomalies on the cloud side based on current and / or past AI application outputs, and further based on vehicle on-board sensor configuration information. Examples of AI application outputs include detected object identifiers, detected object locations, and corresponding confidence scores. For example, if a current AI application output does not match a historical AI application output at the same location, and the confidence level of the current AI application output is high, this mismatch may indicate the presence of an AI algorithm anomaly. For example, if the AI ​​application output identifies a person on a highway with a high level of confidence, even though the historical probability of a person being observed on a highway at that location is very low, it may be judged as an anomaly in the AI ​​algorithm.

[0012] In some embodiments, historical data of AI application output is stored in a database or other type of data structure for each road segment the vehicle traverses. Predictive decisions of the AI ​​application output may be performed for each road segment that the vehicle sensors can cover at the current vehicle position, and this may include taking into account the respective sensor configuration information of the current and past vehicles. As a result, the number of diagnostic processes performed on the vehicle can be reduced by dynamically changing the diagnostic level (e.g., corresponding to the frequency of diagnosis) according to the AI ​​algorithm's determination of anomaly indicators.

[0013] In addition, or by other means, in embodiments of this specification relating to other types of mobile devices, rather than storing historical AI application outputs in association with road segments, the AI ​​application outputs may be stored in association with corresponding location information, such as longitude and latitude coordinates (and optionally altitude / depth). Clustering methods or other thresholds may be applied to group together AI application outputs that are within a threshold distance of each other. When retrieving historical AI application outputs from a database or other data structure, similar thresholds may be applied to location data received from the mobile device to determine that historical AI application outputs that are sufficiently close to the mobile device's receiving location are associated with the mobile device's AI algorithm outputs.

[0014] Conventional methods for implementing AI algorithms generally cannot detect indicators of AI algorithm anomalies. On the other hand, the embodiments described herein can efficiently (i.e., with minimal computational overhead) detect various indicators of AI algorithm anomalies that may be caused by, for example, hardware problems, software problems, and / or security problems. Hardware problems may include, for example, memory location defects (permanent failures), sensor failures, network connectivity failures, and hardware intrusions. Furthermore, software problems may be caused by insufficient training of the AI ​​algorithm, bugs in the programming code, or security problems such as malicious attacks, all of which can lead to unexpected AI behavior. Therefore, the embodiments described herein are intended to ensure that the AI ​​algorithm behaves correctly during its operation.

[0015] During the design and testing phases of AI algorithms, the proper operation of the AI ​​algorithm is typically verified using a large amount of data as an AI algorithm validation dataset. However, the size of the AI ​​validation dataset is often too large to implement on a vehicle computing device. Furthermore, the available computing power of a vehicle computing device is usually limited, and verifying an AI algorithm using such limited computing resources cannot be achieved in a realistic amount of time. Therefore, the embodiments described herein are configured to verify an AI algorithm using the computing resources of a vehicle computing device efficiently. As a result, the number of computationally intensive diagnostic functions required to detect AI anomalies is substantially reduced by the embodiments described herein, and the reliability of the embedded AI system for AD / ADAS ECUs or other vehicle computing devices is improved.

[0016] The embodiments of this specification include an AI diagnosis method for diagnosing abnormalities of AI algorithms implemented in an AD / ADAS ECU or other vehicle computing devices. For example, the vehicle of this specification may include one or more machine learning models (MLMs: Machine Learning Model), such as artificial neural networks or other types of AI algorithms for implementing various functions. In some cases, the AI algorithm may be included in an AI application that is executed in an AD / ADAS ECU or other vehicle computing devices installed in the vehicle, or may be used by the AI application in another form.

[0017] Furthermore, some embodiments of this specification may include an AI diagnosis technology that can efficiently improve the reliability of vehicle AI algorithms and corresponding AI applications. And as described above, the AI diagnosis technology of this specification may be executed to efficiently detect AI algorithms that are operating inappropriately, such as those that may occur due to hardware defects, other hardware abnormalities, security violations, software errors, etc.

[0018] For example, due to hardware defects, the AI algorithm may begin to behave differently over time compared to when the AI algorithm is executed immediately after training and deployment. There are various factors, such as permanent hardware defects and / or hardware intrusion, that can affect the output of the AI algorithm and may cause serious accidents in some cases. Therefore, the embodiments of this specification can determine whether the behavior of the AI algorithm has changed and / or whether the AI algorithm is operating normally.

[0019] The AI diagnosis method in this specification does not require hardware redundancy (e.g., addition of other AI execution hardware) and does not depend on any specific hardware. The AI diagnosis method described in this specification is cost-effective and improves the safety of the vehicle by efficiently examining the operation of the AI algorithm. As an example, when a hardware abnormality occurs in an ECU system that may include an AD / ADAS ECU, a powertrain ECU, a chassis ECU, etc., the embodiments of this specification may efficiently identify the cause of the abnormality, such as which hardware element is related to the AI algorithm that is not operating properly. Consequently, the embodiments of this specification also improve the maintainability and repair efficiency of the ECU system.

[0020] For illustration, some exemplary embodiments are described in the context of an AD / ADAS ECU that executes one or more AI algorithms installed in a vehicle. However, the embodiments of this specification are not limited to the specific embodiments provided and may be extended to other types of computing devices, other types of vehicles, other types of AI algorithms, other types of hardware configurations, other types of local conditions, etc., as will be apparent to those skilled in the art in light of the disclosure of this specification. For example, the solution of this specification is scalable and may be applied to ubiquitous systems such as construction, agricultural, and mining equipment, ships and other marine vehicles in addition to ground vehicles. The embodiments of this specification may also be scaled to smaller application examples such as autonomous robots.

[0021] Figure 1 shows an example of the logical configuration of a system 100 for performing efficient monitoring of an AI algorithm in several embodiments. System 100 includes a vehicle 102, which is an example of a ground vehicle mobile device to which the method described herein may be applied. As described above, in other examples, the method and apparatus configuration described herein may also be applied to other types of mobile devices that may use AI algorithms, such as aerial drones or other types of aircraft, marine drones and other types of marine vehicles, underwater drones and other types of underwater vehicles, mobile autonomous robots and semi-autonomous robots. Consequently, the description herein applies to all of these and other types of mobile devices that use AI algorithms.

[0022] In this example, the vehicle 102 includes one or more vehicle computing devices 104 that can communicate with one or more service computing devices 108 through one or more networks 106. The vehicle 102 may further include one or more on-board sensors 112 and one or more vehicle systems 114 that communicate with the vehicle computing devices 104 via a Controller Area Network (CAN) bus (not shown in Figure 1) or the like.

[0023] One or more networks 106 may include any suitable network, including wireless networks such as cellular networks, wide area networks such as the Internet, local area networks such as intranets, local wireless networks such as Wi-Fi, short-range wireless communication such as Bluetooth®, wired networks including optical fiber and Ethernet, any combination thereof, or any other suitable communication network. The components used in such communication technology may depend at least in part on the type of network, the selected environment, or both. Protocols for communicating through such networks are well known and will not be described in detail herein.

[0024] In this example, a vehicle AI diagnostic program (not shown in Figure 1) running on the service computing device 108 may maintain or otherwise access a sensor configuration database 120, which may include information about the respective sensor configurations of individual vehicles 102, and may access services provided by the service computing device 108 to detect indicators of AI algorithm anomalies. The service computing device 108 may further maintain or otherwise access a history AI application output database 122, which may include outputs of numerous AI applications of numerous other vehicles 102 in the past at a particular location, or other location information, such as those associated with road segments. For example, the location may be determined based on satellite positioning system information received from a satellite receiver mounted on each vehicle, or other location sensing methods, and may include specific geographic location coordinates (e.g., longitude and latitude) corresponding to recognitions such as previously identified landmarks, geographic features, traffic signs / signals / intersections, other vehicles, pedestrians, or any of numerous other recognized objects, road features and conditions.

[0025] In some cases, road segments may be determined for part or all of the roads in a given area. Numerous methods for defining road segments based on mapping information are known in the art. As one non-limiting example, the service computing device 108 may obtain mapping information from a map-providing service server or the like (not shown in Figure 1) and identify routes along the roads shown in the mapping information. The service computing device 108 may divide each identified route into a plurality of waypoints. As an example, the waypoints may be identified on a high-resolution map or a standard map which may correspond to specific longitude and latitude coordinates. The service computing device 108 may then identify each road segment between each consecutive pair of waypoints. The length of each road segment between each pair of waypoints may range from a few centimeters to several hundred meters, depending in part on the road configuration and desired level of granularity. Based at least on the waypoint information, each road segment may be associated with geolocation information. Furthermore, while one method for defining road segments is described herein, numerous variations that are of interest to those skilled in the art will be apparent.

[0026] In addition, while road segments are described in the embodiments herein as one way to associate AI output data with geolocation coordinates, road segments may not be used in other embodiments. Instead, in embodiments herein relating to other types of mobile devices, actual geolocation coordinates may be associated with each set of AI output data, and various methods, such as clustering methods, may be used to associate specific AI output data with specific vehicle geolocation coordinates. For example, a clustering method or other thresholding method may be applied to group together AI application outputs that are within a threshold distance of each other. Subsequently, when retrieving historical AI application outputs from the historical AI application output database 122, the same or different thresholds may be applied to the location data received from the mobile device to determine that historical AI application outputs that are sufficiently close to the receiving location of the mobile device are associated with the mobile device's AI algorithm output.

[0027] In this example, a vehicle AI diagnostic program (not shown in Figure 1) may send the vehicle identifier (ID) and the current location 124 of a specific vehicle 102 to the service computing device 108. The service computing device 108 may use the vehicle ID to extract the sensor configuration of vehicle 102 by accessing the sensor configuration database 120. If the vehicle sensor configuration is not yet included in the sensor configuration database 120, the service computing device 108 may request the sensor configuration from the vehicle computing device 104. In addition, the vehicle computing device 104 may send AI application output data 126 to the service computing device 108 so that the service computing device 108 can detect whether there is an anomaly in the AI ​​algorithm 128 used by the AI ​​application 130 running on the vehicle computing device 104.

[0028] As an example, an AI algorithm 128, such as a convolutional neural network, a deep learning neural network, or another type of machine learning model, may be used for object detection and recognition by an AI application 130 installed in the vehicle 102 (or to perform any of the various other AI functions). The service computing device 108 may receive the AI ​​application output data 126 from the vehicle computing device 104 in order to perform AI anomaly indicator detection 134 on the cloud side by comparing the AI ​​application output data 126 with the historical AI application output of the historical AI application output database 122 for the same approximate vehicle position and the same or similar sensor configuration. Based on the analysis results of whether any indicator of AI anomaly is detected on the cloud side, the service computing device 108 may send a diagnostic level 136 to the vehicle computing device 104.

[0029] The diagnostic level 136 transmitted to the vehicle computing device 104 may instruct the vehicle computing device 104 on which diagnostic level to apply to a particular AI application 130, based on the analysis results for detecting anomalies in the AI ​​algorithm. The diagnostic level may be dynamically changed by the vehicle computing device 104 as appropriate, which helps reduce the computational load on the vehicle computing device 104. For example, as further described below, the vehicle computing device 104 may execute a vehicle AI diagnostic program according to different diagnostic levels.

[0030] In the service computing device 108, the data used to detect indicators of AI anomalies may include historical AI application output extracted from the database information of the historical AI application output database 122, the sensor configuration of the vehicle 102 extracted from the sensor configuration database 120, the vehicle position determined from the vehicle GPS receiver (not shown in Figure 1), the vehicle ID, and the current AI application output data 126 output by the AI ​​application 130 running on the vehicle computing device 104.

[0031] In some embodiments, the history AI application output database 122 may store information on objects recognized by the AI ​​application for each of a plurality of previously identified road segments, corresponding to mapping information about the area in which the vehicle 102 is moving. As one specific example, suppose the history AI application output database 122 indicates that for a selected road segment n, 1050 traffic lights were recognized in 850 camera images, respectively. This information may represent the historical probability of object appearance, for example, indicating that the AI ​​application 130 is likely to recognize a traffic light when the vehicle 102's position corresponds to road segment n. In addition or otherwise, as described above, in other examples, location information (e.g., longitude, latitude, altitude, depth, etc.) and clustering may be used instead of or in addition to road segments.

[0032] The sensor configuration database 120 may include information on the configuration and specifications of the on-board sensors mounted on the vehicle 102. Furthermore, the sensor configuration information may include the direction and range used by the sensors to identify the covered road segments (e.g., road segments n, n+1, n+2). The vehicle position data may include information on the vehicle's current longitude and latitude, which is used to identify the covered road segments and determine the corresponding global coordinates of any detected object. As described above, the vehicle position data may be determined by satellite positioning system information, such as by the output of the vehicle's GPS (Global Positioning System) receiver. In addition, or otherwise, other types of position sensors may be used to determine or verify the vehicle's geographical location, such as WiFi signals, cell phone towers, traffic cameras, or communications from other vehicles. The vehicle ID may be an individually distinguishable value that identifies a particular vehicle 102 (e.g., one unique within system 100) and may be used by the service computing device 108 to access the corresponding sensor configuration information (or at least the vehicle's brand and model) for a particular vehicle 102 from the sensor configuration database 120.

[0033] In some embodiments, the current AI application output data 126 may include information on the detected object name (e.g., at least the type of the recognized object), confidence score, and object location, which is compared with the historical AI application output from the historical AI application output database 122. Based on the comparison and other information described above, the service computing device 108 determines whether there are any indicators of an AI algorithm anomaly. For example, if the AI ​​algorithm recognizes a human on a highway where the probability of observing a human is very low, with a high confidence score, this is determined to be an indicator of an AI anomaly.

[0034] Depending on the analysis results of the service computing device 108 detecting one or more indicators of AI anomalies, the diagnostic level may be dynamically changed on the vehicle side, which can help reduce the computational load caused by the execution of diagnostic operations by the vehicle computing device 104. For example, when performing the first diagnostic level 140 (level 1), the AI ​​application output data 126 may be periodically transmitted from the vehicle computing device 104 to the service computing device 108 with a period of 142 or the like, and when level 1 is active, no other actual diagnostics of the AI ​​algorithm 128 are performed by the vehicle computing device 104, and the execution of the AI ​​application 130 is not affected.

[0035] If the service computing device 108 detects a minor indicator of an AI algorithm anomaly based at least in part on the received AI application output data 126, the service computing device 108 may instruct the vehicle computing device 104 to raise the applicable diagnostic level to a second diagnostic level (level 2). When the second diagnostic level 146 (level 2) is active, a periodic AI diagnosis 148 is performed on the AI ​​algorithm 128 by the vehicle computing device 104, and the AI ​​application output data 126 is also periodically sent to the service computing device 108. In some embodiments, the frequency of the diagnosis is determined at least in part on the availability of computing resources of the vehicle computing device 104.

[0036] In addition, if the service computing device 108 detects a significant indicator of an AI algorithm anomaly from the AI ​​application output data 126, the service computing device 108 may instruct the vehicle computing device 104 to change the diagnostic level to the third diagnostic level (level 3). Once the diagnostic level is set to the third diagnostic level 150 (level 3), a dedicated diagnostic (referred to herein as cause identification and recovery operation 152) may be performed. For example, cause identification and recovery operation 152 may include rebooting the relevant vehicle computing device 104 and performing other diagnostic procedures, such as checking for viruses and malware, persistent memory failures, etc. In some embodiments, the vehicle may be pulled over to the side of the road to perform cause identification and recovery operation 152, or proceed to a maintenance facility, etc. Once cause identification and recovery operation 152 is complete, the vehicle computing device 104 may run the AI ​​application 130 again and periodically send the AI ​​application output data 126 to the service computing device 108 to check whether the AI ​​algorithm anomaly has been resolved. If the anomaly persists after multiple repetitions, the vehicle may, depending on the severity of the anomaly, stop, continue moving at a low speed, or proceed to a maintenance facility.

[0037] In addition to detecting indicators of AI anomalies (normal, minor, or significant anomaly indicators), the diagnostic level may also be modified based on other considerations. For example, vehicle control information may indicate unexpected behavior such as sudden braking, sudden acceleration, or unnecessary lane changes. Another example is if the total amount of time used by the vehicle computing device 104 to perform periodic AI diagnostics 148 exceeds a threshold time; this may also be an indicator that the level should be raised to level 3.

[0038] Figure 2 shows an example of the hardware configuration of the system 100 shown in Figure 1, according to several embodiments. For example, each vehicle computing device 104 may include one or more processors 216, one or more computer-readable media 218, one or more communication interfaces (I / F) 220, and, in some embodiments, one or more vehicle human-machine interfaces (HMI) 222. In some cases, the vehicle computing device 104 may include one or more ECUs (electronic control units) or any of various other types of computing devices. For example, the vehicle computing device 104 may include one or more ADAS / AD ECUs that control at least a part of the vehicle system 114, such as performing ADAS and / or AD tasks such as navigation, braking, steering, acceleration, and deceleration. The vehicle computing device 104 may also include one or more other ECUs that control other systems of the vehicle system 114, and all of these may be collectively included in the vehicle computing device 104 described herein.

[0039] An ECU is a general term for any embedded processing system that controls one or more of a vehicle's systems, subsystems, or components. Software such as the vehicle AI diagnostic program 224, the vehicle control program 228, and / or the AI ​​application 130 may be executed by one or more ECUs and may be stored in a portion of a computer-readable medium 218 associated with each ECU (e.g., a program ROM, solid storage, etc., as described later) that enables the ECU to operate as an embedded system. ECUs may generally communicate with each other via a vehicle bus, such as the CAN bus described above, in accordance with a vehicle bus protocol.

[0040] Each ECU or other vehicle computing device 104 may include one or more processors 216, which may include one or more central processing units (CPUs), graphics processing units (GPUs), microprocessors, microcomputers, microcontrollers, digital signal processors, state machines, logic circuits, system-on-chip processors, and / or any devices that manipulate signals based on operational instructions. For example, the processor 216 may include one or more hardware processors and / or logic circuits of any preferred type that are specifically programmed or configured to perform the algorithms and other processes described herein. The processor 216 may be configured to fetch and execute computer-readable instructions (computer-executable instructions) stored in a computer-readable medium 218 and programmed to perform the functions described herein.

[0041] The computer-readable medium 218 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storing information such as computer-readable instructions, data structures, programs, program modules, and other code or data. For example, the computer-readable medium 218 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, optical storage, solid-state storage, magnetic disks, cloud storage, or any other media that can be used to store desired information and can be accessed by a computing device. Depending on the configuration of the vehicle computing device 104, the computer-readable medium 218 may be a tangible non-temporary medium, insofar as it is a non-temporary computer-readable medium that does not include energy, carrier signals, electromagnetic waves, and / or signals themselves, as referred to. In some cases, the computer-readable medium 218 may be in the same location as the vehicle computing device 104, and in other examples, a portion of the computer-readable medium 218 may be separate from the vehicle computing device 104.

[0042] The computer-readable medium 218 may be used to store any functional components that can be executed by the processor 216. In many embodiments, these functional components include instructions or programs that can be executed by the processor 216, which, when executed, are programmed to cause the processor 216 to take action so that the vehicle computing device 104 has the functions described herein. Examples of functional components stored in the computer-readable medium 218 include one or more vehicle AI diagnostic programs 224, vehicle control programs 228, and AI applications 130, each of which may include one or more computer programs, applications, executable code, or parts thereof. Furthermore, although these programs and applications are illustrated together in this example, some or all of these programs and applications may be executed on separate vehicle computing devices 104 during use. Alternatively, in some embodiments, these programs and applications 130, 224, and 228 may each be part of a single program or application. For example, in some cases, the vehicle control program 228 may also be an AI application 130 that uses one or more AI algorithms 128 to determine at least one value.

[0043] In addition, the computer-readable medium 218 may store data, data structures, machine learning models and other AI algorithms 128, and other information used to implement the functions and services described herein. For example, the computer-readable medium 218 may store AI algorithms 128, which may be one or more machine learning models. Examples of AI algorithms 128 that may be used in some embodiments of this specification include artificial neural networks such as self-organizing neural networks, recurrent neural networks, convolutional neural networks, modular neural networks and deep learning neural networks, as well as probabilistic models such as predictive models, decision trees, classifiers and linear regression models, support vector machines and Markov models and hidden Markov models. The embodiments of this specification are not limited to any particular type of AI algorithm.

[0044] In addition, the computer-readable medium 218 may store sensor data 226 received from the in-vehicle sensor 112, which may be used by the AI ​​application 130 to determine AI application output data 126, which may then be used by the vehicle control program 228 to control one or more vehicle systems 114, subsystems, components, etc. Furthermore, although the data, data structures, and AI algorithms 128 are illustrated collectively in this example, some or all of these elements may be stored in a separate vehicle computing device 104 during use. The vehicle computing device 104 may also contain or hold other functional components and data, including programs, drivers, and data used or generated by functional components. Furthermore, the vehicle computing device 104 may contain many other logical components, programmatic components, and physical components, of which those described above are merely examples relevant to the discussion herein.

[0045] One or more communication interfaces 220 may include one or more software and hardware components that enable communication with various other devices, such as through a CAN bus and / or through one or more networks 106. For example, communication interfaces 220 may enable communication through one or more of the following, as additionally enumerated anywhere in this specification: LAN, Internet, cable network, cellular network, wireless network (e.g., Wi-Fi) and wired network (e.g., CAN, Fibre Channel, optical fiber, Ethernet), direct connection, and short-range wireless communication such as BLUETOOTH®.

[0046] The sensor data 226 may include sensor data received from the on-board sensors 112. For example, the on-board sensors 112 may include any of several different types of sensors, such as a camera system, radar, LiDAR, ultrasonic, a Global Navigation Satellite System (GNSS) receiver (hereinafter referred to by the general term "GPS," which is also intended to include any other satellite positioning system), an accelerometer, and a compass. In addition, the sensor data 226 used by the vehicle control program 228 may include information received from or associated with various vehicle systems 114, such as a suspension controller associated with the suspension system, a steering controller associated with the steering system, and a vehicle speed controller associated with the braking and acceleration systems (not shown in Figure 1).

[0047] For example, the vehicle control program 228 may use rule-based and / or artificial intelligence-based control algorithms (including one or more of the AI ​​algorithms 128) to determine vehicle control parameters based on the output of the AI ​​application 130 (e.g., recognition output). For example, the vehicle control program 228 may determine appropriate actions such as braking, steering, and acceleration, and based on the determined actions, may send one or more control signals to one or more vehicle systems 114. For example, in some applications, the vehicle control program 228 may send control signals to a suspension controller, a steering controller, and / or a vehicle speed controller to control or partially control the vehicle.

[0048] The service computing device 108 may include one or more servers, or other types of computing devices which may be embodied in any way. For example, in the case of a server, the program, other functional components, and data may be implemented in a single server, a cluster of servers, a server farm or data center, a cloud-hosted computing device, etc., but other computer architectures may be used additionally or alternatively.

[0049] Furthermore, although the drawings illustrate the functional components and data of the service computing device 108 as existing in a single location, these functional components and data may be distributed across different computing devices and locations in any manner. Consequently, the functionality may be implemented by one or more service computing devices 108, and the various functions described herein may be distributed across different computing devices in various ways. Multiple service computing devices 108 may be located together or separately and may be organized, for example, as a virtual server, server bank, and / or server farm. The functions described may be provided by a server of a single entity or enterprise, or by servers and / or services of multiple different entities or enterprises.

[0050] In the illustrated examples, each service computing device 108 may include one or more processors 240, one or more computer-readable media 242, and one or more communication interfaces 244. Each processor 240 may be a single processing unit or multiple processing units, and may include one or more computing units or multiple processing cores. The processor 240 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, system-on-chip processors, state machines, logic circuits, and / or any device that manipulates signals based on operation instructions. For example, the processor 240 may be one or more hardware processors and / or logic circuits of any preferred type that are specifically programmed or configured to perform the algorithms and processes described herein. The processor 240 can be configured to fetch and execute computer-readable instructions stored in the computer-readable media 242 and programmed to perform the functions described herein.

[0051] The computer-readable medium 242 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Examples of such computer-readable medium 242 include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, optical storage, solid-state storage, magnetic tape, magnetic disk storage, storage arrays, network-attached storage, storage area networks, cloud storage, or any other media that can be used to store desired information and can be accessed by a computing device. Depending on the configuration of the service computing device 108, the computer-readable medium 242 may be a type of computer-readable medium, and / or, as referred to herein, a tangible non-temporary medium insofar as it does not include a medium such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0052] The computer-readable medium 242 may be used to store any functional components that can be executed by the processor 240. In many embodiments, these functional components include instructions or programs that can be executed by the processor 240, which, when executed, are programmed to cause one or more processors 240 to take action so that the service computing device 108 has the functions described herein. Functional components stored in the computer-readable medium 242 may include a server AI diagnostic program 246 that may be executed to detect indicators of AI algorithm anomalies based on the reception and analysis of the vehicle ID, vehicle location, and AI application output data 126, and to provide instructions for a diagnostic level 136 based on the analysis.

[0053] In addition, the computer-readable medium 242 may store data used to perform the operations described herein. Thus, the computer-readable medium 242 may include the sensor configuration database 120 and the history AI application output database 122 as described above. In addition, the computer-readable medium 242 may store AI application output data 126, at least temporarily. In addition, if no anomaly is detected, the server AI diagnostic program 246 may add the received AI application output data 126 to the history AI application output database 122 for the corresponding road segment (or other location data) and sensor configuration. However, if an anomaly is detected, the received AI application output data 126 is not added to the history AI application output database 122.

[0054] Furthermore, the service computing device 108 may also include or hold other functional components and data not specifically shown in Figure 2, including programs, drivers, and data used or generated by the functional components. In addition, the service computing device 108 may include many other logical, programmatic, and physical components, of which those described above are merely illustrative in connection with the description herein.

[0055] The communication interface 244 may include one or more interfaces and hardware components that enable communication with various other devices, such as through the network 106. For example, the communication interface 244 may enable communication through one or more of the following, as additionally enumerated elsewhere in this specification: the Internet, cable networks, cellular networks, wireless networks (e.g., Wi-Fi), and wired networks (e.g., fiber optics and Ethernet), as well as short-range wireless communications such as BLUETOOTH® and BLUETOOTH® Low Energy.

[0056] Figure 3 shows an example of diagnostic levels 300 in several embodiments. In the example in Figure 3, the first diagnostic level 302 (level 1) corresponds only to transmitting AI application output data to the service computing device 108, the second diagnostic level 304 (level 2) further includes periodically transmitting AI application output data to the service computing device 108 and performing a limited diagnostic (e.g., periodically) on the AI ​​algorithm, and the third diagnostic level 306 (level 3) includes performing a dedicated diagnostic on the AI ​​algorithm, such as identifying the cause of an anomaly and performing recovery.

[0057] In addition, as shown in 308, if an indicator of an AI anomaly classified as a minor indicator is detected, the vehicle computing device 104 is instructed to operate at the second diagnostic level 304 (level 2), i.e., to perform periodic, limited diagnostics on the AI ​​algorithm. In addition, as shown in 310, if there have been no recent detections of AI anomalies, the vehicle computing device 104 may be instructed to return to the first diagnostic level 302 (level 1), and may no longer perform any periodic diagnostics on the AI ​​algorithm. Alternatively, as shown in 312, if the vehicle computing device 104 is currently operating at the second diagnostic level 304 (level 2) and detects an AI anomaly, or an indicator of abnormal vehicle behavior such as sudden braking or sudden lane changes classified as a significant indicator of an AI anomaly, the vehicle computing device 104 may raise the diagnostic level to the third diagnostic level 306 (level 3) to perform a dedicated diagnostic of the corresponding AI algorithm, such as identifying the cause of the anomaly and performing corrective actions.

[0058] Alternatively, as shown in 314, if the service computing device 108 detects an indicator of an AI anomaly that is classified as a prominent indicator, the service computing device 108 (i.e., the server AI diagnostic program 246) may instruct the vehicle computing device 104 to transition to a third diagnostic level 306 (level 3) to initiate a dedicated AI algorithm diagnosis that identifies the cause of the AI ​​anomaly and restores the normal operation of the AI ​​algorithm. Furthermore, as shown in 316, once the cause identification and recovery are complete, the vehicle computing device 104 may return to the first diagnostic level 302 (level 1) to periodically send AI application output data 126 to the service computing device 108.

[0059] Figure 4 is a flowchart illustrating an example of process 400 for performing diagnostics on an AI algorithm, based on several embodiments. The processes described herein are shown in logical flowcharts as a collection of blocks representing a sequence of operations, some or all of which can be implemented in hardware, software, or a combination thereof. In the case of software, blocks may represent computer-readable instructions stored in one or more computer-readable media, programmed to perform the enumerated operations and executed by one or more processors. Generally, computer-readable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or realize a particular data type. The order in which the blocks are described should not be interpreted as limiting. Any number of blocks described can be combined in any order and / or in parallel to realize a process or alternative process, and not all blocks need to be executed. For illustrative purposes, the processes are described with reference to the environments, systems, and devices described in the embodiments herein, but the processes may be implemented in a wide variety of other environments, systems, and devices.

[0060] In some embodiments, process 400 may be performed by the system 100 described above. For example, in the illustrated example, process 400 may be performed by the service computing device 108, such as by executing the server AI diagnostic program 246. Alternatively, if the vehicle computing device 104 has access to sufficient database information and sufficient processing power, the vehicle computing device 104 may, in some cases, perform some or all of process 400.

[0061] In the example of process 400 in Figure 4, one or more indicators of anomalies in the AI ​​algorithm may be detected based on the output of the AI ​​application. Process 400 shows a server-side process for detecting indicators of AI anomalies for a single AI algorithm, either for a single sensor in some embodiments or for multiple sensors in other embodiments. As described above, first, the road segments covered by the sensors are identified. In addition to historical AI output information, the historical AI application output database 122 may include information on the names and locations of stationary objects, such as landmarks, traffic lights, stop signs, buildings, geographical features, utility poles, and trees. In some cases, the process for detecting AI anomalies may be divided into detection of moving objects and detection of stationary objects. For example, for moving objects such as people or vehicles, if the current AI application output includes a person with a high confidence score and the actual presence of the person is unlikely to be based on historical data, this may be classified as a minor indicator of AI anomaly, and periodic diagnostics (level 2) may be performed. Thus, in some embodiments, inconsistencies detected with respect to moving objects may be classified as minor indicators of AI anomalies.

[0062] In addition, if the current AI application output does not include a traffic light that should be detected as a stationary object, and there are no obstacles, this may be classified as an indicator of a significant AI anomaly, and a dedicated diagnosis (level 3) may be performed. Stationary object information (object name (e.g., traffic light), longitude, latitude, height) stored in the history AI application output database 122 may also be used to detect significant indicators of AI anomalies, for example, based on the difference between the actual height of the object and the height estimated by the AI ​​application. Furthermore, if there is a malfunction in detecting the expected number of landmarks or other stationary objects at the current GPS location, and the sensor configuration when there are no obstacles is taken into consideration, this may also be classified as an indicator of a significant AI anomaly, and a dedicated diagnosis (level 3) may be performed. Therefore, in some embodiments, inconsistencies regarding stationary objects and unexpected vehicle behavior may be classified as significant indicators of AI anomalies.

[0063] In step 402, the service computing device 108 may receive vehicle location data, vehicle ID, and past and present AI application outputs from the vehicle computing device 104, which are used to perform diagnostics to detect any indicators of AI algorithm anomalies. As described above, for example with respect to Figure 1, past AI application outputs may be obtained by accessing the history AI application output database 122.

[0064] In step 404, the service computing device 108 may receive vehicle sensor configuration information. For example, based on the vehicle ID received in step 402, the service computing device 108 may access the sensor configuration database 120 to obtain the sensor configuration for a particular vehicle. As an example, the sensor configuration may already be provided to the service computing device 108 by the vehicle computing device 104, such as when registering for the AI ​​anomaly monitoring service provided by the service computing device 108. As another example, the sensor configuration may be a standard sensor configuration for the vehicle type and model corresponding to the vehicle ID received in step 402. Other modifications that are in the interest of the disclosure herein will be obvious to those skilled in the art.

[0065] In step 406, the service computing device 108 may determine the road segments covered by the vehicle sensors based on at least vehicle position data and sensor configuration data relating to the vehicle. In addition or otherwise, in embodiments of this specification, such as those relating to other types of mobile devices other than ground vehicles, instead of determining road segments, the AI ​​application output may be determined based on the received position data and stored in association with corresponding position information, such as longitude and latitude coordinates (and optionally altitude / depth).

[0066] In step 408, the service computing device 108 may determine whether the vehicle AI system to be monitored is currently operational. If it is operational, the process proceeds to step 410. If it is not operational, the process returns to step 402 and waits for the next reception of vehicle information. As an example, the service computing device 108 may send a communication to the vehicle computing device 104 to determine whether the AI ​​system is currently operational.

[0067] In step 410, based on the determination that at least the vehicle AI system is operational, the service computing device 108 may send a command to the vehicle computing device 104 to periodically send AI application output to the service computing device 108, i.e., in accordance with the Level 1 diagnostics described above with respect to Figures 1 to 3.

[0068] In step 412, the service computing device 108 may determine whether there is any discrepancy between the historical AI model output and the current AI application output for stationary objects in the road segment currently under consideration (or, in other embodiments, the currently determined location). If there is a discrepancy, this may be classified as a significant indicator of an AI anomaly, and the process proceeds to step 420. If there is no discrepancy, the process proceeds to step 414.

[0069] In step 414, the service computing device 108 may determine whether there is any discrepancy between past and present AI application outputs for a moving object associated with the currently considered road segment (or other currently determined location). If there is a discrepancy, this may be classified as a minor indicator of an AI anomaly, and the process proceeds to step 416. If there is no discrepancy, the process returns to step 406 to select the next road segment (or next location) to consider the corresponding past and present AI application outputs.

[0070] In step 416, based on the determination of at least minor indicators of AI algorithm anomalies, the service computing device 108 may send a command to the vehicle computing device 104 instructing it to perform a periodic diagnostic (level 2).

[0071] In step 418, the service computing device 108 may determine whether an AI algorithm anomaly is detected by a level 2 periodic diagnosis. If detected, the process proceeds to step 420. If not detected, after a threshold number of iterations, the process may return to step 406.

[0072] In step 420, based on the detection of an AI algorithm anomaly by the Level 2 periodic diagnosis in step 418, or based on the determination that there is a discrepancy between past and present AI application outputs for a stationary object, as determined in step 412, the service computing device 108 may send a command to the vehicle computing device 104 to perform a dedicated diagnosis in accordance with Level 3. In some embodiments, sending a command may include at least one of the following: notifying the vehicle occupants of the AI ​​algorithm anomaly and recommending immediate service of the vehicle; sending a notification to devices associated with vehicle maintenance; changing the vehicle's operating speed or pulling the vehicle to the side of the road to perform a process to identify the cause of the AI ​​anomaly and perform a recovery; or instructing the vehicle to proceed to a maintenance location to identify the cause of the anomaly and perform a recovery.

[0073] In step 422, the service computing device 108 may receive notification of the completion of a Level 3 diagnosis, including cause identification and / or recovery of the AI ​​algorithm. For example, in response to receiving a command from the service computing device 108 in step 420, the vehicle computing device 104 may perform a Level 3 diagnosis on the AI ​​algorithm, such as determining the cause of an anomaly and / or performing a recovery, such as by rebooting the computing device running the AI ​​algorithm. In some embodiments, if the vehicle is currently operating on the road, the vehicle control program 228 may perform a cause identification diagnosis and recovery, such as slowing down, stopping, or taking the vehicle to a maintenance facility, as described above. Once the Level 3 diagnosis is complete, the vehicle computing device 104 may periodically transmit AI application output and current vehicle position data to the service computing device 108 so that the service computing device 108 can continue monitoring the AI ​​system on the vehicle. Numerous other modifications that are of interest to those skilled in the art will be apparent to those skilled in the art.

[0074] Figure 5 shows an example 500 of identifying an AI algorithm that is likely to have an AI anomaly from among several AI algorithms according to several embodiments. In this example, it is assumed that there are three AI algorithms, namely a first AI algorithm 502 that receives sensor data from a first sensor 504 (sensor 1), a second AI algorithm 506 that receives sensor data from a second sensor 508 (sensor 2), and a third AI algorithm 510 that performs sensor fusion on the results of the first sensor 504 (sensor 1) and the second sensor 508 (sensor 2), as shown in 512.

[0075] Furthermore, in this embodiment, as shown in 514, the sensor detection areas of the first sensor 504 (sensor 1) and the second sensor 508 (sensor 2) overlap. For example, the first sensor 504 (sensor 1) may have a wide-angle field of view, and the second sensor 508 (sensor 2) may have a narrower, long-range field of view. As shown in 514, since the sensor detection areas (i.e., fields of view) overlap, the first AI algorithm 502 and the second AI algorithm 506 may detect the same object within their respective fields of view, such as a human 516.

[0076] In addition, when identifying a malfunctioning AI algorithm from among multiple AI algorithms 502, 506, and 510, the embodiments of this specification may consider both short-term and long-term perspectives. From a short-term perspective, if all three AI algorithms show high confidence scores and the names of the identified objects are identical (e.g., "human"), the service computing device 108 (not shown in Figure 5) may determine that there is no indicator of AI anomaly. In particular, as shown in 518, in the first example, we assume that the first AI algorithm 502 identifies a human with 80% confidence, the second AI algorithm 506 identifies a human with 70% confidence, and the third AI algorithm 510 identifies a human with 85% confidence. Thus, the results of these AI algorithm outputs indicate that there is no AI anomaly detected among the multiple AI algorithms 502, 506, and 510.

[0077] On the other hand, as shown in 520, in the second example, we assume that the second AI algorithm 506 exhibits a relatively low confidence score (e.g., 20 percent) regarding the identification of a human (or a different object name), the first AI algorithm 502 identifies a human with 70 percent confidence, and the third AI algorithm 510, which performs sensor fusion, identifies a human with 60 percent confidence. In this embodiment, the service computing device may determine that the results of the AI ​​algorithm output include an indicator of AI anomaly. Furthermore, from a short-term perspective, the second AI algorithm 506 may appear to be an algorithm that includes AI anomaly.

[0078] To track AI anomalies from a long-term perspective, a data structure 522 containing the ID 524 of each AI algorithm and an anomaly counter 526 for each listed AI algorithm may be maintained. Thus, from a long-term perspective, if the output of any AI algorithm does not match the output of another AI algorithm, the value of the anomaly counter 526 may be incremented with respect to that AI algorithm. Subsequently, if the value of the anomaly counter 526 meets a threshold, the corresponding AI algorithm may be flagged as having an indicator of AI anomaly. Based on such an indicator of AI anomaly for each AI algorithm, the diagnostic computing load can be reduced by increasing the diagnostic level only for the AI ​​algorithms whose anomaly counter 526 has reached the threshold limit. Consequently, in the second example, the anomaly counter 526 for the second AI algorithm 506 may be incremented from "410" to "411", but the value of the anomaly counter 526 for the second AI algorithm 506 remains below the threshold for the anomaly indicator, while the anomaly counter 526 for the first AI algorithm 502 has already exceeded the threshold and may be subject to, for example, a level 2 periodic diagnosis.

[0079] Figure 6 shows an example 600 of reducing the number of false positives to detect AI anomalies, based on several embodiments. As an example, the diagnostic level is not changed in the case of some exceptions that may occur in special scenarios in order to reduce the number of false positives for the indicator of detected AI anomalies. For example, as shown in 602, when vehicle 102 enters an area under construction, the ongoing construction often causes the current AI algorithm output to differ from past AI outputs, which can lead to a false indicator of an AI anomaly.

[0080] In this example, vehicle 102 detects one or more traffic cones 604 as it passes through a specific road segment indicating a construction area. Therefore, in this situation, the service computing device 108 may determine that the vehicle has entered a construction area and may stop storing the AI ​​application output data received from vehicle 102. For example, such a special scenario may be treated as an exception, and the diagnostic level for the corresponding AI remains unchanged even though there may be a significant difference between the current recognition result and the recognition result stored in the historical AI application output database 122 for the same road segment. Examples to which such an exception applies include cases where the AI ​​application output data includes the recognition of one or more traffic cones, construction barricades, construction equipment, etc., indicating a construction area, as well as various other special scenarios such as the detection of the presence of ambulances, fire trucks, police vehicles, etc. Numerous other modifications that are of interest to those skilled in the art will be apparent to those skilled in the art.

[0081] Figure 7 shows an example 700 of determining whether or not to add AI application output data 126 to the historical AI application output database 122, according to several embodiments. In this example, the historical AI application output database 122 is populated with data based on a comparison with a confidence score threshold. When adding new data to the historical AI application output database 122, the AI ​​application output data 126 received from each vehicle 102 may be stored for each road segment that each vehicle 102 passes through, using an AI algorithm monitoring service provided by the service computing device 108.

[0082] Generally, if the confidence score of an AI algorithm output is relatively low, the output is more likely to be substantially incorrect. Therefore, in some embodiments of this specification, a threshold for the minimum confidence score (confidence score threshold) is set, and only AI output data with a confidence score above the confidence score threshold is stored in the historical AI application output database 122. By setting a confidence score threshold, the accuracy of detecting indicators of AI anomalies is improved and the number of false positives is reduced.

[0083] In the example shown in Figure 7, vehicle 102 may be able to recognize human 702 on road 704. As shown in 706, in the first example, we assume that the recognized object name is human and the confidence score provided by the AI ​​algorithm is 85 percent. Furthermore, we assume that the confidence score threshold for adding the data to the historical AI application output database 122 is 30 percent. Thus, in the first example, the AI ​​application output data 126 is added to the historical AI application output database 122 for the current road segment (or location) of vehicle 102. On the other hand, as shown in 708, in the second example, we assume that the object name in the AI ​​application output is human and the confidence score of the AI ​​algorithm is 29 percent. Since a confidence score of 29 percent is lower than the confidence score threshold of 30 percent, the AI ​​application output data 126 is not added to the historical AI application output database 122. Furthermore, although a confidence score threshold of 30 percent is used in this example, in other embodiments, the confidence score threshold may differ depending on the specific usage scenario, system configuration, etc.

[0084] The process examples described herein are for illustrative purposes only. Numerous other variations will be obvious to those skilled in the art in light of the disclosure herein. Furthermore, while the disclosure herein describes several examples of preferred frameworks, architectures, and environments for performing the processes, the examples herein are not limited to the specific examples illustrated and described herein. Furthermore, this disclosure provides a variety of exemplary embodiments, such as those described and illustrated in the drawings. However, this disclosure is not limited to the embodiments described and illustrated herein and can be extended to other embodiments that are known or will become known to those skilled in the art.

[0085] The various instructions, processes, and methods described herein may be executed by the processor herein as computer-readable instructions, such as computer programs and applications stored on computer-readable media. Generally, the terms "program" and "application" may be used interchangeably and may include instructions, routines, modules, objects, components, data structures, executable code, etc., for performing a particular task or realizing a particular data type. These programs, applications, etc., may be executed as native code or may be downloaded and executed in a virtual machine or other runtime compilation environment. Generally, the functionality of programs and applications may be combined or distributed as desired in various embodiments. Embodiments of these programs, applications, and methods may be stored on computer-readable media or transmitted between some form of communication medium.

[0086] While the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter as defined in the attached claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms that realize the claims. [Explanation of Symbols]

[0087] 104... Vehicle computing devices 106…Network 108…Service Computing Devices 112... Vehicle sensors 114... Vehicle System 120...Sensor configuration database 122...History AI Application Output DB 126…AI application output data 128…AI Algorithms 130…AI applications 216… Processor 218… Computer-readable media 220...Communication I / F 222…Human-Machine Interface (HMI) for Vehicles 224... Vehicle AI diagnostic program 226... Sensor data 228... Vehicle control program 240… Processor 242… Computer-readable media 244…Communication I / F 246…Server AI Diagnostic Program 504...Sensor 1 508...Sensor 2 510... Sensor Fusion AI 524...ID of the AI ​​algorithm 526... Anomaly Counter

Claims

1. A system for detecting abnormalities in artificial intelligence (AI) systems installed in vehicles, The system comprises one or more processors configured to execute processing by executable instructions, wherein the processing is, The system receives at least vehicle position data and AI output data from the aforementioned vehicle. Determining the road segment corresponding to the aforementioned vehicle position data, Accessing past AI output data stored in a data structure and associated with the road segment, based at least on the road segment, The received AI output data is compared with the past AI output data stored in the data structure with respect to the road segment. Based at least a comparison of the received AI output data with the past AI output data associated with the road segment, a command is transmitted to the vehicle to cause it to perform at least one action. The confidence score associated with the received AI output data is compared with a confidence score threshold. Based at least on the confidence score associated with the received AI output data that satisfies the confidence score threshold, the received AI output data is stored in the data structure associated with the road segment. including, system.

2. The system according to claim 1, The aforementioned process, Receiving a vehicle identifier from the aforementioned vehicle, The further includes determining the sensor configuration for the vehicle based on at least the received vehicle identifier, The comparison between the received AI output data and the past AI output data relating to the road segment further includes taking into account the sensor configuration for the vehicle. system.

3. The system according to claim 1, Comparing the received AI output data with the past AI output data includes detecting at least one difference between a moving object identified in the received AI output data and a moving object identified in the past AI output data. system.

4. The system according to claim 3, Based at least on the detection of the at least one difference, the command transmitted to the vehicle includes a command for the vehicle to perform a periodic diagnosis of the AI ​​algorithm corresponding to the AI ​​output data to check for anomalies in the AI ​​algorithm. system.

5. The system according to claim 1, Comparing the received AI output data with the past AI output data includes detecting at least one difference between a stationary object identified in the received AI output data and a stationary object identified in the past AI output data. system.

6. The system according to claim 5, Based at least on the detection of the at least one difference, the command transmitted to the vehicle includes a command for the vehicle to check for an anomaly in the AI ​​algorithm by performing at least one of the cause identification or recovery processes for the AI ​​algorithm corresponding to the AI ​​output data. system.

7. The system according to claim 1, Comparing the received AI output data with the past AI output data includes not identifying indicators of AI algorithm anomalies. Based at least on the failure to identify the indicator of the AI ​​algorithm anomaly, the command transmitted to the vehicle includes a command to cause the vehicle to periodically transmit additional AI output data in order to continue monitoring the AI ​​algorithm anomaly. system.

8. A method for detecting anomalies in an artificial intelligence (AI) system installed in a mobile device, One or more processors receive at least mobile device position data and AI output data from the mobile device. Determine the position corresponding to the aforementioned mobile device position data, Based at least the aforementioned location, access past AI output data stored in a data structure and associated with the aforementioned location, The received AI output data is compared with the past AI output data stored in the data structure with respect to the position. Based at least a comparison of the received AI output data with the past AI output data associated with the position, a command is transmitted to the moving device to cause it to perform at least one action, and further, The confidence score associated with the received AI output data is compared with the confidence score threshold. Based at least on the confidence score associated with the received AI output data that satisfies the confidence score threshold, the received AI output data is stored in the data structure associated with the location. method.

9. The method according to claim 8, further, The mobile device identifier is received from the aforementioned mobile device. Based on at least the received mobile device identifier, the sensor configuration for the mobile device is determined. When comparing the received AI output data with the past AI output data relating to the position, the sensor configuration for the moving device is taken into consideration. method.

10. The method according to claim 8, Comparing the received AI output data with the past AI output data includes detecting at least one difference between a moving object identified in the received AI output data and a moving object identified in the past AI output data. method.

11. The method according to claim 10, Based at least on the detection of the at least one difference, the command transmitted to the mobile device includes a command for the mobile device to perform a periodic diagnosis of the AI ​​algorithm corresponding to the AI ​​output data to check for anomalies in the AI ​​algorithm. method.

12. The method according to claim 8, Comparing the received AI output data with the past AI output data includes detecting at least one difference between a stationary object identified in the received AI output data and a stationary object identified in the past AI output data. method.

13. The method according to claim 12, Based at least on the detection of the at least one difference, the command transmitted to the mobile device includes a command for the mobile device to check for an anomaly in the AI ​​algorithm by performing at least one of the cause identification or recovery processes for the AI ​​algorithm corresponding to the AI ​​output data. method.

14. The method according to claim 8, Comparing the received AI output data with the past AI output data includes not identifying indicators of AI algorithm anomalies. Based at least on the failure to identify the indicator of the AI ​​algorithm anomaly, the command transmitted to the mobile device includes a command to cause the mobile device to periodically transmit additional AI output data in order to continue monitoring the AI ​​algorithm anomaly. method.

15. One or more non-temporary computer-readable media for storing instructions that can be executed by one or more processors, wherein the processing is performed by The system receives at least mobile device position data and artificial intelligence (AI) output data from the mobile device, Determining the position corresponding to the aforementioned mobile device position data, Accessing past AI output data stored in a data structure and associated with the location, based at least on the aforementioned location, The received AI output data is compared with the past AI output data stored in the data structure with respect to the position. Based on a comparison of the received AI output data with the past AI output data associated with the position, a command is transmitted to the mobile device to cause it to perform at least one action. The confidence score associated with the received AI output data is compared with a confidence score threshold. Based at least on the confidence score associated with the received AI output data that satisfies the confidence score threshold, the received AI output data is stored in the data structure associated with the location. including, Non-temporary computer-readable media.

16. One or more non-temporary computer-readable media according to claim 15, The aforementioned process, Receiving a mobile device identifier from the aforementioned mobile device, The further includes determining the sensor configuration for the mobile device based on at least the received mobile device identifier, The comparison between the received AI output data and the past AI output data relating to the position further includes taking into account the sensor configuration for the moving device. Non-temporary computer-readable media.

17. One or more non-temporary computer-readable media according to claim 15, Comparing the received AI output data with the past AI output data includes detecting at least one difference between a moving object identified in the received AI output data and a moving object identified in the past AI output data. Based at least on the detection of the at least one difference, the command transmitted to the mobile device includes a command for the mobile device to perform a periodic diagnosis of the AI ​​algorithm corresponding to the AI ​​output data to check for anomalies in the AI ​​algorithm. Non-temporary computer-readable media.

18. One or more non-temporary computer-readable media according to claim 15, Comparing the received AI output data with the past AI output data includes detecting at least one difference between a stationary object identified in the received AI output data and a stationary object identified in the past AI output data. Based at least on the detection of the at least one difference, the command transmitted to the mobile device includes a command for the mobile device to check for an anomaly in the AI ​​algorithm by performing at least one of the cause identification or recovery processes for the AI ​​algorithm corresponding to the AI ​​output data. Non-temporary computer-readable media.

Images