Contextual root cause analysis for vehicle software system

Abstract

A method of diagnosing a software system of a vehicle includes receiving data related to the software system of the vehicle, identifying an anomalous event based on a pattern of the received data, and collecting contextual information related to the anomalous event. The method further includes inputting the anomalous event and the contextual information to a machine learning model, determining a root cause of the anomalous event through the machine learning model, and performing a mitigation action based on determining that the anomalous event corresponds to a fault.

Description

TECHNICAL FIELD

[0001] The subject disclosure relates to error or failure detection, and more specifically to diagnosis of root causes of anomalous signals. BACKGROUND

[0002] Many modern vehicles (e.g., automobiles, motorcycles, boats, or any other type of motorized vehicle) include control systems that represent a complex integration of hardware and software components. Such control systems utilize information from many sources (e.g., sensors and control units) to monitor and control vehicle operation and provide various features. As a result, vehicles can rely on complex software architectures that are monitored to ensure proper operation and identify faults, failures, and other issues. SUMMARY

[0003] In one example embodiment, a method of diagnosing a software system of a vehicle includes receiving data related to the software system of the vehicle, identifying an anomalous event based on a pattern of the received data, and collecting contextual information related to the anomalous event. The method further includes inputting the anomalous event and the contextual information to a machine learning model, determining a root cause of the anomalous event by the machine learning model, and performing a mitigation action based on determining that the anomalous event corresponds to a fault.

[0004] In addition to one or more features described herein, identifying the anomalous event includes clustering a plurality of similar events, and associating the anomalous event with the cluster.

[0005] In addition to one or more features described herein, the contextual information further includes at least one of an identity context, a temporal context, a location context, and an environmental context.

[0006] In addition to one or more features described herein, the machine learning model is a large language model for a specific domain that is configured to output a diagnostic report including a concise language description of the anomalous event and the root cause.

[0007] In addition to one or more features described herein, the large language model is configured to interact with a user and provide diagnostic information using Retrieval-Augmented Generation (RAG) in response to a question posed by the user.

[0008] In addition to one or more features described herein, the method further includes proactively training the large language model based on the identified anomalous event and the associated contextual information, wherein the training includes iteratively presenting questions to the machine learning model.

[0009] In addition to one or more of the features described herein, the machine learning model includes a graphical machine learning (GML) model configured to associate the anomalous event with contextual information, the GML model is configured to generate a composite list of anomalous events, and each anomalous event is assigned a salience score.

[0010] In addition to one or more of the features described herein, the GML model generates a contextual graph including a plurality of nodes including a node for the anomalous event and a node for each context specified by the contextual information, and the GML model performs link prediction to determine contextual associations between the plurality of nodes.

[0011] In addition to one or more of the features described herein, the anomalous event is identified using an anomaly detection machine learning model.

[0012] In addition to one or more of the features described herein, performing a mitigation action includes at least one of: presenting an alert to a user, a vehicle control system, or a remote entity; applying a correction or update to the software system; and controlling operation of a vehicle.

[0013] In another example embodiment, a system for diagnosing a software system includes a data collection module configured to receive data from the software system and a root cause analysis tool configured to identify an anomalous event based on a pattern in the received data, collect contextual information related to the anomalous event, input the anomalous event and the contextual information to a machine learning model, and determine a root cause of the anomalous event based on the contextual information by the machine learning model.

[0014] In addition to one or more of the features described herein, the contextual information includes at least one of an identity context, a temporal context, a location context, and an environmental context.

[0015] In addition to one or more of the features described herein, the machine learning model is a large language model for a particular domain configured to output a diagnostic report including a concise language description of the anomalous event and the root cause.

[0016] In addition to one or more of the features described herein, the large language model is configured to interact with a user and provide diagnostic information using retrieval-augmented generation (RAG) in response to a question posed by the user.

[0017] In addition to one or more of the features described herein, the root cause analysis tool is configured to proactively train the large language model based on the identified anomalous event and associated contextual information, wherein the training includes iteratively presenting questions to the machine learning model.

[0018] In addition to one or more of the features described herein, determining the root cause includes generating a context graph comprising a plurality of nodes including a node for the abnormal event and a node for each context specified by the context information, and performing a contextual graph embedding for input to the large language model.

[0019] In yet another example embodiment, a vehicle system includes a memory having computer readable instructions and a processing device for executing the computer readable instructions, the computer readable instructions controlling the processing device to perform a method comprising receiving data from a software system of a vehicle, identifying an abnormal event based on a pattern of the received data, collecting context information related to the abnormal event, inputting the abnormal event and the context information to a machine learning model, and determining a root cause of the abnormal event by the machine learning model based on the context information.

[0020] In addition to one or more of the features described herein, identifying the abnormal event includes clustering a plurality of similar events, and associating the abnormal event with the cluster.

[0021] In addition to one or more of the features described herein, the context information further includes at least one of an identity context, a temporal context, a location context, and an environmental context.

[0022] In addition to one or more of the features described herein, the machine learning model is a domain-specific large language model configured to output a diagnostic report including a concise language description of the abnormal event and the root cause.

[0023] The above features and advantages of the present disclosure, and other features and advantages, are readily apparent from the following detailed description when taken in connection with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0024] Other features, aspects, and details are described below with reference to the drawings. The drawings are provided and are part of this specification and are included to further illustrate aspects and features of the present disclosure.

[0025] Figure 1 is a top view of a motor vehicle including various processing devices in accordance with example embodiments;

[0026] Figure 2 depicts a computer system in accordance with example embodiments;

[0027] Figure 3 depicts a software observability system in accordance with example embodiments;

[0028] Figure 4 depicts a root cause analysis (RCA) tool in accordance with example embodiments;

[0029] Figure 5 depicts components of the RCA tool and byFigure 4 aspects of exception detection and context correlation performed by the RCA tool of

[0030] Figure 6 Aspects of a method of correlating exception events identified from software telemetry data with context information according to example embodiments are depicted;

[0031] Figure 7 Examples of a method of Figure 6 are depicted;

[0032] Figure 8 Aspects of generating context information used by the RCA tool of Figure 4 are depicted according to example embodiments;

[0033] Figure 9 Aspects of in-context active learning by machine learning models used by the RCA tool of Figure 4 are depicted according to example embodiments;

[0034] Figure 10 Examples of 5-Whys visualizations that can be included as part of a diagnostic report generated by the RCA tool of Figure 4 are depicted; and

[0035] Figure 11 Aspects of interaction between an expert and machine learning models of the RCA tool of Figure 4 are depicted according to example embodiments. DETAILED DESCRIPTION

[0036] The following description is merely exemplary in nature and is not intended to limit the present disclosure, its application, or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features.

[0037] Apparatuses, systems, and methods are provided for observing software systems and diagnosing system exceptions based on data collected from vehicle systems and context information. Embodiments of the system are configured to analyze software data (e.g., telemetry data, source code, documentation, event logs, historical records of events or exceptions, etc.) and identify exception events from patterns in the data. “Software data” or “received data” refers to any data collected from a software system and / or related to the operation of the software system that can be used to assess the performance of the software system and / or its components.

[0038] The system collects context information that is used to characterize the identified exception events and / or determine whether such events represent actual faults or conditions (e.g., errors, mistakes, or other suboptimal operations, or any significant abnormal behavior) that should be corrected or resolved.

[0039] In embodiments, the contextual information includes an identity context, a temporal context, a location or spatial context, and / or an environmental context. In embodiments, the contextual information and the detected abnormal event are input to a machine learning model, such as a large language model, for determining underlying or root causes and contributing factors. In embodiments, the machine learning model is configured to output a plain language description of the event, the abnormality, the underlying root cause, and / or a recommended action, as well as any other relevant or useful information.

[0040] The embodiments described herein present a number of advantages and technical effects. In complex systems, such as vehicle systems, there is often a potentially large number of potential abnormal causes. As a result, it can be difficult and time consuming to identify the actual root cause of an abnormality. Embodiments provide an efficient system for automatically identifying root causes in an interpretable manner and / or providing root cause information to a user, such that a human user can understand the detection process and trust the results. Embodiments reduce the time and complexity associated with diagnosing.

[0041] Other advantages include an enhanced ability to process noisy log events and reduce alert fatigue, as well as shorter mean time to resolution / repair (MTTR). Additionally, embodiments can be used to build, update, or use a knowledge base to facilitate identification of underlying causes and contributing factors. The knowledge base can be continuously or periodically updated; for example, the knowledge base is an evolving knowledge base (EKB).

[0042] Embodiments can also enhance existing platforms for identifying faults or abnormalities and for root cause analysis. For example, there are existing software observability platforms for aggregating and visualizing telemetry data, identifying problems, identifying root causes, and implementing troubleshooting. Embodiments enhance such systems by providing contextual analysis, which leads to improved identification of causes of detected events, as well as improved correlation of detected events to real problems.

[0043] Figure 1 An embodiment of a motor vehicle 10 is shown, which includes a vehicle body 12 that at least partially defines a passenger cabin 14. The vehicle body 12 also supports various vehicle subsystems, including a propulsion system 16, as well as other subsystems to support the functionality of the propulsion system 16 and other vehicle components, such as a fuel system, a brake system, a suspension system, a steering subsystem, an exhaust system, etc.

[0044] The vehicle can be a combustion engine vehicle, an electric vehicle (EV), or a hybrid vehicle. In an example, the vehicle 10 is a hybrid vehicle that includes a combustion engine 20 and an electric motor 22.

[0045] The vehicle also includes various control systems for controlling aspects of the vehicle systems. For example, one or more electronic control units (ECUs) 24 are provided. Aspects of the diagnostic and control methods described herein can be performed by any suitable controller or processing device, such as the ECU 24 and / or controllers in respective subsystems.

[0046] Embodiments of the vehicle 10 include devices and / or systems for communicating with other vehicles and / or objects external to the vehicle. For example, the vehicle 10 includes a communication system having a telematics unit 26 or other suitable device, including an antenna or other transmitter / receiver for communicating with a network 28.

[0047] The network 28 represents any one or combination of different types of suitable communication networks, such as public networks (e.g., the Internet), private networks, wireless networks, cellular networks, or any other suitable private and / or public networks. Moreover, the network 28 can have any suitable communication range associated therewith and can include, for example, global networks (e.g., the Internet), metropolitan area networks (MANs), wide area networks (WANs), local area networks (LANs), personal area networks (PANs). The network 28 can communicate via any suitable communication modalities, such as short-range wireless, radio frequency, satellite communication, or any combination thereof.

[0048] In embodiments, the network 28 connects the vehicle 10 to communicate with various entities. For example, the network 28 can connect to a server 30, a database 32, and / or other remote entities 34, such as workstations, control centers, other vehicles, etc.

[0049] The vehicle 10 also includes a computer system 36 that includes one or more processing devices 38 and a user interface 40. The various processing devices and units can communicate with each other via a communication device or system, such as a controller area network (CAN) or a transmission control protocol (TCP) bus.

[0050] Figure 2 Aspects of an embodiment of a computer system 240 that can perform various aspects of the embodiments described herein are shown. The computer system 240 includes at least one processing device 242, which generally includes one or more processors for performing aspects of the image acquisition and analysis methods described herein.

[0051] The components of computer system 240 include a processing device 242, such as one or more processors or processing units, a memory 244, and a bus 246 that couples various system components including the system memory 244 to the processing device 242. The system memory 244 can be non-transitory computer- readable media, and can include various types of computer-system readable media. Such media can be any available media that is accessible by processing device 242 and includes both volatile and non-volatile media, and removable and non-removable media.

[0052] For example, the system memory 244 includes a non-volatile memory 248, such as a hard disk drive, and can also include a volatile memory 250, such as a random access memory (RAM) and / or a cache memory. The computer system 240 can also include other removable / non-removable, volatile / non-volatile computer system storage media.

[0053] The system memory 244 can include at least one program product having a set (at least one) of program modules that configure the processing device 242 to perform the functions of the embodiments described herein. For example, the system memory 244 stores various program modules that generally carry out the functions and / or methodologies of the embodiments described herein. A module 250 for performing functions related to acquiring signals and data can be included, and a module 252 for performing functions related to anomaly detection and diagnostics as discussed herein can be included. The system 240 is not limited in this regard, as other modules can be included. As used herein, the term "module" refers to processing circuitry, which can include an Application Specific Integrated Circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that execute one or more software or firmware programs, a combinational logic circuit, and / or other suitable components that provide the described functionality.

[0054] The processing device 242 can also communicate with one or more external devices 256, such as a keyboard, a pointing device, and / or any device (e.g., a network card, a modem, etc.) that enables the processing device 242 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interfaces 264 and 265.

[0055] The processing device 242 can also communicate with one or more networks 266, such as a Local Area Network (LAN), a General Wide Area Network (WAN), bus networks, and / or public networks (e.g., the Internet) via a network adapter 268. It will be appreciated that, although not shown, other hardware and / or software components that are

[0056] Figure 3 Embodiments of a software monitoring or observability system 50 for monitoring a software system to detect anomalies and / or determine root causes of anomalies are depicted. “Software observability” refers to the ability to infer the internal state of a software system based on its external outputs. The software monitoring system 50 can be embodied in any suitable processing device or system, such as the computer system 240, the vehicle computer system 36, and / or the ECUs 24.

[0057] The observability system 50 obtains output data, such as telemetry or monitoring data related to software components, from a device or system. It should be noted that while the software components are described as software used by the vehicle 10 and / or software used in relation to vehicle operation, embodiments are applicable to any suitable software system.

[0058] The observability system 50 includes a data collection module 52 configured to collect telemetry data, which can be any form of data related to software performance. Examples of telemetry data include metrics, traces, logs, and profiles.

[0059] The data collection module 52 inputs the collected data to a software observability platform 54 for aggregating and visualizing telemetry data, identifying problems, identifying root causes, and enabling troubleshooting. The platform 54 is capable of providing insights into the internal state of a software system during runtime, thereby allowing developers and operators to understand the behavior of the software system, diagnose problems, and optimize performance.

[0060] The observability platform 54 includes processing components or modules for performing functions related to monitoring and diagnostics. For example, the observability platform 54 includes a monitoring module 56 that continuously or periodically receives software data (e.g., metrics used in descriptive analytics, distributed traces, event logs, reports, etc.) and provides the received data to an anomaly identification module 58. The anomaly identification module 58 correlates patterns of data to problems or anomalies. Examples of such anomalies include interruptions, performance bottlenecks, errors, and the like.

[0061] A root cause analysis module 60 determines underlying or root causes and / or factors that contribute to an anomaly or problem. A recommendation module 62 can be included that recommends corrective actions to address the anomaly or problem. Such corrective actions can be provided to correct the anomaly, contain the anomaly, or otherwise resolve or remediate the anomaly.

[0062] An interface is provided to allow users to interact with the observability platform 54. For example, an operations dashboard 64 is accessible by a user, such as a technician, a site reliability engineer (SRE), or other software expert 66. Generally, an “expert” refers to any person or entity having expertise in the software system being monitored.

[0063] The observability system 50 includes or is connected to one or more components that provide context-based anomaly detection and root cause determination. For example, a context root cause analysis (RCA) tool 70 is in communication with components of the observability platform 54. Although the context RCA tool 70 is shown as a separate module or system, embodiments are not so limited as the RCA tool 70 (or components thereof) can be incorporated directly into the observability platform 54.

[0064] In embodiments, the context RCA tool 70 includes an anomaly detection and context correlation module 72 (also referred to as a “detection and correlation module”) that detects anomalies based on patterns in collected data and correlates detected anomalies with context information. In embodiments, the detection and correlation module 72 includes or is configured with a machine learning model (anomaly detection model) that learns data patterns indicative of various anomalies.

[0065] In embodiments, context information is obtained by a context analysis module 74 or context analyzer. The context analyzer 74 collects any information suitable for determining the context in which telemetry data was collected. “Context” or “context information” refers to any information that characterizes the situation, environment, or conditions under which an anomaly occurred. Context information allows the observability platform and / or RCA tool 70 to characterize anomalies in received data and / or determine whether anomalies in received data correspond to actual bugs or problems with the monitored software.

[0066] In an embodiment, the detection and correlation module 72 and the context analyzer 74 provide input to a machine learning model 76, such as a large language model (LLM). The machine learning model 76 can be a generalized LLM, or can be a fine-tuned model trained for a particular domain, and / or include frozen layers that limit the training data to domain-specificize the model 76.

[0067] Aspects of the observability system 50 can be included or stored at various locations. For example, the data collection module 52 can be disposed in the vehicle 10, and the observability platform stored in the server 30. The observability system 50 includes an interface, such as the operations dashboard 64, stored at the workstation 34 and usable by a site reliability engineer (SRE), technician, dealer, telematics live advisor, or other user.

[0068] In use, observability system 50 receives data collected from the software system and analyzes the received data to identify one or more anomalous events (i.e., anomalies) based on patterns in the collected data. The anomalous events are consolidated and fed to machine learning model 76. Contextual information is provided and fed to machine learning model 76, which characterizes the anomalous events (e.g., describes the particular context in which the anomalies occurred), and / or determines whether the one or more anomalous events indicate a fault or otherwise should be addressed or remediated. Machine learning model 76 outputs a plain language report to a user that summarizes any detected anomalies and their context, describes root causes and other underlying factors, and can provide solutions or suggested actions. In embodiments, the user can interact with machine learning model 76 to input questions and receive plain language answers.

[0069] Figure 4 Embodiments of RCA tool 70 and aspects of the root cause analysis method are illustratively depicted. In this embodiment, context analyzer 74 provides contextual information that includes descriptions or indications of various contexts (e.g., type of vehicle, geographic location, etc.) and can also include reference data. For various contexts, the reference data indicates data patterns that are considered to be associated with anomalous operation and / or data patterns that are associated with normal operation.

[0070] Context analyzer 74 determines the context based on any suitable information. For example, a user can input context information (e.g., via operations dashboard 64) and / or context analyzer 74 can infer the context from the combination of inputs. Reference data can be obtained from suitable sources, such as knowledge base 78.

[0071] “Contextual information” includes any data or information that allows RCA tool 70 to characterize or categorize an anomalous event or otherwise determine whether the anomalous event is benign or reflective of an underlying problem. Contextual information includes, for example, textual descriptors about the circumstances associated with the event (e.g., from event logs), temporal information (e.g., when the event occurred, frequency of the event, seasonality, etc.), event name or other identifier, and location information.

[0072] Contextual information and the anomalous events are provided to model 76, such as LLM 76. LLM 76 outputs a description of any detected anomalies, as well as the context and one or more root causes and / or underlying factors determined by LLM 76. For example, LLM 76 outputs diagnostic report 80, which includes a plain language summary of the anomaly, the context associated with the anomaly, and / or one or more root causes associated therewith. Diagnostic report 80 can follow a template, such as RCA template 82, which provides a format for the report. Other templates can be used for interaction with a user, such as question and answer (Q&A) template 84.

[0073] For example, the diagnostic report 80 includes a brief language summary of the problem or anomaly, as well as a list of root causes and any other contributing factors. In embodiments, the diagnostic report 80 includes visualizations (which can be interactive), such as a display of the results of a word cloud or a “5-Whys” interactive analysis (which allows the LLM 76 to ask a series of questions in order to build a knowledge / causal graph or otherwise build causal information).

[0074] A knowledge / causal graph is a graphically structured data type that evolves over time and captures various aspects of the metadata (e.g., detected events, deep reasons, and contributing factors). In addition, the knowledge / causal graph can encode interaction data, representing causal effects that allow understanding of relationships and dynamics within a software system.

[0075] The diagnostic report 80 can be output to the expert 66 (e.g., an SRE) via the operations dashboard 64. The expert 66 can interact with the operations dashboard 64 and / or the diagnostic report 80 to ask questions and provide information as described herein.

[0076] The diagnostic report 80 can also be provided to one or more other users 68 (e.g., an owner, a field consultant, a service person, etc.). In an embodiment, the other user(s) 68 can interact with the diagnostic report 80 to give the other user(s) 68 the ability to ask questions for further elaboration. Giving external users (e.g., in addition to the expert 66) access can enrich the causal information, as these users can have different perspectives. A default setting can be provided that limits two-way communication to the expert 66 only.

[0077] In embodiments, the RCA tool 70 includes a learning module 86. The learning module 86 is used for active in-context learning. The learning module 86 can also include the ability for users to interact with the LLM 76 to ask questions and retrieve information. In embodiments, the learning module 86 provides large language model augmentation techniques for improving or enhancing answers and information provided in response to user questions. An example of such a technique is Retrieval-Augmented Generation (RAG), which enhances the accuracy and relevance of answers to user questions by accessing external information (i.e., information not already considered by the LLM 76). For example, in response to a user (e.g., the expert 66 or other user 68) entering a question, the learning module 86 queries an external source (such as the knowledge base 78) to augment the answer.

[0078] Figure 5 Components of an embodiment of the anomaly detection and correlation module 72 are depicted. Figure 5 Aspects of associating anomalies with contextual information using the detection and correlation module 72 are also depicted.

[0079] The detection and correlation module is configured to receive telemetry and / or other software data such as flow logs 90 and input the flow logs 90 to an event parser 92. The event parser 92 structures and classifies events in the flow logs 90. The structured and classified events are input to an unsupervised pattern recognition model 94 that learns patterns for each type of log event and outputs structured log events.

[0080] The pattern recognition model 94 is used to build a baseline for each type of log event, and these baselines are used to detect anomalies (represented by anomaly detection 96). The anomaly detection 96 involves receiving log events from the parser 92 and comparing the patterns in each log event to the corresponding baseline pattern. If the log event pattern does not match the baseline (e.g., differs from the baseline by more than a threshold amount, or otherwise deviates from the baseline pattern), the log event is identified as an anomalous event.

[0081] Anomalous events are collected and analyzed to correlate the events with contextual information (contextual correlation 98). For example, anomalous events are compared to stored anomalies. In another example, anomalous events are correlated with their temporal aspects (e.g., peak times or seasonality) and / or geographic aspects (e.g., geographic location). In embodiments, similar events can be clustered to identify related clusters of anomalies across flow logs (or other data files) to reduce noise and / or identify core events.

[0082] The contextual correlation 98 correlates each anomalous event with one or more contexts. The output and information from the contextual correlation is provided to the LLM 76 for characterizing the anomalous events. “Characterizing” the anomalous events can include indicating whether the anomalous event is normal or abnormal, and if the anomalous event is abnormal, identifying one or more root causes.

[0083] Figure 6 An example of the correlation performed by the detection and correlation module 72 is shown. As discussed herein, anomaly detection 96 is performed based on learned baselines and events. Anomalous events can be classified by rarity and severity (block 100), and similar events are clustered (block 102).

[0084] Various criteria, such as scarcity and severity, can be used to assign a score (called a "salience score") to each anomalous event based on its significance. "Scarcity" refers to how rare the anomalous event is, where high scarcity corresponds to a higher significance score (e.g., a higher score will be given to the first occurrence of an event). "Severity" reflects how bad or serious the anomalous event is (e.g., an event with higher severity will receive a higher significance score than an event with lower severity). Severity can be quantified based on the deviation of the anomalous event from the baseline and the duration of that deviation. High deviation and longer duration result in higher severity (and a higher confidence score) compared to a momentary or shorter duration and / or low deviation. Rare and severe events will receive a higher total score than events that are only severe or only rare.

[0085] Contextual information from context analyzer 74 is associated with anomalous events (or clusters of anomalous events). In embodiments, contextual information includes identity context, temporal context, location or spatial context, and / or environmental context.

[0086] "Identity context" relates to the identity and capabilities of vehicle 10 (e.g., vehicle identification number, manufacturer information). "Temporal context" refers to the timing and / or frequency of the occurrence of the anomalous event. "Spatial context" refers to geographic location and / or spatial location (e.g., geographic location or GPS). "Environmental context" refers to any conditions or circumstances that affect whether the anomalous event is anomalous (e.g., vehicle health status). Other contextual information may include "spatiotemporal context" reflecting changes in both time and space (e.g., timing and location changes in social events).

[0087] exist Figure 6 In the example, at box 104, anomalous events and contextual information are incorporated into the context graph (box 104). Context graph 106 is constructed to combine clustered anomalous events, environmental contexts (such as similar current or previous anomalous events), temporal contexts (such as peak times or seasonality), and spatial contexts of geographic anomalies.

[0088] Context graph 106 includes nodes for a cluster of events, such as a given anomalous event or similar event. Additionally, context graph 106 includes nodes for each context provided by context information. Nodes are linked via links or edges representing relationships between nodes.

[0089] exist Figure 6In the example of FIG. 1, the context graph 106 includes a node AE representing the aggregated abnormal events, a node TC for temporal context, a node SC for environmental context, and a node GC for geospatial context. Each node includes one or more node features xi. For example, the node AE includes identifying information such as an identifier (“id”) and / or a vehicle identification number (“vin”), as well as metrics from the telemetry data. The node GC can include spatial information such as a geographic location, GPS coordinates, etc. The TC node can include temporal information (e.g., a particular date / time, seasonality, etc.). The context information in the SC node can include similar or prior events or abnormalities.

[0090] A graph embedding based on graph machine learning is generated (block 108) to learn a mapping from a discrete high-dimensional graph domain to a low-dimensional continuous domain. The graph embedding 108 can be generated using one or more graph machine learning (GML) models such as a graph neural network, a convolutional graph network (CGN), and / or a graph attention network (GAT) to convert the context graph 106 into a feature vector for further inductive reasoning tasks. Attention mechanisms can also be used to significantly improve the task model by selectively weighting the importance of different components of the context graph 106. Link prediction is performed (block 110) to identify context associations (links) between nodes. The resulting abnormal event information and context associations are then provided to the LLM 76 to determine root causes. For example, the GML model provides a comprehensive list of abnormal events, along with corresponding significance scores and context associations.

[0091] Figure 7 An example of generating the context graph 106 is shown. In this example, the telemetry data includes metrics 112 and associated event logs 114. The metrics 112 are formulated as table data, and the event logs 114 are used to create word embeddings for the LLM 76, and the table data and word embeddings are used to generate nodes such as the nodes AE, SC, GC, and TC. Edges are initially assigned randomly (block 116), and graph embeddings 108 are generated using negative sampling (block 118) and unsupervised contrastive loss learning (block 120). The graph embeddings 108 are provided for link prediction 110, and the node information and link prediction 110 results are used to construct the context graph 106.

[0092] Figure 8 An embodiment of a method 121 performed by the context analyzer 74 is depicted. When determining a context associated with received data, the received data (e.g., telemetry data including metrics and event logs) is input to the context analyzer 74 from the observability platform 54. Additionally, a semantic search of the knowledge base 78 is performed (block 122) to obtain a set of context data 124.

[0093] The context analyzer 74 receives information and data from the vehicle 10 (e.g., health status, diagnostic codes, battery status, data drift information, etc.) as well as any interaction data from active learning (block 126). Content filtering is performed to remove irrelevant information (block 128), and context information is identified (block 130) and extracted (block 132). The context information (shown as element 133) is output to, for example, the detection and correlation module 72 and / or the LLM 76 Figure 4

[0094] The following is an example of explanatory information to generate context information 133. Generally, the context information 113 includes one or more contexts and reference data used to determine the context.

[0095] In this example, the context analyzer 74 receives information in the form of average RAM utilization (68% over the past 12 hours), performance data including CPU usage (0.2%) and vehicle speed (zero). The context analyzer 74 also receives identifying information (vehicle identification number (vin), device identifier, and version and log identifiers) and location information.

[0096] Based on these inputs, the context analyzer 74 determines an identity context (vehicle identification data), a temporal context (past 12 hours), a spatial context, and an environmental context. In this example, the environmental context is “high memory usage with low CPU utilization and vehicle stationary.”

[0097] The context and reference data are then provided to the LLM 76. The LLM 76 also receives abnormal event data related to the high RAM utilization. The LLM 76 explains the root cause as “possible memory leak.” The root cause and context information 133 can be provided to a user, such as the expert 66. The expert 66 can also interact with the LLM 76 to ask questions or answer questions and further probe the root cause of the high RAM utilization.

[0098] Figure 9 An embodiment is depicted in which the LLM 76 is configured to actively learn deep root causes and contributing factors of various anomalies or abnormal events in various contexts. The active learning in context (block 140) can be used to build or enhance the knowledge base 78.

[0099] In an embodiment, the active learning 140 is implemented through interaction with an SRE or other expert 66. In an embodiment, the active learning 140 involves questioning and answering (“Q&A”). An example of an active learning process is the “5-Whys” method, which involves answering a series of five questions (or other desired or suitable number of questions).

[0100] ​Active learning 140 can include other root cause analysis methods, such as an Ishikawa diagram, a fault tree analysis (FTA), a failure modes and effects analysis (FMEA). Such methods can be performed in conjunction with the question answering. For example, the question answering can support the FMEA process through initial screening to identify common failure modes for particular software components and / or to identify trends and patterns that can be helpful in understanding root causes, if historical data for such failure modes is available.

[0101] In embodiments, active learning is based on a 5-Whys root cause analysis method, which is an iterative process that helps identify the root cause of a problem and leads to knowledge that is later used to identify deeper causes and contributing factors. During each iteration, the LLM 76 generates a question (i.e., why did the identified anomaly occur) and generates an answer. The results of the 5-Whys analysis can be presented to the expert 66 (and / or other users) to provide a level of confidence in the root cause determination by the LLM 76. Note that embodiments are not limited to the 5-Whys analysis, as any suitable analysis method that provides insight into the reasoning of the LLM 76 can be used.

[0102] Learning in specific contexts can involve utilizing a static LLM with few-shot (FS), one-shot (1S), or zero-shot (0S) configurations, or alternatively, a fine-tuned (FT) LLM 76. In the FS method, the LLM 76 is presented with a small number of task demonstrations during reasoning, used as tuning, without updating any weights. These demonstrations include K instances of a context and a completion of a root cause determination, followed by a single context example in which the LLM 76 is expected to generate a corresponding completion. The 1S method is similar to the FS method, but K is set to 1. The 0S method mirrors the FS, but instead of examples, a brief language description of the task is presented to the LLM 76. Fine-tuning (FT) involves updating the weights of a pre-trained model by training on a number of supervised labels specific to the intended task. Fine-tuning requires adjusting the weights of a pre-existing model by exposing the pre-existing model to extensive training with many supervised labels tailored to the target task.

[0103] Reports such as the diagnostic report 80 Figure 4 include a brief language summary of the one or more anomalies, as well as a list or other description of the deep root causes and contributing factors. The report 80 can also include a word cloud, a display of the 5-Whys analysis results, and / or any other visualization. The RCA report can include interactive visualizations.

[0104] Figure 10An example of a visualization 140 of the results of an interactive 5-Whys analysis that can be provided to the expert 66 via the operations dashboard 64 is depicted. In this example, an identified anomaly in the form of increased central processing unit (CPU) usage at the vehicle bus level is discovered. The visualization 144 shows a series of five “why” questions (denoted as Wl, W2, W3, W4, and W5), where each why question asks why the anomaly has occurred. Each why question is generated by the LLM 76 and also answered by the LLM 76. In Figure 10 In the example, the answer Al to the first why question Wl is “increased overhead,” the answer A2 to the second why question W2 is “a large number of running apps,” the answer A3 to the third why question W3 is “rolling out a new version or configuration update,” the answer A4 to the fourth why question W4 is “handling a recently detected security and data breach,” and the answer A5 to the fifth why question W5 is “detected shadow and zombie APIs.” The section labeled “Root Cause?” indicates whether the answer is a root cause.

[0105] The expert 66 can interact with the operations dashboard 64 to provide answers to the 5-Whys analysis (e.g., if the LLM 76 is unable to generate and answer a sufficient number of questions during the 5-Whys analysis), to allow the expert 66 to provide knowledge to the RCA tool 70, and / or to allow the expert 66 to ask questions.

[0106] Figure 11 An embodiment of the RCA tool 70 is depicted, and aspects of the interaction between the expert and the RCA tool 70 are depicted. As shown, the expert 66 can interact with the operations dashboard 64 to ask questions of the LLM 76. In embodiments, the LLM 76 is configured to answer questions and can augment answers by querying external sources of information, such as the knowledge base 78. For example, the LLM 76 is a general-purpose LLM and is configured to use Retrieval Augmented Generation (RAG) 150 to enable the LLM 76 to provide domain-specific answers to questions asked by the expert 66. RAG is not limited to use with general-purpose LLMs (e.g., RAG can be used to augment answers by a domain-specific LLM). Further, the RCA tool 70 can be configured to allow other users 68 to ask questions.

[0107] The terms “a” and “an” do not denote a limitation of quantity, but rather denote the presence of at least one of the referenced items. The term “or” means “and / or” unless clearly indicated otherwise. The reference in the specification to “an aspect” means that a particular element described in connection with the aspect is included in at least one aspect described herein and can or can not be present in other aspects. In addition, it should be understood that the described elements can be combined in any suitable manner in the various aspects.

[0108] When an element such as a layer, film, region, or substrate is referred to as being “on” another element, it can be directly on the other element or intervening elements can also be present. In contrast, when an element is referred to as being “directly on” another element, there are no intervening elements present.

[0109] Unless otherwise indicated herein, all test standards are the most recent standard in effect as of the filing date of this application, or, if priority is claimed, the filing date of the earliest priority application in which the test standard appears.

[0110] Unless defined otherwise, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0111] While the foregoing disclosure has described exemplary embodiments, one of ordinary skill in the art will understand that various changes can be made therein without departing from the scope of the disclosure. For example, various elements of the disclosed embodiments can be combined, substituted, or deleted, and the material of construction of the disclosed embodiments can be varied, all without departing from the scope of the disclosure. In addition, many modifications can be made to adapt a particular situation or material to the teachings of the disclosure without departing from the central scope thereof. Therefore, it is intended that the disclosure not be limited to the particular embodiments disclosed, but that the disclosure will include all embodiments falling within the scope of the disclosure.

Claims

1. A method of diagnosing a software system of a vehicle, comprising: receiving data related to the software system of the vehicle; identifying an anomaly event based on a pattern of the received data; collecting contextual information related to the anomaly event; inputting the anomaly event and the contextual information to a machine learning model; determining a root cause of the anomaly event by the machine learning model; and based on determining that the anomaly event corresponds to a fault, performing a mitigation action.

2. The method of claim 1, wherein identifying the anomaly event comprises clustering a plurality of similar events, and associating the anomaly event with a cluster.

3. The method of claim 1, wherein, the contextual information comprises at least one of an identity context, a temporal context, a location context, and an environmental context.

4. The method of claim 1, wherein, the machine learning model is a domain-specific large language model configured to output a diagnostic report comprising a concise language description of the anomaly event and the root cause.

5. The method of claim 4, wherein, the large language model is configured to interact with a user and provide diagnostic information using Retrieval-Augmented Generation (RAG) in response to a question posed by the user.

6. The method of claim 4, further comprising proactively training the large language model based on the identified anomaly event and associated contextual information, wherein training comprises iteratively presenting questions to a machine learning model.

7. The method of claim 1, wherein, the machine learning model comprises a graph machine learning (GML) model configured to associate the anomaly event with the contextual information, the GML model is configured to generate a comprehensive list of anomaly events, and each of the anomaly events is assigned a saliency score.

8. The method of claim 7, wherein, the GML model generates a context graph comprising a plurality of nodes including a node for the anomaly event and a node for each context specified by the contextual information, and the GML model performs link prediction to determine contextual associations between the plurality of nodes.

9. A system for diagnosing a software system, comprising: a data collection module configured to receive data from the software system; and a root cause analysis tool configured to perform: identifying an anomaly event based on a pattern of the received data; collecting contextual information related to the anomaly event; inputting the anomaly event and the contextual information to a machine learning model; and determining a root cause of the anomaly event by the machine learning model based on the contextual information.

10. The system of claim 9, wherein, the machine learning model is a domain-specific large language model configured to output a diagnostic report comprising a concise language description of the anomaly event and the root cause.

Images