System and method for generating remedial action recommendations for power and performance issues within semiconductor software and hardware

Abstract

A computer-implemented method for generating remedial action recommendations for power and performance issues within semiconductor software and hardware is disclosed. For example, the disclosed systems and methods can apply a rule-based model to telemetry data to generate a rule-based root cause output and a telemetry-based unknown output. The disclosed systems and methods can also apply a root cause machine learning model to the telemetry-based unknown output to utilize the telemetry-based unknown output to analyze deep and complex failure modes, thereby ultimately generating one or more root cause remedial action recommendations that are specific to the identified fault and the client computing device that is experiencing the fault.

Description

Background Art

[0001] The example automated analysis system generally performs rules-based analysis on telemetry data associated with semiconductor software and / or hardware failures, performance, and power issues. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The accompanying drawings illustrate several example implementations and are a part of the specification. Together with the following description, these drawings demonstrate and explain the various principles of the present disclosure.

[0003] Figure 1 is a flow chart of an exemplary computer-implemented method for generating remedial action recommendations for power and performance issues within semiconductor software and hardware in accordance with one or more implementations.

[0004] Figure 2 is an overview sequence diagram of a remedial action recommendation system for generating device-specific remedial action recommendations that address one or more root causes of a fault, power, or performance issue according to one or more specific implementations.

[0005] Figure 3 is a detailed sequence diagram of a remedial action recommendation system for generating device-specific remedial action recommendations according to one or more specific implementations.

[0006] Figure 4 is a detailed diagram of a system for recommending remedial actions based on one or more specific implementations.

[0007] In all figures, the same reference numerals and descriptions indicate similar but not necessarily identical elements. Although the example implementations described herein are susceptible to various modifications and alternative forms, specific implementations have been shown by way of example in the drawings and will be described in detail herein. However, the example implementations described herein are not intended to be limited to the specific forms disclosed. On the contrary, the present disclosure covers all modifications, equivalents, and alternatives that fall within the scope of the appended claims. DETAILED DESCRIPTION

[0008] The present disclosure as a whole relates to systems and methods for generating remedial action recommendations for power and performance issues within semiconductor software and hardware. As described above, an example analysis system can use a rule-based approach to analyze telemetry data from one or more telemetry sources associated with a computing device. Typically, such telemetry data can be complex and / or meaningless (e.g., "noisy"). The example analysis system is typically unable to identify such complex and noisy data patterns associated with failures, performance, and power issues associated with semiconductor software and hardware. In addition, the example analysis system is typically strictly limited to expected telemetry data patterns and progressions. Therefore, when unexpected telemetry is received, the example analysis system typically produces output that requires additional manual analysis.

[0009] In more detail, example analysis systems typically employ rigorous, rule-based analysis of telemetry data in an attempt to determine the root cause of various types of system failures. However, this rule-based approach only works when the telemetry data conforms to an expected format and a fairly simple pattern. As a result, example analysis systems are typically limited to high-level problem analysis. Nonetheless, as described above, the root causes of some system failures and other problems can only be determined through in-depth analysis in response to complex patterns within noisy telemetry data. Given this, example analysis systems may be unable to remedy up to 50% of reported system failures.

[0010] To address these and other issues, the systems and methods described herein effectively generate accurate and device-specific remedial actions for faults, performance, and power issues indicated by complex and potentially noisy telemetry data. For example, the systems and methods described herein can receive and transform telemetry data from various telemetry sources based on both rule-based models and machine learning-based models. The systems and methods described herein can also apply these models to the transformed telemetry data to generate accurate predictions indicating the root causes of problems experienced by semiconductor software and / or hardware. In addition, the systems and methods described herein can also generate remedial action recommendations based on the indicated root causes, which end users can apply to client computing devices to resolve these problems.

[0011] Therefore, the systems and methods described herein provide accurate, efficient, flexible, and resilient solutions to the problems experienced by the example analysis system. For example, as described above, the example system strictly requires that telemetry data be received in an expected format and a fairly simple pattern. Therefore, when such example systems receive complex and noisy data, these systems are unable to generate meaningful and accurate root cause predictions. In contrast, the systems and methods described herein include a resilient data pipeline that is capable of receiving complex and noisy / unexpected telemetry data and converting it into a usable format. By applying rule-based and machine learning-based models to such converted telemetry data within a resilient data pipeline, the systems and methods described herein are capable of analyzing deep failure patterns to generate accurate root cause predictions. The systems and methods described herein are also capable of generating accurate remedial action recommendations customized for end-user computing devices based on these predictions to address failures, performance, and power issues experienced by the computing device.

[0012] As will be described in more detail below, the present disclosure describes various systems and methods for generating accurate and device-specific remediations for faults, performance, and power issues. In one specific implementation, a method for generating device-specific remediations for faults, performance, and power issues can include: generating a rule-based root cause output and a telemetry-based unknown output based on telemetry data from a client computing device, and generating a machine learning-based root cause prediction by applying a root cause machine learning model to the telemetry-based unknown output. The method can also include: generating one or more root cause recommendations specific to the client computing device based on the rule-based root cause output and the machine learning-based root cause prediction.

[0013] In one or more specific implementations, the method can also include: before generating the rule-based root cause output and the telemetry-based unknown output, receiving telemetry data from the client computing device and parsing the telemetry data based on the rule-based model and the root cause machine learning model. In some specific implementations, receiving telemetry data from the client computing device can include receiving raw and unstructured data from one or more of an offline log and a real-time telemetry source.

[0014] In addition, in some specific implementations, parsing telemetry data based on the rule-based model and the root cause machine learning model can include: parsing the telemetry data into structured data based on one or more of offline logs and real-time telemetry sources, parsing the structured data into attributes specific to the rule-based model, and parsing the structured data into attributes specific to the root cause machine learning model. In addition, parsing telemetry data based on the rule-based model and the root cause machine learning model can also include generating one or more of engineering features, data encoding, custom mapping, and data enhancement based on the attributes specific to the rule-based model and the attributes specific to the root cause machine learning model.

[0015] In one or more implementations, the method can also include: modifying the rule-based root cause output according to the machine learning-based root cause prediction, and generating a client-specific root cause result according to the machine learning-based root cause prediction and the modified rule-based root cause output. In some implementations, one or more root cause recommendations specific to the client computing device are generated by applying a recommendation engine to the client-specific root cause results, and the recommendation engine can include multiple trained machine learning-based recommendation units. In some implementations, the recommendation engine can also include one or more of a lookup table or a rule-based recommendation.

[0016] In at least one specific implementation, the method can further include: generating filtered remediation recommendations tailored to a user security level associated with the client computing device based on one or more root cause recommendations specific to the client computing device and end user data extracted from telemetry data from the client computing device. In addition, in at least one specific implementation, the method can further include: generating a machine learning-based unknown quantity by applying the root cause machine learning model to the telemetry-based unknown output, and retraining the root cause-based machine learning model based on the machine learning-based unknown quantity.

[0017] In one example implementation, a system for generating device-specific remedial actions for fault, performance, and power issues can include at least one processor and physical memory including computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform actions including: generating rule-based root cause predictions and telemetry-based unknown outputs based on telemetry data from a client computing device, generating machine learning-based root cause predictions by applying a root cause machine learning model to the telemetry-based unknown outputs, and generating one or more root cause recommendations specific to the client computing device based on the rule-based root cause outputs and the machine learning-based root cause predictions.

[0018] In some exemplary implementations, the above methods can be encoded as computer-readable instructions on a non-transitory computer-readable medium. For example, the computer-readable medium can include one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to generate rule-based root cause outputs and telemetry-based unknown outputs based on telemetry data from a client computing device, generate machine learning-based root cause predictions by applying a root cause machine learning model to telemetry-based known outputs, and generate one or more root cause recommendations specific to the client computing device based on the rule-based root cause outputs and the machine learning-based root cause predictions.

[0019] According to the general principles described herein, the features of any specific implementation described herein can be used in combination with each other. These and other specific implementations, features and advantages will be more fully understood after reading the following detailed description in conjunction with the accompanying drawings and claims.

[0020] The following will refer to Figures 1 to 3 A detailed description of example systems and methods for generating resiliency and device-specific remediation actions for failure, performance, and power issues is provided. For example, Figure 1 A detailed description of the corresponding computer-implemented method is provided. In addition, Figure 2 An example overview sequence diagram of a remediation action recommendation system that generates device-specific remediation action recommendations that address one or more root causes of a fault, power, or performance issue indicated by telemetry data associated with a client computing device is shown. In addition, Figure 3 An example detailed sequence diagram of the same process is shown. Finally, Figure 4 Detailed diagrams of the systems and methods described herein are shown.

[0021] Figure 1 is a flow chart of an example computer-implemented method 100 for generating resiliency and device-specific remedial action recommendations that address root causes of failures, performance, and power issues. Figure 1 The steps shown in can be performed by any suitable computer executable code and / or computer hardware. In one example, Figure 1 Each of the steps shown in can represent an algorithm whose structure includes and / or is represented by multiple sub-steps, examples of which are provided in more detail below.

[0022] like Figure 1As shown, at step 102, the remedial action recommendation system can generate a rule-based root cause output and a telemetry-based unknown output based on telemetry data from the client computing device. For example, the remedial action recommendation system can receive telemetry data from any one of a plurality of telemetry sources (e.g., including the client computing device and other telemetry sources associated with the client computing device), such as offline logs and / or other real-time sources. In one or more specific implementations, the remedial action recommendation system can parse, process, and otherwise transform the received telemetry data based on one or more models. For example, the remedial action recommendation system can parse the telemetry data based on a rule-based model and based on a model based on root cause machine learning.

[0023] In at least one specific implementation, the remediation recommendation system can apply a rule-based model to the parsed telemetry data to generate a rule-based root cause output and a telemetry-based unknown output. For example, the rule-based model can include one or more of rules, heuristics, lookup tables, decision trees, etc. Therefore, in at least one specific implementation, the rule-based model generates a rule-based root cause output by applying the rule-based model to the parsed telemetry data, and the parsed telemetry data includes known or expected patterns, values, features, attributes, etc. Therefore, in some specific implementations, the telemetry-based unknown output can include the remaining telemetry data that the rule-based model cannot process - possibly including complex patterns and / or noisy data.

[0024] In addition, at step 104, the remedial action recommendation system can generate a machine learning-based root cause prediction by applying the root cause machine learning model to the unknown output based on telemetry. For example, in at least one specific implementation, the remedial action recommendation system can apply the root cause machine learning model to the unknown output based on telemetry to generate a machine learning-based root cause prediction. In one or more specific implementations, the remedial action recommendation system can train the root cause machine learning model to predict the root causes of failures, performance, and power issues based on telemetry data that cannot be analyzed by rule-based models. Therefore, the root cause machine learning model is able to understand the deep failure modes presented by the telemetry data, which were previously unable to be analyzed under the rule-based approach, thereby predicting the root causes of various types of failures.

[0025] In addition, if Figure 1As further shown, at step 106, the remediation recommendation system can generate one or more root cause remediation recommendations specific to the client computing device based on the rule-based root cause output and the machine learning-based root cause prediction. For example, the remediation recommendation system can apply the recommendation engine to the root cause indicated by the rule-based root cause output and the machine learning-based root cause prediction to generate at least one root cause remediation recommendation specific to the client computing device. In one or more specific implementations, the remediation recommendation generated by the recommendation engine can include instructions specific to the client computing device, such as software updates, disabling problematic system features, enabling various system features, modifying or reconfiguring certain software or hardware features, etc.

[0026] In one or more specific implementations, the remedial action recommendation system described herein refers to a number of terms and phrases. For example, the term "root cause" can refer to an action performed by a client computing device that causes a problem. To illustrate, the root cause of a software failure problem can include a specific erroneous instruction executed by a client computing device. In some specific implementations, the root cause can also include an action performed by a client computing device that did not cause the problem but instead caused other actions that caused the problem (e.g., a cascading failure).

[0027] As used herein, the term "model" can refer to any set of computer instructions that, when applied to one or more inputs, generate one or more outputs. For example, a "rule-based model" can apply one or more rule-based heuristics (e.g., decision trees, if / then tables, graphs) that, when applied to one or more inputs comprising structured data in an expected format, generate one or more outputs. In addition, a "machine learning-based model" or "root cause machine learning model" can refer to a trained layer of weighted nodes that is trained to generate predictions at an acceptable confidence level when applied to one or more input vectors. The machine learning model can include one or more of a convolutional neural network, a recursive neural network, a feedforward neural network, or a perceptron neural network. In additional implementations, the machine learning model can include other types of artificial networks.

[0028] As used herein, "telemetry data" can refer to a collection of observed data values ​​associated with the functionality of a client computing device. For example, telemetry data can include register values, stack values, accelerator measurement values ​​(e.g., thermal measurements, clock speed measurements), recorded values ​​(e.g., debug statements, software write output), etc. Telemetry data can be collected at one or more telemetry sources. For example, telemetry data can be collected by one or more components of a client computing device (e.g., installed software), or can be collected by a third-party telemetry source (e.g., a service that monitors the client computing device or receives monitoring statements from the client computing device). Telemetry data can be collected, transmitted, and / or received in real time as well as collected, transmitted, and / or received after the fact (e.g., as with telemetry data from offline logs).

[0029] As used herein, "remediation recommendation" can refer to a generated output that includes information associated with a root cause of a problem experienced by a client computing device. For example, the remediation recommendation can include a complete description of the problem and the root cause of the problem. The remediation recommendation can also include instructions to address the root cause of the problem. For example, the remediation recommendation can include instructions to permanently address the problem (e.g., "update your game app to version xyz"), instructions to temporarily address the problem (e.g., "increase your fan speed by 70% to reduce thermal throttling"), or instructions to bypass the problem. In one or more specific implementations, the remediation recommendation can be formatted as an email, SMS message, pop-up window, or other type of notification. Additionally, in some specific implementations, the remediation recommendation can include an optional option for automatically fixing the identified problem.

[0030] As used herein, the term "user security level" can refer to a classification associated with an end user of a client computing device. For example, the systems and methods discussed herein can generate remediation recommendations based on the end user's user security level, such that the remediation recommendations do not include information that is too sensitive or inappropriate for the end user to view. An end user can have a higher user security level, such as "internal user" or "administrator." Conversely, an end user can have a lower user security level, such as "external user."

[0031] As described above, the remediation recommendation system is capable of efficiently and accurately generating remediation recommendations that are specific to a problem experienced by a particular client computing device. Figure 2 An overview sequence diagram of a remedial action recommendation system 202 (eg, the remedial action recommendation system described above) is shown that generates a remedial action recommendation 216 for a client computing device 204 based on telemetry data 206 associated with the client computing device 204 .

[0032] In more detail, the remedial action recommendation system 202 can receive the telemetry data 206 at the data block 208. In one or more specific implementations, the data block 208 can include one or more parsers, processors, translators, etc. For example, the components of the data block 208 can parse the received telemetry data 206 according to the analyzer block 210. In at least one specific implementation, the data block 208 can parse the telemetry data 206 according to one or more models within the analyzer block 210.

[0033] In one or more specific implementations, Figure 2 As further shown in FIG. 2 , the data block 208 can transmit the parsed telemetry data 206 to the analyzer block 210 for root cause analysis. In at least one specific implementation, the analyzer block 210 can include multiple models that process some or all of the parsed telemetry data 206 in parallel or sequentially. In addition, the analyzer block 210 can utilize the results of one model as some or all of the inputs in another model. In one or more specific implementations, the analyzer block 210 generates client-specific root cause results based on the results, outputs, and / or predictions generated by one or more models within the analyzer block 210.

[0034] Utilizing the client-specific root cause results of the analyzer block 210, the remediation recommendation system 202 can generate one or more root cause remediation recommendations. For example, the remediation recommendation system 202 can apply the recommendation engine 212 to the client-specific root cause results and a subset of the parsed telemetry data generated by the data block 208. In one or more specific implementations, the recommendation engine 212 can include one or more of a machine learning-based unit, a lookup table, rule-based recommendations, and other heuristics. Therefore, the remediation recommendation system 202 can apply the recommendation engine 212 to the client-specific root cause results and other telemetry data to generate a root cause remediation recommendation, which includes specific instructions for modifying, configuring, installing, disabling, enabling, and / or upgrading one or more components of the client computing device 204 to resolve specific faults, performance, and / or power issues experienced by the client computing device 204.

[0035] In at least one specific implementation, the client computing device 204 can be associated with a particular user security level. To illustrate, the client computing device 204 can be associated with a high security level (e.g., like an end user with administrative access rights to the client computing device 204) or a low security level (e.g., like a standard non-administrative end user). Thus, the remediation recommendation system 202 can generate filtered remediation recommendations 216 by applying the filtering engine 214 to one or more root cause remediation recommendations generated by the recommendation engine 212.

[0036] For example, the filtering engine 214 can determine the security level of the client computing device 204 based on the end-user data extracted from the telemetry data 206 by the data block 208. Based on the security level, the filtering engine 214 can filter one or more root cause remediation recommendations to include information and instructions appropriate for the security level. For example, the filtering engine 214 can filter one or more root cause remediation recommendations to include data appropriate for the security level, which may include one or more of a detailed failure explanation, a failure mode, and a client computing device-specific recommendation. The filtering engine 214 can then transmit the filtered remediation recommendations 216 to the client computing device 204.

[0037] Figure 3 Detailed diagram of remedial action recommendation system 202 is shown for generating one or more remedial action recommendations 216. For example, Figure 3 As shown, the remedial action recommendation system 202 is capable of receiving telemetry data 206 at an initial parser 302 of a data block 208. In one or more specific implementations, the initial parser 302 is capable of receiving raw and unstructured data from various sources, including offline logs and real-time telemetry sources. The initial parser 302 is also capable of separating, converting, processing, translating, and otherwise configuring the telemetry data 206 based on one or more sources of the telemetry data 206. For illustration, the initial parser 302 is capable of parsing telemetry data from a specific telemetry source based on knowledge of how the telemetry source patterns and / or structures its telemetry data. In addition, the initial parser 302 is also capable of extracting all information from the received telemetry data 206—even without knowing the format, pattern, and / or structure of the telemetry source. For example, the initial parser 302 can extract all information and attempt to tag and / or otherwise categorize the extracted data into a structured format that can be used by the rule-based parser 304, the machine learning-based parser 306, the analyzer block 210, the recommendation engine 212, and / or the filtering engine 214.

[0038] In one or more implementations, the data block 208 can also include a rule-based parser 304 and a machine learning-based parser 306. In at least one implementation, the rule-based parser 304 and the machine learning-based parser 306 can also parse the telemetry data 206 based on one or more models within the analyzer block 210. For example, the rule-based parser 304 can also parse the output of the initial parser 302 based on the input structure of the rule-based model 310 within the analyzer block 210. Similarly, the machine learning-based parser 306 can also parse the output of the initial parser 302 based on the input structure of the root cause machine learning model 312. For example, either or both of the rule-based parser 304 and the machine learning-based parser 306 can further parse the telemetry data 206 by extracting certain attributes, features, and values ​​(numerical and categorical) according to the rule-based model 310 and the root cause machine learning model 312, respectively. In some implementations, the rule-based parser 304 and the machine learning-based parser 306 can parse the output of the initial parser 302 in parallel. In other implementations, the rule-based parser 304 and the machine learning-based parser 306 can operate serially or sequentially.

[0039] like Figure 3 As further shown, the data block 208 can also include a data processor 308 that can receive outputs from the rule-based parser 304 and the machine learning-based parser 306. In one or more implementations, the data processor 308 can generate, enumerate, and encode various data patterns (e.g., sequential patterns with timing information, logical patterns, etc.). For example, the data processor 308 can perform these tasks by generating one or more of engineering features, data encoding (e.g., via one hot encoding), custom mapping, and / or data enhancement. In at least one implementation, the data processor 308 can generate any of these outputs based on attributes specific to the rule-based model and / or attributes specific to the root cause machine learning model.

[0040] As mentioned above, Figure 3As shown, the remedial action recommendation system 202 is capable of generating a root cause prediction at the analyzer block 210. In more detail, the rule-based model 310 is capable of receiving the converted telemetry data 206 from the data processor 308 of the data block 208. In one or more specific implementations, the rule-based model 310 generates a rule-based root cause output and a telemetry-based unknown output based on the received data. For example, the rule-based model 310 can include one or more of heuristics, lookup tables, decision trees, random forests, if / then statements, etc. associated with the potential root causes of various potential problems that the client computing device may experience. Therefore, the rule-based model 310 is capable of generating a rule-based root cause output by matching or otherwise associating the received data with these heuristics, lookup tables, decision trees, random forests, if / then statements, etc. In at least one specific implementation, the rule-based model 310 is capable of generating a telemetry-based unknown output based on a subset of the received data, which subsets are not related to any rule-based method embodied by the rule-based model 310.

[0041] In one or more specific implementations, as described above, the analyzer block 210 can also include a root cause machine learning model 312. In at least one specific implementation, the root cause machine learning model 312 can generate one or more machine learning-based root cause predictions based on telemetry-based unknown outputs. For example, the root cause machine learning model 312 can include a shallow neural network, such as a multilayer perceptron. In addition, the remediation recommendation system 202 can train the root cause machine learning model 312 to learn telemetry-based patterns, which can be complex and / or can include noisy data. In some specific implementations, in addition to the telemetry-based unknown outputs, the remediation recommendation system 202 can also apply the root cause machine learning model 312 to the rule-based root cause outputs from the rule-based model 310.

[0042] In addition, the analyzer block 210 can also include a post-processor 314 that further modifies one or more outputs of the rule-based model 310 and the root cause machine learning model 312. For example, in one implementation, the post-processor 314 can analyze the rule-based root cause outputs and / or the machine learning-based root cause predictions to check for erroneous outputs, null values, and other unexpected outputs. In addition, in some implementations, the post-processor 314 can modify the rule-based root cause outputs generated by the rule-based model 310 based on the machine learning-based root cause predictions generated by the root cause machine learning model 312. For illustration, the post-processor 314 can modify the rule-based root cause outputs based on the confidence level associated with the machine learning-based root cause predictions. Thereafter, the post-processor 314 can generate client-specific root cause results based on the machine learning-based root cause predictions and the possibly modified rule-based root cause outputs. In one or more implementations, the client-specific root cause results indicate specific components, settings, and / or configurations of the hardware and / or software of the client computing device 204 that caused the failure, power, or performance issue experienced by the client computing device 204.

[0043] The remediation recommendation system 202 can also generate one or more root cause remediation recommendations by applying the recommendation engine 212 to the client-specific root cause results generated by the post-processor 314 in the analyzer block 210 based on the rule-based root cause output and the machine learning-based root cause prediction. For example, in one or more specific implementations, the recommendation engine 212 can include one or more of the machine learning-based recommendation unit 316, the lookup table 318, and the rule-based recommendation 320. For example, the recommendation engine 212 can apply the machine learning-based recommendation unit 316 to the client-specific root cause results to generate one or more root cause remediation recommendations specific to the client computing device 204. Similarly, the recommendation engine 212 can associate the client-specific root cause results with the lookup table 318 and / or the rule-based recommendation 320 to further generate one or more root cause remediation recommendations.

[0044] In some implementations, the recommendation engine 212 can utilize the lookup table 318 and the rule-based recommendation 320 in response to determining that the machine-based recommendation unit 316 has generated a root cause remediation recommendation with a low confidence level. In addition, in some implementations, the recommendation engine 212 can combine the remediation recommendations output by the machine-based recommendation unit 316, the lookup table 318, and the rule-based recommendation 320 to generate a final root cause remediation recommendation. In additional implementations, the recommendation engine 212 can include fewer components or additional components (e.g., decision trees, random forests, etc.).

[0045] In at least one specific implementation, the recommendation engine 212 can utilize end-user specific data (e.g., from the data block 208) when generating the root cause remediation recommendations. To illustrate, the recommendation engine 212 can receive data from the data block 208 indicating that an end user of the client computing device 204 frequently overclocks an accelerated processor of the client computing device 204. In response to determining that overclocking may be a root cause of the problem being experienced by the client computing device 204, the recommendation engine 212 can incorporate this user behavior into the root cause remediation recommendations. For example, the recommendation engine 212 can generate a root cause remediation recommendation that includes a recommended accelerated processor clock setting for a user who does not overclock, but can generate a root cause remediation recommendation that includes a maximum accelerated processor clock setting for a user who may overclock. Thus, the recommendation engine 212 can generate root cause remediation recommendations that are not only device specific, but also specific to the user's observed behavior.

[0046] As described above, the remediation recommendation system 202 can also customize the remediation recommendations according to the security level associated with the client computing device 204. For example, the remediation recommendation system 202 can apply the end-user filter 322 to the root cause remediation recommendation associated with the end-user data to generate the final remediation recommendation 216. In more detail, the end-user filter 322 can receive the end-user data extracted by the data block 208. Based on the end-user data, the end-user filter 322 can determine the security level associated with the client computing device 204 (e.g., associated with the user of the client computing device 204). The end-user filter 322 can then filter, reconfigure, enhance or edit the root cause remediation recommendation generated by the recommendation engine 212 according to the determined security level. For example, the end-user filter 322 can add information to or remove information from the root cause remediation recommendation, including but not limited to: detailed fault explanation, fault mode, and detailed recommendations for modifying, configuring, bypassing one or more hardware and / or software components. The filtering engine 214 can also transmit the filtered root cause remediation recommendation to the client computing device 204.

[0047] In one or more implementations, the filtering engine 214 can also generate the final remediation recommendation 216 in various ways. For example, the filtering engine 214 can generate the final remediation recommendation 216 as part of a report (e.g., independently or within an application installed on the client computing device 204). In another implementation, the filtering engine 214 can generate the final remediation recommendation 216 as a notification (e.g., a pop-up window, a banner notification, a system log entry) of the client computing device 204. In addition, the filtering engine 214 can generate the final remediation recommendation 216 as an electronic message to a user of the client computing device 204. For example, the filtering engine 214 can generate the final remediation recommendation 216 as an email, a text message, a social media message, etc.

[0048] The remedial action recommendation system 202 described herein can be implemented in a variety of ways. Figure 4 4 shows a block diagram of an example implementation 400 of the remedial action recommendation system 202 discussed throughout. Figure 4 As shown, implementation 400 can include remedial action recommendation system 202 operating on server 408. As shown, remedial action recommendation system 202 can receive telemetry data from one or more of telemetry service 406 and telemetry application 402 installed on client computing device 204 via network 404.

[0049] In more detail, the remediation recommendation system 202 can operate as software or firmware and can include a parsing manager 410, a model manager 412, a recommendation manager 414, and a security filter manager 416. Figure 4 The components 410 to 416 of the remedial action recommendation system 202 are shown as separate components, but one or more of the components 410 to 416 of the remedial action recommendation system 202 can be combined in additional implementations. Similarly, in additional implementations, the remedial action recommendation system 202 can include additional, fewer, or different components.

[0050] In some implementations, the remedial action recommendation system 202 can represent one or more software applications or programs that, when executed by a processor, can cause the processor to perform one or more tasks. For example, as will be described in more detail below, one or more of the components 410 to 416 of the remedial action recommendation system 202 can represent software stored and configured to run on one or more computing devices. Figure 4 One or more of the components 410 through 416 of the remedial action recommendation system 202 shown in FIG. 2 can also represent all or part of one or more special-purpose computers configured to perform one or more tasks.

[0051] As mentioned above, Figure 4 As shown, the remedial action recommendation system 202 can include a parsing manager 410. In one or more specific implementations, the parsing manager 410 can perform any of the functions described above in conjunction with the data block 208. For example, the parsing manager 410 can include and / or guide the initial parser 302, the rule-based parser 304, the machine learning-based parser 306, and / or the data processor 308. Therefore, the parsing manager 410 can receive telemetry data from one or more sources (for example, including the telemetry application 402, the telemetry service 406, and other applications and services). In at least one specific implementation, the parsing manager 410 can generate and provide one or more interfaces that enable additional configuration of one or more of the initial parser 302, the rule-based parser 304, the machine learning-based parser 306, and the data processor 308. For illustration, the parsing manager 410 can enable reconfiguration of one or more of the components of the data block 208. Parse manager 410 can also allow one or more additional components to be included within data block 208 , or can allow one or more of the components of data block 208 to be deleted.

[0052] In addition, if Figure 4 As shown, the remediation recommendation system 202 can include a model manager 412. In one or more specific implementations, the model manager 412 can generate, train, retrain one or more models, and apply the one or more models to various inputs. For example, the model manager 412 can generate a rule-based model 310 within the analyzer block 210, including one or more lookup tables, decision trees, decision graphs, random forests, and other heuristics. In addition, the model manager 412 can generate and train a root cause machine learning model 312. For example, the model manager 412 can generate and train a multilayer perceptron to learn complex patterns within noisy telemetry data. In addition, the model manager 412 can generate and train a machine learning-based recommendation unit 316 of the recommendation engine 212. For example, the model manager 412 can train a machine learning-based recommendation unit 316 to generate root cause remediation recommendations based on client-specific root cause results generated by the post-processor 314 of the analyzer block 210 and end-user behavior data extracted by the data block 208.

[0053] In at least one specific implementation, the model manager 412 can periodically retrain one or more of the models used by the remediation recommendation system 202. For example, in one specific implementation, the root cause machine learning model 312 can detect failures in a subset of unknown outputs based on telemetry (e.g., improved by up to 25%). In this specific implementation, the model manager 412 can retrain the root cause machine learning model 312 with the remaining portion of the unknown outputs based on telemetry. In addition, the model manager 412 can retrain the root cause machine learning model 312 based on user feedback associated with the final remediation recommendation 216 (e.g., whether the final remediation recommendation 216 is accurate, whether the final remediation recommendation 216 correctly solves the problem).

[0054] Additionally, as described above, the remediation recommendation system 202 can include a recommendation manager 414. In one or more specific implementations, the recommendation manager 414 generates, maintains, and applies the lookup table 318 and the rule-based recommendations 320 within the recommendation engine 212. Additionally, the recommendation manager 414 can generate and provide one or more interfaces that allow for reconfiguration and additions associated with the lookup table 318 and the rule-based recommendations 320. For example, the recommendation manager 414 can enable reconfiguration of the lookup table 318 and / or the rule-based recommendations 320 to include updated rules and remediation recommendations.

[0055] As also described above, the remediation recommendation system 202 can include a security filter manager 416. In one or more specific implementations, the security filter manager 416 can generate, maintain, and apply the final user filter 322 of the filtering engine 214. For example, the security filter manager 416 can generate the final user filter 322, which includes the correlation between the user security level and the allowed remediation recommendation information for each user security level. In at least one specific implementation, the security filter manager 416 can generate the final remediation recommendation 216 in any of a variety of formats (e.g., log entry, notification, email).

[0056] like Figure 4 As shown, the client computing device 204 can include a telemetry application 402 or be installed with the telemetry application. In one or more specific implementations, the telemetry application 402 can be a component-specific monitoring application (e.g., an application that monitors and records telemetry data associated with an accelerated processor). In addition, the telemetry application 402 can be a third-party application that monitors multiple system components (e.g., hardware and / or software components) to record and transmit telemetry data. In some specific implementations, the telemetry application 402 can transmit telemetry data to the remediation recommendation system 202 in real time or via one or more logs or other reports.

[0057] In addition, if Figure 4 As shown, implementation 400 can include telemetry service 406. In one or more implementations, telemetry service 406 receives, requests, monitors, and / or transmits telemetry data associated with client computing device 204. In some implementations, telemetry service 406 can transmit telemetry data to remediation recommendation system 202 via telemetry application 402, and vice versa. In some implementations, telemetry service 406 can transmit telemetry data to remediation recommendation system 202 in real time or via one or more logs or other reports.

[0058] Client computing device 204, telemetry service 406, and server 408 can be communicatively coupled to each other via network 404. Network 404 can represent any type or form of communications network, such as the Internet, and can include one or more physical connections, such as a LAN, and / or wireless connections, such as a WAN.

[0059] although Figure 4 An implementation 400 (e.g., a cloud-based implementation) of the remedial action recommendation system 202 is shown including a server 408, but other arrangements are possible. For example, the remedial action recommendation system 202 can be implemented on a client computing device 204. In addition, the remedial action recommendation system 202 can be implemented within a local network (e.g., a WAN or LAN) to receive telemetry data from a group of client computing devices (e.g., via Ethernet or Wi-Fi). In addition, in some implementations, the remedial action recommendation system 202 can only receive telemetry data from the telemetry application 402 or the telemetry service 406, but not from both at the same time.

[0060] Thus, as described throughout, the remediation recommendation system 202 provides a solution to the rigid and often inaccurate remediation recommendations generated by previous systems. For example, by implementing a rule-based approach that further enhances the machine learning-based approach, the remediation recommendation system 202 provides a resilient pipeline that is able to detect a significantly higher percentage of faults in noisy telemetry data. Because the remediation recommendation system 202 is more tolerant of noisy data, the remediation recommendations generated by the remediation recommendation system 202 are more accurate for deep and complex root causes of problems. In addition, because the remediation recommendation system 202 provides a fully automated solution to the generation of remediation recommendations, these remediation recommendations can be quickly generated in minutes, rather than days or even weeks as in previous systems.

[0061] The process parameters and order of steps described and / or shown herein are provided as examples only and can be changed as desired. For example, although the steps described and / or shown herein can be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order shown or discussed. The various exemplary methods described and / or shown herein can also omit one or more of the steps described or shown herein, or include additional steps in addition to those disclosed.

[0062] Although various implementations have been described and / or shown herein in the context of a fully functional computing system, one or more of these exemplary implementations can be distributed as a program product in various forms, regardless of the specific type of computer-readable medium used to actually perform the distribution. The implementation disclosed herein can also be implemented using modules that perform specific tasks. These modules can include scripts, batch processing, or other executable files that can be stored on a computer-readable storage medium or stored in a computing system. In some implementations, these modules can configure a computing system to perform one or more of the exemplary implementations disclosed herein.

[0063] The foregoing description has been provided to enable others skilled in the art to best utilize various aspects of the example implementations disclosed herein. This example description is not intended to be exhaustive or limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the present disclosure. The implementations disclosed herein should be considered in all respects to be illustrative and not restrictive. In determining the scope of the present disclosure, reference should be made to the appended claims and their equivalents.

[0064] Unless otherwise indicated, the terms "connected to" and "coupled to" (and their derivatives) as used in the specification and claims will be deemed to allow both direct and indirect (i.e., via other elements or components) connections. In addition, the terms "a" or "an" as used in the specification and claims will be deemed to mean "at least one". Finally, for ease of use, the terms "including" and "having" (and their derivatives) as used in the specification and claims are interchangeable with the word "including" and have the same meaning as the word "comprising".

Claims

1. A computer-implemented method comprising: generating rule-based root cause outputs and telemetry-based unknown outputs based on telemetry data from client computing devices; generating a machine learning-based root cause prediction by applying a root cause machine learning model to the telemetry-based unknown output; as well as Based on the rule-based root cause outputs and the machine learning-based root cause predictions, one or more root cause remediation recommendations specific to the client computing device are generated.

2. The computer-implemented method of claim 1 , further comprising, before generating the rule-based root cause output and the telemetry-based unknown output: receiving the telemetry data from the client computing device; and The telemetry data is parsed based on a rule-based model and the root cause machine learning model. 3 . The computer-implemented method of claim 2 , wherein receiving the telemetry data from the client computing device comprises receiving raw and unstructured data from one or more of an offline log and a real-time telemetry source.

4. The computer-implemented method of claim 3, wherein parsing the telemetry data based on the rule-based model and the root cause machine learning model comprises: parsing the telemetry data into structured data based on one or more of the offline log and the real-time telemetry source; parsing the structured data into attributes specific to the rule-based model; and The structured data is parsed into attributes specific to the root cause machine learning model.

5. The computer-implemented method of claim 4, wherein parsing the telemetry data based on the rule-based model and the root cause machine learning model further comprises generating one or more of engineering features, data encodings, custom mappings, and data augmentations based on the attributes specific to the rule-based model and the attributes specific to the root cause machine learning model.

6. The computer-implemented method of claim 1 , further comprising: modifying the rule-based root cause output based on the machine learning-based root cause prediction; as well as Client-specific root cause results are generated based on the machine learning-based root cause predictions and the modified rule-based root cause outputs.

7. The computer-implemented method of claim 6, wherein the one or more root cause remediation recommendations specific to the client computing device are generated by applying a recommendation engine to the client-specific root cause results.

8. The computer-implemented method of claim 7, wherein the recommendation engine comprises a plurality of trained machine learning-based recommendation units.

9. The computer-implemented method of claim 8, wherein the recommendation engine further comprises one or more of a lookup table or rule-based recommendations.

10. The computer-implemented method of claim 2, further comprising generating filtered remediation recommendations customized for a user security level associated with the client computing device based on the one or more root cause remediation recommendations specific to the client computing device and end user data extracted from the telemetry data from the client computing device.

11. The computer-implemented method of claim 1 , further comprising: generating a machine learning-based unknown quantity by applying the root cause machine learning model to the telemetry-based unknown output; as well as The root cause machine learning model is retrained based on the machine learning unknowns.

12. A system comprising: at least one processor; and a physical memory comprising computer executable instructions that, when executed by the at least one processor, cause the at least one processor to perform actions comprising: generating rule-based root cause outputs and telemetry-based unknown outputs based on telemetry data from client computing devices; generating a machine learning-based root cause prediction by applying a root cause machine learning model to the telemetry-based unknown output; as well as Based on the rule-based root cause outputs and the machine learning-based root cause predictions, one or more root cause remediation recommendations specific to the client computing device are generated.

13. The system of claim 12, further comprising computer executable instructions that, when executed by the at least one processor, cause the at least one processor to perform actions including the following before generating the rule-based root cause output and the telemetry-based unknown output: receiving the telemetry data from the client computing device; and The telemetry data is parsed based on a rule-based model and the root cause machine learning model.

14. The system of claim 13, wherein receiving the telemetry data from the client computing device comprises receiving raw and unstructured data from one or more of an offline log and a real-time telemetry source.

15. The system of claim 14, wherein parsing the telemetry data based on the rule-based model and the root cause machine learning model comprises: parsing the telemetry data into structured data based on one or more of the offline log and the real-time telemetry source; parsing the structured data into attributes specific to the rule-based model; and The structured data is parsed into attributes specific to the root cause machine learning model.

16. The system of claim 15, wherein parsing the telemetry data based on the rule-based model and the root cause machine learning model further comprises generating one or more of engineering features, data encoding, custom mapping, and data augmentation based on the attributes specific to the rule-based model and the attributes specific to the root cause machine learning model.

17. The system of claim 12, further comprising computer executable instructions that, when executed by the at least one processor, cause the at least one processor to perform actions comprising: modifying the rule-based root cause output based on the machine learning-based root cause prediction; and Client-specific root cause results are generated based on the machine learning-based root cause predictions and the modified rule-based root cause outputs.

18. The system of claim 17, wherein the one or more root cause remediation recommendations specific to the client computing device are generated by applying a recommendation engine to the client-specific root cause results.

19. The system of claim 18, wherein the recommendation engine comprises a plurality of trained machine learning-based recommendation units.

20. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to: generating rule-based root cause outputs and telemetry-based unknown outputs based on telemetry data from client computing devices; generating a machine learning-based root cause prediction by applying a root cause machine learning model to the telemetry-based unknown output; as well as Based on the rule-based root cause outputs and the machine learning-based root cause predictions, one or more root cause remediation recommendations specific to the client computing device are generated.

Images