Test result classification for failed code verification

Abstract

Disclosed herein are systems, methods, and software for classifying test results of failed code validation. Test results indicating failed code validation performed in a client device are received. Failure data indicating one or more failure modes occurring during the failed code validation are generated based on the test results. Subsequent actions associated with each of the one or more failure modes are determined by comparing each of the one or more failure modes with previously identified failure modes. Instructions are transmitted to the client device based on the determined subsequent actions associated with each of the one or more failure modes.

Description

Technical Field

[0001] Aspects of the present disclosure relate to computing hardware and software techniques, and in particular to classifying test results of failed code validation. Background Art

[0002] Modern electronic devices such as computers, tablet computers, mobile phones, wearable devices, etc. have become a common part of modern life. Many users of electronic devices routinely utilize various types of software applications for business and personal activities. Examples of software applications can include word processors, spreadsheet applications, email clients, note taking software, presentation applications, games, computing software, etc. These software applications can also be used to perform calculations, generate charts, organize data, receive and send emails, communicate with others in real time, etc. Software applications can range from simple software to very complex software.

[0003] The development of complex software is often performed by a team of software developers. When software code has been added or modified, tests are usually run against a controlled environment. These tests provide feedback on whether the output of the code is as expected and repeatable. If the test passes, the test indicates that there are no problems with the code. If the test fails, the test indicates that the code contains errors.

[0004] In many cases, tests may fail due to undetermined reasons unrelated to possible code configurations, such as delayed virtual machines, poor network connections, timeouts, security verification failures, etc. These uncertain tests (also known as flaky tests) typically require a large amount of developer resources to investigate and classify failures that occur during testing. In addition, test failures typically include a variety of different failure modes that require different responses. For example, one failure mode may be the result of a poor network connection, which only requires retrying the test. However, another failure mode may be the result of a hard failure, which should not be retried.

[0005] Previous solutions to this problem include maintaining basic data about whether a test has a basic non-deterministic state (i.e., a flaky state). However, this solution does not provide any feedback about different failure modes. Each of these failure modes may have a root cause. However, a test is not determined to be non-deterministic until the developer manually parses the code and determines that there is no problem with the code when the test fails. Unfortunately, previous solutions require many resources to only determine that a test is non-deterministic, without any additional information about what the various failure modes are, let alone how to respond to the various failure modes.

[0006] In addition, previous solutions treat all test cases with the same name the same. This can be very inefficient when running test cases with the same name in various environments. For example, a test case running on an operating system (OS) typically has a different failure mode than a test case with the same name running on a central processing unit (CPU) architecture. Unfortunately, previous solutions do not treat these failure modes differently for test cases with the same name but running in different environments. This limits the ability of the application to enhance the developer's experience when investigating and responding to failed code tests. Summary of the invention

[0007] Disclosed herein are enhanced systems, methods, and software for improving the classification of test results for failed code validation. Test results indicating failed code validation performed in a client device are received. Failure data indicating one or more failure modes that occurred during the failed code validation are generated based on the test results. Then, subsequent actions associated with each of the one or more failure modes are determined by comparing each of the one or more failure modes with previously identified failure modes. Instructions are transmitted to the client device based on the determined subsequent actions associated with each of the one or more failure modes.

[0008] This summary is provided to introduce a selection of concepts in simplified form that are further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the description that follows and in part will become apparent from the description, or may be learned by practice of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Many aspects of the present disclosure can be better understood with reference to the following drawings. Although several implementations are described in conjunction with these drawings, the present disclosure is not limited to the implementations disclosed herein. On the contrary, it is intended to cover all alternatives, modifications and equivalents.

[0010] Figure 1 An operational architecture for implementing an enhanced application to triage test results for failed code validation is illustrated.

[0011] Figure 2 An indication process employed in the implementation of an enhancement application for triaging test results of failed code validation is illustrated.

[0012] Figure 3An operational scenario in an implementation of an enhancement application is illustrated to illustrate an exemplary user interface in an implementation of the enhancement application for triaging test results of failed code validation.

[0013] Figure 4 An exemplary user interface in an implementation of an enhanced application for triaging test results of failed code validation is illustrated.

[0014] Figure 5 A flow chart in the implementation of an enhancement application for detecting failure modes in reliability operations is illustrated.

[0015] Figure 6 A flow chart in an implementation of an enhancement application for matching failures in a modified code verification test with known failures in a reliability run is illustrated.

[0016] Figure 7 The diagram illustrates a computing system suitable for implementing the techniques disclosed herein, including any of the architectures, processes, operating scenarios, and operating sequences illustrated in the figures and discussed in the technical description that follows. DETAILED DESCRIPTION

[0017] An example of the present disclosure describes an application for classifying test results of failed code verification. A test result indicating failed code verification performed in a client device is received. Failure data is generated based on the test result, and the failure data indicates one or more failure modes that occurred during the failed code verification. Then, a subsequent action associated with each of the one or more failure modes is determined by comparing each of the one or more failure modes with a previously identified failure mode. Instructions are transmitted to the client device based on the determined subsequent actions associated with each of the one or more failure modes.

[0018] The technical effect that can be appreciated from this discussion is to improve the efficiency of automatically investigating multiple failure modes in failing tests. The application described in this article also improves the efficiency of determining whether a failure mode is known, and if known, how to best respond to it, such as by preventing retries or by allowing retries. Advantageously, developers do not need to manually parse test results to determine why a test failed and what the various test failures are, which may take a lot of time when a large number of failure modes occur.

[0019] With reference to the accompanying drawings, Figure 1An exemplary operational architecture 100 is illustrated in connection with processing operations for managing an exemplary enhanced system, with which aspects of the present disclosure may be practiced. The operational architecture 100 includes a failure mode service 101 connected to client devices 110-112. The failure mode service 101 is also connected to a data repository 120. The failure mode service 101 also includes a failure classification process 200.

[0020] Failure mode service 101 provides an organization for categorizing test results of failed code verification in software applications. Failure mode service 101 can receive test results from client devices 110-112. Failure mode service 101 can also query data repository 120 for known failure modes and their associated subsequent actions. Failure mode service 101 can be implemented on one or more computing systems, Figure 7 The computing system 700 in FIG. 1 is representative thereof. The failure mode service 101 provides an improvement in efficiency because previous failure mode data can be used to automatically identify and classify various failure modes included in the test results. The failure mode service 101 additionally improves the efficiency of automatically determining how to respond to each failure mode (e.g., allowing or preventing retries) in the best manner.

[0021] Failure mode service 101 represents any device capable of running an application locally or streaming an application in the context of a web browser or executing an application in any other manner. Examples of failure mode service 101 include, but are not limited to: a server, a personal computer, a mobile phone, a tablet computer, a desktop computer, a laptop computer, a wearable computing device, or any other form factor, including any combination of computers or variations thereof. Failure mode service 101 may include various hardware and software elements in a support architecture suitable for performing failure classification process 200.

[0022] The failure mode service 101 also includes a software application or application component capable of classifying the failure mode of the failed test verification according to the process described herein. The software application can be implemented as a locally installed and executed application, a web application hosted in the context of a browser, a streaming or streaming application, a mobile application, or any variation or combination thereof.

[0023] More specifically, Figure 2 The failure classification process 200 is illustrated, which, as mentioned, can be used by the failure mode service 101 to classify the failure mode of the failed test verification as described herein. Some or all of the steps of the failure classification process 200 can be implemented in program instructions in the context of one or more components of the application for executing the indicated characteristics. The program instructions direct the failure mode service 101 to operate as follows, as attached in Figure 1 Reference in context Figure 2 steps.

[0024] In operation, the failure mode service 101 receives test results 130 indicating failed code verification performed in the client device 110 (step 201). The test can be run against known and controlled code, such as the current code base. In some scenarios, these tests can be referred to as reliability runs. The test can be run regularly or in response to specific changes to the code. For example, developers can change the code and verify their changes to the code. When there is a failure in the test when running against known controlled code, the test can be considered to be in a known failure state. These test results 130 can then be sent to a server or server system, such as the failure mode service 101, to analyze for persistent failures. Test results 130-132 include test descriptions indicating whether the test passed or failed.

[0025] In some examples, the failure mode service 101 may also determine that the duration of the failed code verification exceeds the code verification timer or the number of failure modes exceeds the failure mode threshold and responsively transmit instructions to prevent the retry of the failed code verification performed in the client device. For example, if the number of failure modes includes more than 3% of the test results, it may be determined that the test has failed catastrophically and the analysis of the failure mode may be terminated.

[0026] In the next operation, the failure mode service 101 generates failure data indicating one or more failure modes that occurred during the failed code verification based on the test results 130 (step 202). To generate the failure data, the failure mode service 101 may first normalize the test results. The normalization may include cleansing the test results by removing certain details from the test results and then hashing the normalized code string from the test results into the failure data. The failure data may include 32 bits of DNA. This hashing of the test results into the failure data allows failure modes that occur at different times, different versions of code, different machines, etc. to be identified and analyzed together.

[0027] Advantageously, the failure data can indicate whether different failure patterns are repeated in different runs, even when the call stacks are slightly different or when the test is run on multiple machines. For example, the machine name can be part of the test description. The machine name can then be sanitized, which allows the failure pattern to look similar enough to determine that the failure is the same as another in the test results.

[0028] In a next operation, the failure mode service 101 determines a subsequent action associated with each of the one or more failure modes by comparing each of the one or more failure modes to previously identified failure modes (step 203). To compare the one or more failure modes, the failure mode service 101 may query a data repository, such as the data repository 120 for previously identified failure modes. The failure mode service 101 may query the data repository 120 by transmitting the one or more failure modes to the data repository 120 for comparison, by receiving previously stored failure mode data from the data repository 120, or by transmitting the one or more failure modes and previously identified failure modes to an external server system or application service for comparison.

[0029] In some exemplary scenarios, one or more failure modes may be determined to be known failures. For example, when a developer makes a change to the code and determines that the failure mode is known, it can be determined that the failure of the test verification may not be caused by the change to the code, but may be caused by some factors external to the code. In this exemplary scenario, the subsequent action associated with one or more failure modes will likely be to retry the test, because the failure is not the result of a change to the code itself.

[0030] In other scenarios, the failure mode may be determined to be an unknown failure. For example, when the failure mode does not match any previously recorded failure mode, the reason why the test failed is unknown, and the failure may be caused by a change to the code. In this exemplary scenario, subsequent actions associated with one or more failure modes will prevent any retry of the test. Advantageously, this allows the test to not be retried when the probability of success is low. This not only saves computing time and resources, but also allows the test to reach the retry limit unnecessarily. For example, a test may be allowed to be retried four times before being blocked. However, if the same failure mode is identified twice, additional retries for that failure mode may be blocked before the retry limit is reached.

[0031] In one example, failure data (e.g., DNA) run in one test may be compared to failure data that occurs in any other test. For example, the same failure pattern may be identified for a large number of tests when a login server is down. It may then be determined that the failure occurred due to the login server being down and not due to changes to the code itself. However, in other examples, the failure pattern or patterns may not be compared to any other test and must match the failure pattern for that particular test in order to be considered known.

[0032] In some implementations, the same test case can be used to test in different environments. The test case can be run against a production environment, an internal environment, or some other type of environment. In some examples, failure modes for different environments can be compared to identify previously known failure modes. For example, and Microsoft Run the same tests. Although each environment has a different type of backend, it has the same application program interface (API). However, in other cases, it may not allow failure modes from one environment to be compared with failure modes from another environment, even when running the same test case. For example, failure modes may be very specific to online applications, which should not be compared with native applications. Advantageously, this allows failure modes to occur in a specific environment to be clustered and identified.

[0033] In some examples, a subsequent action associated with each of the one or more failure modes can be determined by comparing each of the one or more failure modes with a previously identified failure mode having a corresponding OS type. In other examples, a subsequent action associated with each of the one or more failure modes can be determined by comparing each of the one or more failure modes with a previously identified failure mode having a corresponding CPU type. In yet another example, a subsequent action associated with each of the one or more failure modes can be determined by comparing each of the one or more failure modes with a previously identified failure mode having a corresponding network-based platform type. It should be noted that by including an indicator of the test case environment in the test name, the test case can be run for each of the different OS types, CPU types, and network-based platform types.

[0034] In the next operation, the failure mode service 101 transmits instructions to the client device 110 based on the determined subsequent actions associated with each of the one or more failure modes (step 204). The instructions may be transmitted to the client device 110 using a retry filter in the test tool. In some scenarios, the instructions may include instructions for retrying failed code validation performed in the client device 110. In other scenarios, the instructions may include instructions for preventing the retry of failed code validation performed in the client device 110 for one or more failure modes.

[0035] It should be noted that in some scenarios, different failure modes may have different instructions. For example, the test may be retried for some failure modes, while the test may be blocked for other failure modes. In another scenario, the instructions may instruct the client device 110 to retry the code verification in various ways based on previously identified failure modes. For example, if the test continues to fail on a specific login server, the failure mode service 101 may instruct the client device 110 to retry the code verification on another login server. This may allow successful retries in other ways when the root cause of the failure mode has been determined, and may be mitigated using alternating retry processes.

[0036] Figure 3 An operational architecture in an implementation of an enhanced application for classifying failure modes of failed code validation is illustrated. Figure 3 An operational scenario 300 is illustrated relating to what occurs when a classification engine provides a classification for each of the failure modes based on the stored failure modes. The operational scenario 300 includes a failure mode service 301, a virtual machine 310, a data repository 320, a classification engine 330, and a retry filter 340. The virtual machine 310 runs tests on both a reliability run and a developer code verification run to collect test results. The data repository 320 maintains a data structure of previously reported failure modes. The classification engine 330 classifies each of the failure modes as a known or unknown failure mode. In some examples, the classification engine 330 may be a Microsoft Azure Active Directory authentication service. Retry filter 340 processes instructions as to whether a test should be retried or blocked.

[0037] As illustrated in the data structure for the data repository 320, each stored failure mode can be associated with an environment, a state, and a subsequent action. For example, failure mode A is associated with the OS type, is a known failure, and the subsequent action is to retry the test. On the other hand, failure mode B is associated with the CPU type, is an unknown failure, and the subsequent action indicates that the retry should be blocked for the failure. In the final failure mode C, the failure mode is associated with the online platform environment and is known. The subsequent instructions associated with failure mode C indicate that the test should be retried on another server. This may be due to many uncertain tests generated by server failures.

[0038] Figure 4 A sequence diagram in an implementation of an enhanced application for classifying failure modes of failed code validation is illustrated. Figure 4The sequence diagram of illustrates the operations performed according to the components previously described in the operational scenario 300. In the first operation, the failure mode service 301 receives the test results from the virtual machine 310. The failure mode service 301 then hashes the test results to generate failure data indicating various failure modes. Assuming that the test does not time out or exceed the failure percentage threshold, the failure mode service 301 requests the classification engine 330 to classify each of the failure modes.

[0039] In order to classify each of the failure modes, the classification engine 330 queries the data repository 320 for similar stored failure modes. The failure mode is then compared to similar stored failure modes to determine whether each of the failure modes is known or unknown. The classification is then transmitted to the failure mode service 301. If the classification of the failure mode is known, the failure mode service 301 transmits an instruction to the retry filter 340, indicating that the retry filter 340 should instruct the virtual machine 310 to retry the test for the known failure mode. If the classification of the failure mode is unknown, the failure mode 301 transmits an instruction to the retry filter 340, indicating that the retry filter 340 should prevent any retries of the test for the unknown failure mode. Although in Figure 4 Not illustrated, but the failure mode will then be iterated and stored in the data repository 320 .

[0040] Figure 5 A flow chart in an implementation of an enhanced application for detecting failure modes in a reliability run is illustrated. In a first step, a reliability run is performed for post-verification analysis (step 501). The reliability run may be a CIT test. After verification of the reliability run has been completed, the test results are run through a circuit breaker. It is then determined whether the test has a high failure rate (step 502). If the test takes a significant amount of time to complete or if there are a significant number of failure modes occurring, it may be determined that the test has a high failure rate. If there is a high failure rate, it is assumed that there is a catastrophic failure of the test (e.g., a network shutdown or power outage) and the test results of the reliability run are ignored (step 503).

[0041] If there is no high failure rate, each failure is iterated (step 504). In the next step, reliability errors are archived for each failure (step 505). The automatic listening technology can deduplicate each of these failures based on the test failure DNA. Therefore, duplicate instances of the failure are not opened. Alternatively, an annotation can be added to the existing failure indicating that an additional failure has occurred. If the DNA of the reliability run matches the DNA of the existing run, it is marked as flaky (step 506).

[0042] Thus, a known failure that is not caused by the code may be marked as a flaky or non-deterministic test. Advantageously, the service can automatically determine that a test is flaky when similar failures occur across multiple developers. Thus, the same error is not investigated over and over again when the failure is known based on comparisons to other reliability run results.

[0043] In addition, the service investigates previously archived reliability errors and determines when the failure has been fixed (step 507). If the failure has been fixed, it is removed from the archived reliability data structure and marked as no longer flaky (508). This allows the database to be up to date as to which failures are still considered flaky (i.e., known failures) and which failures are determined to be unknown failures. It should be noted that after a failure has been marked as resolved, if the same error reappears, it will be marked as a new failure in the database because the previously marked failure has been considered fixed.

[0044] Figure 6 A flow chart in an implementation of an enhanced application for matching failures in a changed code verification test with known failures in a reliability run is illustrated. In a first step, a pull request (PR) verification run is performed for post-verification analysis (step 601). When the verification of the PR run has been completed, the test results will be run through a circuit breaker. Then, it is determined whether the test has a high failure rate (step 602). If the test takes a lot of time to complete or if there are a lot of failure modes occurring, it can be determined that the test has a high failure rate. If there is a high failure rate, it is assumed that there is a catastrophic failure of the test (e.g., a network shutdown or power outage) and the test results of the reliability run are ignored (step 603).

[0045] If there is no high failure rate, each failure is iterated (step 604). If there is a known failure, it is marked as flaky (step 605). Thus, a known failure that is not caused by a change to the code can be marked as a flaky or indeterminate test. Advantageously, when a known failure occurs, the service can automatically determine that the test is flaky. The code change can then be allowed because the change is not the cause of the failure. If there is no known failure, the verification suite fails and the change is blocked (607).

[0046] Figure 7A computing system 701 is illustrated, which represents any system or visual representation of a system in which the various applications, services, scenarios, and processes disclosed herein may be implemented. Examples of computing system 701 include, but are not limited to: server computers, rack servers, network servers, cloud computing platforms, and data center equipment, as well as any other type of physical or virtual server machines, containers, and any variations or combinations thereof. Other examples may include smartphones, laptops, tablet computers, desktop computers, hybrid computers, gaming consoles, virtual reality devices, smart televisions, smart watches and other wearable devices, and any variations or combinations thereof.

[0047] The computing system 701 may be implemented as a single device, system or apparatus, or may be implemented as multiple devices, systems or apparatuses in a distributed manner. The computing system 701 includes, but is not limited to, a processing system 702, a storage system 703, software 705, a communication interface system 707, and a user interface system 709. The processing system 702 is operably coupled to the storage system 703, the communication interface system 707, and the user interface system 709.

[0048] Processing system 702 loads and executes software 705 from storage system 703. Software 705 includes process 706, which represents the above Figure 1-6 The processes discussed include the failure classification process 200. When executed by the processing system 702 to enhance the application, the software 705 directs the processing system 702 to operate as described herein at least for the various processes, operating scenarios, and sequences discussed in the foregoing implementations. The computing system 701 may optionally include additional devices, features, or functions that are not discussed for the sake of brevity.

[0049] Still reference Figure 7 , processing system 702 may include a microprocessor and other circuitry to retrieve and execute software 705 from storage system 703. Processing system 702 may be implemented within a single processing device, but may also be distributed across multiple processing devices or subsystems that collaborate to execute program instructions. Examples of processing system 702 include general-purpose central processing units, graphics processing units, special-purpose processors and logic devices, and any other type of processing device, combination, or variation.

[0050] The storage system 703 may include any computer-readable storage medium that is readable by the processing system 702 and capable of storing the software 705. The storage system 703 may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Examples of storage media include random access memory, read-only memory, disk, optical disk, flash memory, virtual memory and non-virtual memory, cassettes, tapes, disk storage devices or other magnetic storage devices, or other suitable storage media, except for propagation signals. The storage system 703 may be implemented as a single storage device, but may also be implemented across multiple storage devices or subsystems that are co-located or distributed relative to each other. The storage system 703 may include additional elements that can communicate with the processing system 702 or possibly with other systems, such as a controller.

[0051] The software 705 may be implemented in program instructions and, among other functions, when executed by the processing system 702, may direct the processing system 702 to operate as described with respect to the various operational scenarios, sequences, and processes illustrated herein. The software 705 may include program instructions for implementing the failure classification process 200.

[0052] Specifically, the program instructions may include various components or modules that cooperate or otherwise interact to perform various processes and operation scenarios described herein. The various components or modules may be embodied in compiled or interpreted instructions, or in some other variations or combinations of instructions. The various components or modules may be executed in a single-threaded environment or a multi-threaded environment in a synchronous or asynchronous manner, serially or in parallel, or according to any other suitable execution paradigm, variation or combination thereof. Software 705 may include additional processes, programs or components, such as operating system software, virtual machine software or other application software, in addition to or including process 706. Software 705 may also include firmware or some other form of machine-readable processing instructions executable by processing system 702.

[0053] In general, software 705, when loaded into processing system 702 and executed, converts the entirety of a suitable device, system, or apparatus (of which computing system 701 is representative) from a general purpose computing system to a special purpose computing system to enhance canvas services for graphically organizing content in a user interface. Alternatively, encoded software 705 on storage system 703 may convert the physical structure of storage system 703. The specific conversion of the physical structure may depend on various factors in different implementations of the description. These factors may include, but are not limited to, the technology of the storage medium used to implement storage system 703 and the characteristics of the computer storage medium, whether it is a primary storage device or a secondary storage device, as well as other factors.

[0054] If the computer-readable storage medium is implemented as a semiconductor-based memory, then when program instructions are encoded therein, the software 705 can transform the physical state of the semiconductor memory, such as by transforming the state of transistors, capacitors, or other discrete circuit elements that make up the semiconductor memory. Similar transformations may occur with respect to magnetic or optical media. Other transformations of the physical media are possible without departing from the scope of this description, and the foregoing examples are provided merely to facilitate this discussion.

[0055] The communication interface system 707 may include communication connections and devices that allow communication with other computing systems (not shown) through a communication network (not shown). Examples of connections and devices that together allow inter-system communication may include network interface cards, antennas, power amplifiers, RF circuits, transceivers, and other communication circuits. The connections and devices may communicate through a communication medium to exchange communications with other computing systems or system networks, such as metal, glass, air, or any other suitable communication medium. The above-mentioned media, connections, and devices are well known and need not be discussed in detail herein.

[0056] The user interface system 709 may include a keyboard, a mouse, a voice input device, a touch input device for receiving touch gestures from a user, a motion input device for detecting non-touch gestures and other movements of the user, and other comparable input devices and associated processing elements that can receive user input from a user. Output devices such as displays, speakers, tactile devices, and other types of output devices may also be included in the user interface system 709. In some cases, input and output devices may be combined in a single device, such as a display that can display images and receive touch gestures. The aforementioned user input and output devices are well known in the art and do not need to be discussed in detail herein. The user interface system 709 may also include associated user interface software that can be executed by the processing system 702 to support the various user input and output devices discussed above.

[0057] Communication between computing system 701 and other computing systems (not shown) can occur through one or more communication networks and according to various communication protocols, combinations of protocols, or variations thereof. Examples include an intranet, the Internet, a local area network, a wide area network, a wireless network, a wired network, a virtual network, a software-defined network, a data center bus, a computing backplane, or any other type of network, combination of networks, or variations thereof. The aforementioned communication networks and protocols are well known and need not be discussed in detail here.

[0058] In any of the foregoing examples in which data, content, or any other type of information is exchanged, the exchange of information can occur according to any of a variety of protocols, including FTP (File Transfer Protocol), HTTP (Hypertext Transfer Protocol), HTTPS, REST (Representational State Transfer), WebSocket, DOM (Document Object Model), HTML (Hypertext Markup Language), CSS (Cascading Style Sheets), HTML5, XML (Extensible Markup Language), JavaScript, JSON (JavaScript Object Notation) and AJAX (Asynchronous JavaScript and XML), as well as any other suitable protocols, variations, or combinations thereof.

[0059] Certain inventive aspects can be appreciated from the foregoing disclosure, of which the following are various examples.

[0060] The functional block diagrams, operation scenarios and sequences and flow charts provided in the figure represent exemplary systems, environments and methods for executing the novel aspects of the present disclosure. Although for the sake of simplicity of explanation, the method included in this article can be in the form of a functional diagram, an operation scenario or a sequence or a flow chart, and can be described as a series of actions, it should be understood and appreciated that the method is not limited by the order of actions, because according to these actions, some actions can occur in an order different from the order shown and described in this article and / or simultaneously with other actions. Those skilled in the art will understand and appreciate that the method can be represented as a series of related states or events instead, such as in a state diagram. In addition, for novel implementations, not all actions illustrated in the method are necessary.

[0061] Example 1. A computer device comprising: one or more computer-readable storage media; one or more processors operably coupled to the one or more computer-readable storage media; and program instructions stored on the one or more computer-readable storage media, wherein the program instructions, when executed by the one or more processors, direct the one or more processors to at least: receive a test result indicating a failed code verification performed in a client device; generate failure data indicating one or more failure modes occurring during the failed code verification based on the test result; determine a subsequent action associated with each of the one or more failure modes by comparing each of the one or more failure modes with a previously identified failure mode; and transmit instructions to the client device based on the determined subsequent action associated with each of the one or more failure modes.

[0062] Example 2. A computer device of Example 1, wherein, in order to determine the subsequent action associated with each of the one or more failure modes, the program instructions direct the one or more processors to classify each of the one or more failure modes as known or unknown by comparing each of the one or more failure modes with known failure modes.

[0063] Example 3, a computer device of Examples 1-2, wherein the instructions to the client device based on the subsequent actions associated with each of the one or more failure modes determined include instructions for preventing the retry of the failure code verification performed in the client device when the one or more failure modes of the failure modes are unknown.

[0064] Example 4, a computer device of Examples 1-3, wherein the instructions to the client device based on the subsequent actions associated with each of the one or more failure modes determined include instructions for retrying the failure code verification performed in the client device when the one or more failure modes of the failure modes are known.

[0065] Example 5, a computer device of Examples 1-4, wherein the program instructions are also configured to determine that the duration of the failed code verification exceeds the code verification timer or the number of failure modes exceeds a failure mode threshold, and responsively transmit instructions to prevent retry of the failed code verification performed in the client device.

[0066] Example 6. A computer device as described in Examples 1-5, wherein, in order to determine the subsequent action associated with each of the one or more failure modes, the program instructions direct the one or more processors to compare each of the one or more failure modes with a previously identified failure mode having a corresponding operating system (OS) type to determine the subsequent action associated with each of the one or more failure modes.

[0067] Example 7, a computer device of Examples 1-6, wherein, in order to determine the subsequent action associated with each of the one or more failure modes, the program instructions direct the one or more processors to compare each of the one or more failure modes with a previously identified failure mode having a corresponding central processing unit (CPU) type to determine the subsequent action associated with each of the one or more failure modes.

[0068] Example 8, a computer device of Examples 1-7, wherein, in order to determine the subsequent action associated with each of the one or more failure modes, the program instructions direct the one or more processors to compare each of the one or more failure modes with a previously identified failure mode having a corresponding network-based platform type to determine the subsequent action associated with each of the one or more failure modes.

[0069] Example 9, a computer device of Examples 1-8, wherein, in order to generate the failure data based on the test results, the program instructions direct the one or more processors to standardize the test results by removing instant specific details from the test results, and hash the standardized test results to provide the failure data indicating the one or more failure modes that occurred during the failure code verification.

[0070] Example 10. A method comprising: receiving a test result indicating a failed code verification performed in a client device; generating failure data indicating one or more failure modes that occurred during the failed code verification based on the test result; determining a subsequent action associated with each of the one or more failure modes by comparing each of the one or more failure modes with a previously identified failure mode; and transmitting instructions to the client device based on the determined subsequent action associated with each of the one or more failure modes.

[0071] Example 11, the method of Example 10, wherein determining the subsequent action associated with each of the one or more failure modes includes: classifying each of the one or more failure modes as known or unknown by comparing each of the one or more failure modes with known failure modes.

[0072] Example 12, the method of Examples 10-11, wherein the instructions to the client device based on the subsequent actions associated with each of the one or more failure modes determined include instructions for preventing a retry of the failure code verification performed in the client device when the one or more of the failure modes are unknown.

[0073] Example 13, the method of Examples 10-12, wherein the instructions to the client device based on the subsequent actions associated with each of the one or more failure modes determined include instructions to retry the failure code verification performed in the client device when the one or more failure modes of the failure modes are known.

[0074] Example 14, the method of Examples 10-13, further comprising: determining that the duration of the failed code verification exceeds the code verification timer or the number of failure modes exceeds a failure mode threshold, and responsively transmitting instructions to prevent retry of the failed code verification performed in the client device.

[0075] Example 15, the method of Examples 10-14, wherein determining the subsequent action associated with each of the one or more failure modes includes comparing each of the one or more failure modes with a previously identified failure mode having a corresponding operating system (OS) type to determine the subsequent action associated with each of the one or more failure modes.

[0076] Example 16, the method of Examples 10-15, wherein determining the subsequent action associated with each of the one or more failure modes includes comparing each of the one or more failure modes with a previously identified failure mode having a corresponding central processing unit (CPU) type to determine the subsequent action associated with each of the one or more failure modes.

[0077] Example 17, the method of Examples 10-16, wherein determining the subsequent action associated with each of the one or more failure modes includes comparing each of the one or more failure modes with a previously identified failure mode having a corresponding network-based platform type to determine the subsequent action associated with each of the one or more failure modes.

[0078] Example 18, the method of Examples 10-17, wherein, in order to generate the failure data based on the test results, the program instructions direct the one or more processors to standardize the test results by removing instant specific details from the test results, and hash the standardized test results to provide the failure data indicating the one or more failure modes that occurred during the failure code verification.

[0079] Example 19. One or more computer-readable storage media having program instructions stored thereon, wherein the program instructions, when executed by one or more processors, direct the one or more processors to at least: receive a test result indicating a failed code verification performed in a client device; generate failure data indicating one or more failure modes occurring during the failed code verification based on the test result; classify each of the one or more failure modes as known or unknown by comparing each of the one or more failure modes with known failure modes; and transmit instructions to the client device based on a subsequent action determined to be associated with each of the one or more failure modes.

[0080] One or more computer-readable storage media of Example 20 and Example 19, wherein the instructions to the client device based on the determined subsequent actions associated with each of the one or more failure modes include instructions for retrying the failure code verification performed in the client device when the one or more failure modes of the failure modes are unknown, and instructions for retrying the failure code verification performed in the client device when one or more failure modes of the failure modes are known.

Claims

1. A computer device, comprising: one or more computer-readable storage media; one or more processors operably coupled to the one or more computer-readable storage media; as well as Program instructions stored on the one or more computer-readable storage media, which when executed by the one or more processors direct the one or more processors to at least: receiving a test result indicating a failed code validation performed in the client device; generating, based on the test results, failure data indicative of one or more failure modes that occurred during the failure code verification; determining a subsequent action associated with each of the one or more failure modes by comparing each of the one or more failure modes to previously identified failure modes, wherein, to determine the subsequent action associated with each of the one or more failure modes, the program instructions direct the one or more processors to at least: classify each of the one or more failure modes as known or unknown by comparing each of the one or more failure modes to known failure modes; and Instructions are transmitted to the client device based on subsequent actions determined to be associated with each of the one or more failure modes, wherein the instructions to the client device based on the subsequent actions determined to be associated with each of the one or more failure modes include instructions that direct the one or more processors to perform at least the following operations: preventing a retry of the failure code verification performed in the client device based on a determination that one or more of the failure modes are unknown.

2. The computer device according to claim 1, wherein: The program instructions for generating the failure data include program instructions that, when executed by the one or more processors, direct the one or more processors to at least: normalize the test results.

3. The computer device according to claim 2, wherein: The program instructions for normalizing the test results include additional instructions that, when executed by the one or more processors, direct the one or more processors to sanitize the test results by removing specific details from the test results to generate a normalized code string.

4. The computer device according to any one of claims 1 to 3, wherein: The instructions to the client device based on the determined subsequent action associated with each of the one or more failure modes include instructions for retrying the failed code verification performed in the client device based on determining that one or more of the failure modes are known.

5. The computer device according to any one of claims 1 to 3, wherein: The program instructions are further configured to transmit instructions to prevent retry of the failed code verification performed in the client device in response to determining that a duration of the failed code verification exceeds a code verification timer or a number of failure modes exceeds a failure mode threshold.

6. The computer device according to any one of claims 1 to 3, wherein: In order to determine the subsequent action associated with each of the one or more failure modes, the program instructions direct the one or more processors to compare each of the one or more failure modes with previously identified failure modes having corresponding operating system (OS) types to determine the subsequent action associated with each of the one or more failure modes.

7. The computer device according to any one of claims 1 to 3, wherein: In order to determine the subsequent action associated with each of the one or more failure modes, the program instructions direct the one or more processors to compare each of the one or more failure modes with a previously identified failure mode having a corresponding central processing unit (CPU) type to determine the subsequent action associated with each of the one or more failure modes.

8. The computer device according to any one of claims 1 to 3, wherein: In order to determine the subsequent action associated with each of the one or more failure modes, the program instructions direct the one or more processors to compare each of the one or more failure modes with previously identified failure modes having corresponding network-based platform types to determine the subsequent action associated with each of the one or more failure modes.

9. The computer device according to any one of claims 1 to 3, wherein: To generate the failure data based on the test results, the program instructions direct the one or more processors to normalize the test results by removing specific details from the test results, and hash the normalized test results to provide the failure data indicative of the one or more failure modes that occurred during the failing code verification.

10. A method for classifying test results of failed code verification, comprising: receiving a test result indicating a failed code validation performed in the client device; generating, based on the test results, failure data indicative of one or more failure modes that occurred during the failure code verification; Determining a subsequent action associated with each of the one or more failure modes by comparing each of the one or more failure modes to previously identified failure modes, wherein determining the subsequent action associated with each of the one or more failure modes comprises: classifying each of the one or more failure modes as known or unknown by comparing each of the one or more failure modes to known failure modes; and Instructions are transmitted to the client device based on a subsequent action determined to be associated with each of the one or more failure modes, wherein the instructions to the client device based on the subsequent action determined to be associated with each of the one or more failure modes include instructions for performing the following operations: preventing a retry of the failure code verification performed in the client device based on a determination that one or more of the failure modes are unknown.

11. The method according to claim 10, wherein: Generating the failure data includes normalizing the test results.

12. The method according to claim 11, wherein: Normalizing the test results includes cleansing the test results by removing certain details from the test results to generate a normalized code string.

13. The method according to any one of claims 10 to 12, wherein: The instructions to the client device based on the determined subsequent action associated with each of the one or more failure modes include instructions for retrying the failed code verification performed in the client device based on one or more of the failure modes being known.

14. The method according to any one of claims 10 to 12, further comprising: In response to determining that a duration of the failed code verification exceeds a code verification timer or a number of failure modes exceeds a failure mode threshold, instructions are transmitted to prevent a retry of the failed code verification performed in the client device.

15. One or more computer-readable storage media having program instructions stored thereon, wherein: The program instructions, when executed by one or more processors, direct the one or more processors to at least: receiving a test result indicating a failed code validation performed in the client device; generating, based on the test results, failure data indicative of one or more failure modes that occurred during the failure code verification; determining a subsequent action associated with each of the one or more failure modes by comparing each of the one or more failure modes to previously identified failure modes, wherein the program instructions for determining the subsequent action include program instructions that, when executed by the one or more processors, direct the one or more processors to at least: classify each of the one or more failure modes as known or unknown by comparing each of the one or more failure modes to known failure modes; and Instructions are transmitted to the client device based on a subsequent action determined to be associated with each of the one or more failure modes, wherein the program instructions for transmitting the instructions to the client device include program instructions that, when executed by the one or more processors, direct the one or more processors to perform at least the following operations: preventing a retry of the failure code verification performed in the client device based on a determination that one or more of the failure modes are unknown.

Images