Application Crash Test Platform

Abstract

To direct exemplary embodiments described in the present specification to an application crash testing platform that tests provided applications through a machine learning algorithm.SOLUTION: A plurality of instances of an application are spawned on a cloud based platform. A machine learning algorithm provides input to each of the instances until an incident causing the application to become inoperable is reached.SELECTED DRAWING: Figure 6

Description

[Technical field]

[0001] This U.S. patent application is based on U.S. Provisional Patent Application No. 63 / 180,866, filed April 28, 2021, and claims the benefit of domestic priority under 35 U.S.C. § 119(e), the disclosure of which is incorporated by reference in its entirety herein.

[0002] The present disclosure relates generally to software as a service (SaaS) platforms, and more particularly to using machine learning to assist the platform for application crash testing. [Background technology]

[0003] In the related art, developers often employ quality assurance testers to test applications for bugs, memory faults, and crashes. Such quality assurance testers often spend many man-hours trying various inputs to the application to test whether the application will crash under certain conditions. Summary of the Invention [Means for solving the problem]

[0004] Exemplary implementations described herein leverage advances in machine learning and cloud platforms to assist an application crash testing platform to test applications for specific, definable crashes or faults. Exemplary implementations described herein allow quality assurance testers to utilize the platform to test applications for specific types of crashes, faults, or other definable incidents that render the application inoperable. If such incidents are definable and set as win conditions for the machine learning algorithm, the machine learning algorithm can assist with inputs in the virtual environment in an attempt to reach the win condition, which can then be populated into an input log for review. Such inputs can include, but are not limited to, mouse clicks on specific pixels of a screen, keystrokes, joystick button presses or stick movements, virtual reality headset head position coordinates, etc., depending on the desired implementation. In other exemplary implementations, video replays can be generated for review by the quality assurance tester. Additionally, for certain types of hardware, such as video game consoles, the inputs and hardware can be defined in terms of the virtual platform, which allows the quality assurance tester to test under specific environmental conditions. Although the example implementations described herein are directed to a cloud-based platform, the example implementations may also be executed on a logical device or any other device as needed to support the desired implementation.

[0005] Aspects of the disclosure may include a method for testing an application using a machine learning algorithm, the method including: executing (e.g., on a cloud-based system) one or more instances of the application, where the one or more instances are configured to receive input from a machine learning algorithm configured to provide the input to the application to cause an incident; executing the machine learning algorithm to provide the input to the one or more executed instances; and, for an incident detected (e.g., by the cloud-based system) in an instance of the one or more instances that caused the application in the one or more instances to be inoperable, generating an output indicative of the incident that caused the application to be inoperable.

[0006] Aspects of the disclosure may include a computer program for testing an application using a machine learning algorithm, the computer program having instructions including: executing one or more instances of the application (e.g., on a cloud-based system), the one or more instances configured to receive input from a machine learning algorithm configured to provide the input to the application to cause an incident; executing the machine learning algorithm to provide the input to the one or more executed instances; and, for an incident detected (e.g., by the cloud-based system) in an instance of the one or more instances that caused the application in the one or more instances to be inoperable, generating an output indicative of the incident that caused the application to be inoperable. The computer program may be stored on a non-transitory computer-readable medium for execution by one or more processors.

[0007] Aspects of the disclosure may include an apparatus for testing an application using a machine learning algorithm, the apparatus having a processor configured to execute instructions including: executing one or more instances of the application (e.g., on a cloud-based system), the one or more instances configured to receive input from a machine learning algorithm configured to provide the input to the application to cause an incident; executing the machine learning algorithm to provide the input to the one or more executed instances; and for an incident detected (e.g., by the cloud-based system) in an instance of the one or more instances that caused the application in the one or more instances to be inoperable, generating an output indicative of the incident that caused the application to be inoperable. The processor may be any processor known in the art, such as a central processing unit (CPU), or dedicated hardware, such as a field programmable gate array (FPGA), a graphical processing unit (GPU), a tensor unit, etc., according to a desired implementation.

[0008] Aspects of the disclosure may include a system for testing an application using a machine learning algorithm, the system including: means for executing (e.g., on a cloud-based system) one or more instances of the application, the one or more instances configured to receive input from a machine learning algorithm configured to provide the input to the application to cause an incident; means for executing the machine learning algorithm to provide the input to the one or more executed instances; and, for an incident detected (e.g., by the cloud-based system) in an instance of the one or more instances that caused the application in the one or more instances to be inoperable, means for generating an output indicative of the incident that caused the application to be inoperable. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example system in which example implementations described herein may be implemented. [Diagram 2] FIG. 2 illustrates an example server / storage system architecture for a cloud in accordance with an example implementation. [Diagram 3] FIG. 2 illustrates a table of example input devices for defining acceptable inputs for an application, according to an example implementation. [Figure 4] FIG. 2 illustrates an example incidents table according to an example implementation. [Diagram 5] FIG. 13 illustrates an example of management information for a cloud-based system according to an example implementation. [Figure 6] FIG. 1 illustrates an example flow diagram of a cloud-based system according to an example implementation. [Figure 7] FIG. 13 illustrates an example user interface for outputting incidents according to an example implementation. [Figure 8] FIG. 13 illustrates an example user interface for submitting parameters for application and incident tests according to an example implementation. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The following detailed description provides further details of the figures and example implementations of the present application. Reference numbers and descriptions of elements that overlap between figures have been omitted for clarity. Terms used throughout the specification are provided by way of example and are not intended to be limiting. For example, use of the term "automatic" can include fully automatic or semi-automatic implementations that include user or administrator control over certain aspects of the implementation, depending on the desired implementation of the skilled artisan practicing the implementation of the present application.

[0011] FIG. 1 illustrates an example system in which example implementations described herein may be implemented. In the example system, there are one or more client devices 101 that send a test application to a cloud platform 102. The cloud platform 102 may include one or more servers and storage systems configured to receive the application, generate a virtual machine or virtual platform for spawning multiple instances of the application, and control the application in each instance through inputs provided by a machine learning algorithm. The client devices 101 and the cloud platform 102 may be interconnected by a network 103. The network 103 may be a wireless or wired connection and may be supported as a local area network (LAN) or over the Internet.

[0012] 2 illustrates an example server / storage system architecture for cloud 102, according to an example implementation. Cloud 102 may be comprised of one or more servers 200 that manage one or more storage systems 220 through one or more internal networks 210. Internal network 210 may be a network, such as a Storage Area Network (SAN), a Wide Area Network (WAN), or other network, depending on the desired implementation.

[0013] Each of the one or more servers 200 may include an I / O device 201, a processor 202, which may take the form of one or more physical processors, such as a central processing unit (CPU) or the like, as described herein, a memory 203, and a network interface 204. The memory 203 may include a cache memory for ingesting live broadcast streaming data processed in real time from the stream broadcast system, and other non-transient memory for loading instructions to the processor 202 for executing algorithms associated with the server 200. In an exemplary implementation, the memory 203 may be in the form of a flash memory, such as a solid state device (SSD), although other implementations, such as a hard disk drive (HDD) or a redundant array of inexpensive disks (RAID), may be utilized depending on the desired implementation. Server 200 may also interact with one or more storage systems 220, each of which may include I / O devices 221, storage controllers 222, which may take the form of one or more physical processors, storage devices 223, and network interfaces 224. In an example implementation, data transmitted from the stream broadcast system and managed in memory 203 may be destaged to one or more storage systems 220 to be used in a database for machine learning algorithms.

[0014] One or more storage systems 220 may be configured as a pool that acts as a database for machine learning algorithms. Storage controller 222 is configured to manage the data in storage systems 220 by providing data from storage devices 223 for use in a pool or for use in a logical volume.

[0015] Depending on the desired implementation, the processor 202 may be in the form of one or more general purpose computing on graphical processing units (GPGPUs), tensor units, streaming processors, or any other type of dedicated hardware processors specifically configured to perform machine learning algorithm processing on data according to the example implementations. Such dedicated hardware processors may also be configured as clusters to assist with the processing power required for the machine learning algorithms to meet the latency or real-time requirements of the example implementations described herein. In such example implementations, utilizing dedicated hardware configured to perform the particular machine learning algorithms of the example implementations may help ensure that the platform provides replays or input logs to client devices within a desired time frame. In other example implementations, the processor 202 may be in the form of a general purpose hardware processor, such as a central processing unit (CPU), or a pool of CPUs allocated from a cloud environment according to the desired implementation.

[0016] In an example implementation, the processor 202 may execute a method or instructions for testing an application with a machine learning algorithm, including executing one or more instances of the application, where the one or more instances are configured to receive input from the machine learning algorithm; executing one or more instances of the application, where the one or more instances are configured to receive input from the machine learning algorithm; executing the machine learning algorithm to provide input to the one or more executed instances; and, for an incident detected in an instance of the one or more instances that caused the application in the one or more instances to be inoperable, generating an output indicative of the incident that caused the application to be inoperable.

[0017] Depending on the desired implementation, such an incident may be a fault screen. A fault screen may be any type of error screen displayed by an underlying operating system or application in the event of a system or application error. Such fault screens may be generated when hardware malfunctions, unexpected process terminations, system crashes, etc. occur, as known in the art. Such fault screens are well known in the art to be consistent in form, and therefore may be easily detected by screenshot feature extraction via any object detection machine learning algorithm (e.g., by detecting features associated with consistent background color and overlaid text that do not change over a threshold time period), or via a custom detection algorithm that detects specific fault screens from screenshots / videos of instances and provides feedback to the machine learning algorithm. One skilled in the art may train a machine learning algorithm against the desired fault screen to be detected in order to configure the machine learning algorithm to detect the desired fault screen according to the desired implementation. Illustrative examples of fault screens may include, but are not limited to, blue screen of death, black screen of death, other similar screens of death, kill screens in gaming applications, hardware error screens, etc.

[0018] Depending on the desired implementation, the output may include a video of said instance of one or more instances. Such a video may be generated according to any desired implementation as known in the art. For example, the instance may be executed with a screen / video recording application as known in the art configured to cache the most recent time period of the incident for a threshold time period or until a certain amount of memory is consumed. The video may also be compiled from stored screenshots, where screenshots are taken at any desired frame rate according to the desired implementation. Furthermore, if dedicated resources are required, the screenshots / videos may be recorded or processed by a dedicated machine or otherwise according to the desired implementation (e.g., from one server via a video splitter to another server dedicated to screenshot / video processing).

[0019] Depending on a desired implementation, the output may include audio of the one or more instances. Such audio may be generated according to any desired implementation as known in the art. For example, the instances may be executed with an audio recording application as known in the art that is configured to record audio for a period of time immediately preceding the incident, for a threshold time period or until a certain amount of memory is consumed.

[0020] Depending on the desired implementation, the output may include haptic feedback from the application in one or more instances. Such haptics may be generated according to any desired implementation as known in the art. For example, the application may be associated with a device (e.g., a glove, a chair, etc.) configured to provide haptic feedback, and control signals from the application directed to such device may be recorded and analyzed against desired thresholds.

[0021] Depending on the desired implementation, the output may include an input log for the application. For example, the instance may run in parallel with a detection application that may be configured to output an input log, such as a keystroke logger, mouse logging application, input recorder, etc., according to the desired implementation. In another example implementation, the input provided by the machine learning algorithm may be stored in a log file in a desired format through any desired implementation.

[0022] Depending on the desired implementation, inputs may be restricted based on the device selected to be tested on the application. For example, if the input device is a joystick or game controller, certain inputs (e.g., simultaneous left and right directions) may not be physically possible and are therefore restricted. Input restrictions can be used to distinguish between different types of input devices (e.g., two-button mouse vs. three-button mouse), to reject input from one device when another device is providing input (e.g., prioritizing voice over button input), to test multiple inputs from multiple devices based on available device connections, etc. Further details are provided in conjunction with FIG. 3.

[0023] Depending on the desired implementation, the input may include network data transmitted to the application. The network data may include any data transmitted to the application over a network (e.g., the Internet, a local area network, cellular, etc.). In such example implementations, the application may be, for example, a mobile device application configured to receive input from the Internet, a router application configurable to receive input from multiple devices, an application for an edge device configured to receive instructions from a server, a server input to a computer application, etc., according to the desired implementation. Such network data may include packets transmitted according to any desired protocol (e.g., TCP / IP, UDP, etc.) to simulate network communication with the application. In such example implementations, the machine learning algorithm may be configured to simulate such network data as input, according to the desired implementation.

[0024] Depending on the desired implementation, the incident may be a freeze of operation. Any freeze of operation may be used to detect the incident. A freeze of operation includes incidents that lock the application and make it unresponsive, such as, but not limited to, an infinite loop, a frozen screen, etc., according to the desired implementation. Such incidents may be detected by a machine learning algorithm via feature extraction or other methods according to any desired implementation, or via a separate detection application configured to provide feedback to the machine learning algorithm. For example, a frozen screen may be detected when a screenshot detection function detects the same screenshot despite input being provided for more than a threshold time period, which may be detected by the machine learning algorithm or by a separate screenshot detection application. An infinite loop may similarly be detectable based on no response to input, either by a machine learning algorithm or a separate application, according to a similar implementation.

[0025] Depending on the desired implementation, the incident may be determined from the application logs of the application. Depending on the underlying application, some applications may generate their own application logs for error handlers, exception handlers, etc., that can be used as a win condition for the machine learning algorithm. For example, if the application only generates application logs in response to an error or exception, the generation of such application logs can be used as feedback to the machine learning algorithm that an incident has occurred. In another example implementation, the application generates consistent application logs to record and output certain metrics associated with the application. In such an example implementation, such application logs can be read by the machine learning algorithm to determine whether an incident has occurred based on certain metrics (e.g., user-specified) in the application logs that are deemed to be indicative of an incident. For example, if CPU usage exceeds a threshold, the application log that records CPU usage can be used to indicate that an incident has occurred.

[0026] Depending on the desired implementation, the incident may be determined from a debug hook. In some example implementations, a user may want to insert code or execute at a particular runtime line of an application or intercept actions related to the application for analysis to read variable values. The output of the debug hook (e.g., variable values, error handlers, etc.) may be used as feedback to detect the incident as defined according to the desired implementation.

[0027] Although the incidents are described as being detected by a machine learning algorithm, other detection applications can be used and such detection applications can simply inform the machine learning algorithm that an incident has occurred according to a desired implementation. For example, the detection application can be run across multiple instances of the application to reduce the burden on the machine learning algorithm and can be programmed to detect the desired incident.

[0028] Although the example implementations described herein are directed to cloud-based implementations, they are also extensible to a user's user device or microservices. A user may, for example, implement the example implementations herein locally on a microservice with dedicated hardware and a computing environment such as that shown in FIG. 2, and the models are loaded into the microservice via cloud delivery. In other example implementations, it can also be easily installed on a local user device such as a mobile phone or laptop.

[0029] FIG. 3 illustrates an example input device table for defining acceptable inputs for an application, according to an example implementation. In an example implementation, some applications may operate in an environment where only certain types of inputs are allowed. For example, a document viewer application may allow keyboard, touch, and mouse inputs, but not necessarily voice inputs. In another example, a video game console may only accept inputs from certain peripheral devices (e.g., joysticks, controllers, hitboxes, etc.) and may have certain types of inputs disabled by its configuration (e.g., joysticks do not allow simultaneous left / right inputs, the console only allows certain directional and button inputs, etc.). This input device table is used to configure machine learning algorithms to restrict the types of inputs available to an application to simulate an environment or application configuration. The inputs may be provided in any manner according to the desired implementation. For example, the inputs may be simulated in software, by using software hooks to simulate the inputs, by an emulator to provide the inputs, or through driver inputs, according to the desired implementation. If desired, a physical input implementation can be utilized, such as a robot or special hardware to provide physical input on the hardware. In an example implementation, a virtual debug hook can be used that navigates not only visually on the screen but also through the code (e.g., a hierarchical view, etc.) to directly interact with specific input / GUI elements. In such an example implementation, the debug hook can include a program that can intercept the application's code or execution for analysis and insertion into its operation. One example is to define a debug hook to stop at a particular line to execute code, to stop at a line to read the value of a variable, or to wait for execution of a particular line, such as an error handler.

[0030] Input devices can also be associated with a type of input to match the connection ports of the device of the underlying application under test. For example, a game console may only have two USB ports and two joystick input ports. In this case, when selecting a device for testing, up to two USB input devices and two joystick devices can be selected through the test user interface.

[0031] In another example implementation, there may be a direct hook into speech recognition such that text input (e.g., "turn on the lights") is used as opposed to utilizing an audio recording. In another example implementation, audio recordings may also be used to input audio that has not yet been translated into text (e.g., to test for noisy environments or specific frequencies). Input may also include sensor data, such as from global positioning satellites (GPS), gyroscope measurements, location information, etc., according to the desired implementation.

[0032] FIG. 4 illustrates an example incident table according to an example implementation. The incident table defines the types of incidents that render an application inoperable and the conditions that define such incidents. For example, a fault incident is detected when a fault screen appears. A machine learning algorithm can use such definitions to determine the winning or target conditions for generating such incidents. The detection can be screenshot-based, where reaching a particular frame or screen then determines the conditions that the machine learning algorithm must meet, as shown in FIG. 4. However, other inputs besides screenshots can also be used if available, and the disclosure is not limited thereto. For example, logs, text, and code can also be detected as inputs. In an example implementation, debug hooks can also be used as inputs for detection (e.g., detecting if a variable's value is null). Code tracing can also be utilized if the code is available for analysis in parallel with the execution of the application. Additionally, logs output by the application itself or by the application's compiler can also be utilized in real time to detect certain winning conditions that indicate an application that is inoperable.

[0033] For example, some applications, such as games, may have their own definable incidents. In one instance of a platform game, a player's character may be detected repeatedly from observing the same set of frames, regardless of all possible inputs, as being stuck in a wall and unable to move regardless of all possible inputs. Another approach to setting win conditions based on code tracing may include, but is not limited to, detecting whether the player's character coordinates overlap with the coordinates of a wall, or whether the character coordinates trigger or exceed some limit (e.g., crossing the boundaries of the game screen). Another example implementation may utilize a debug hook that returns an exception regarding the character coordinates.

[0034] Other non-screen-based conditions may also be utilized depending on the desired implementation. For example, another type of incident may be whether an application consumes more random access memory (RAM) than the environment allows, whether a deadlock occurs in the underlying central processing unit (CPU), whether power consumption exceeds a threshold, whether latency or input latency exceeds a threshold, etc., depending on the desired implementation.

[0035] An incident may be defined based on various elements, such as output logs, stack traces, error codes, exceptions, specifically defined errors, and / or debug hooks configured to listen to the application for errors or exceptions in its internal logs. Depending on the desired implementation, such elements may be separate entities, such as files stored on a storage system, hard disk drive, or other medium. Such elements may also be compilations created by the application itself as well as by third-party applications (e.g., operating system specific event logs), depending on the desired implementation.

[0036] Other incidents may also be utilized as win or target conditions, and the disclosure is not specifically limited thereto. One example may include detection of a reboot of a device, since some devices may not have a fault screen and may simply perform a reboot in response to a kernel violation. Another example may include certain audio feedback (e.g., "Sorry, I didn't understand" in the case of a voice recognition application), or when an audio output infinitely loops the previous byte due to a kernel crash. Other incidents may also include, but are not limited to, continuous CPU usage, power fluctuations, levels of power draw, and unresponsive controls. Depending on the desired implementation, incident detection may be a separate standalone program (e.g., a monitoring application that monitors CPU usage and feeds measurements to a machine learning algorithm) that measures the desired metrics for incident detection.

[0037] The example implementations described herein may also be extended to test interoperability between machines, according to a desired implementation. Such examples may include performing distributed denial of service (DDoS) attacks using multiple machines against one machine to determine resiliency, sending code via text message from one mobile device to a single mobile device to render the single targeted mobile device inoperable, simulating a packet relay attack, etc., according to a desired implementation. In such example implementations, the input system and the monitoring system may be instantiated as separate entities residing on different machines, communicating via a switch, network, or some other hub, according to a desired implementation.

[0038] Depending on the application's underlying platform / physical hardware, physical conditions can also be win or target conditions. For example, detection of a smartphone vibrating indefinitely without resolution can be used as a condition. A misalignment of the head of a hard disk drive platter, or any other robotic arm, or other edge device with mechanical parts (e.g., garage door, sensor, etc.) can also be another incident. In another example involving a holographic implementation, conditions set based on output to the hologram can also be utilized. Non-screen hardware representations of incident states (e.g., specific LEDs triggered in a car or elevator) may be utilized to aid in the desired implementation. Timing-based representations can also be used to indicate incidents, such as an inaccurate clock, an LED blinking in a rhythmic fashion, Morse code, etc. Additionally, the absence of an expected state or output can also be considered an incident (e.g., a speaker not producing the expected audio output).

[0039] Incidents that will be used as win or target conditions can also be targeted to specific memory consumption levels (eg, an application consuming more than 1GB of RAM or 100% of the CPU).

[0040] An infinite loop may also be used as an incident. An example may include the repetition of the same output frame repeated beyond a threshold. Such implementations may be modified to support desired implementations. For example, in a game, there may be an infinite combo that is continuously looped by the player's character. In another example, an incident may be clicking "Ok" on a pop-up dialog resulting in the same pop-up dialog reappearing. In other example implementations, other infinite loops, such as infinite audio loops, infinite device state loops, etc., may be detected via other methods depending on the underlying application.

[0041] Other incidents may further include exceptions generated by the application under test such as segmentation faults or stack overflows, blue screens or any other kernel crashes, misinterpreted / lost input (e.g., mouse can only move cursor in bottom half of screen or input is not working at all), or application specific error boxes / messages either from third party applications (e.g., error reporters) or via dialog boxes within the application. Incidents may also include inability to interact with expected elements or inability to exit the program. Moreover, such applications may also be tested via a virtual machine or kernel, according to the desired implementation.

[0042] 5 illustrates an example of management information for a cloud-based system according to an example implementation. Such management information may include an application identifier (ID) associated with a particular application, the type of incident a user selects to test the application, the application's hardware / software / input environment configuration settings, and the user ID of the user to whom the replay input output is intended.

[0043] 6 illustrates an example flow diagram of a cloud-based system according to an example implementation. First, the cloud-based system receives a test application from a user at 601 along with certain parameters such as the type of incident to be tested, environment definitions, input device restrictions, etc. At 602, multiple instances of the application are spawned according to the specified parameters.

[0044] At 603, input is provided to each of the multiple instances by a machine learning algorithm for the purpose of arriving at an incident of a type specified in the parameters. In an example implementation, a neural network such as a Convolutional Neural Network (CNN) or other machine learning implementation such as unsupervised learning may be used to analyze screenshots, metrics, logs, audio, and / or other inputs of the application in a frame-by-frame or time-step manner based on the provided input, as such machine learning algorithms are suitable for understanding a particular screen, such as a fault screen, and / or the accompanying metrics, logs, audio, and / or other inputs. However, other machine learning algorithms may be utilized according to a desired implementation, and the disclosure is not limited thereto.

[0045] At 604, the machine learning algorithm will continue providing inputs to the instance until an incident is detected. If so (yes), then at 605 the application state and inputs can be output as a log so that the user can see the inputs used to recreate the incident. Depending on the desired implementation, the output can be in any desired form, such as, but not limited to, a video based on accumulated screenshots, an input log that can be run for replay on a user device or user application, audio, GPS, gyroscope readings, an output log describing the state of a machine or process as a result of the incident, etc. Further examples are provided in connection with FIG. 7. The log can include any other aspect of the instance in addition to the inputs, and the disclosure is not limited thereto.

[0046] There are several approaches to constructing and training machine learning algorithms. In an exemplary implementation, unsupervised learning can be utilized on the application under test. The unsupervised learning algorithm can be configured to provide inputs to the application and is rewarded for causing the desired incidents of the test (e.g., fault screen, lost input, exception, error code, etc.). Another exemplary implementation can utilize supervised learning, where the supervised learning algorithm is trained on inputs that cause known incidents known to the application developer. In the case of a screenshot-based implementation, a CNN can be applied to monitor screenshots of the instance under test, and the CNN can be trained to determine the appropriate input to cause the incident based on features extracted from the screenshots. According to a desired implementation, the CNN can be trained on previous screenshots of the incident, with corresponding input logs of known incidents of the application being tested or otherwise. In another exemplary implementation, the machine learning algorithm can be constructed via a generative adversarial network (GAN), where the adversarial network attempts to check for incidents in response to inputs provided by the input network.

[0047] FIG. 7 illustrates an example user interface for output of an incident, according to an example implementation. In one instance, a user can view a replay of the incident for the application provided as provided by the cloud-based system. Such output may include a final screenshot of the incident, the type of incident detected, application and environmental parameters, input and output logs, and a video replay of the application, as shown in interface 702. Depending on the desired implementation, the output may also be in the form of audio feedback for audio-based applications, such as speech recognition software, or for applications that simply provide audio output. For further information, process / machine metrics may also be provided, such as metrics associated with CPU, memory, GPU, temperature, power consumption, network utilization, latency, etc. Other outputs may be utilized according to the desired implementation and underlying platform of the application, and the disclosure is not limited thereto. For example, an application directed to a haptic system may also include a replay of the underlying haptic output, and applications directed to systems including different types of audio, tactile, olfactory, gustatory, or optical outputs may also be configured to provide corresponding outputs. Other outputs for replay can also include location-based inputs (e.g., based on detections from cameras, depth cameras, etc.), and can utilize GPS measurements, altimeter measurements, location measurements, etc. Additionally, example implementations described herein can also be configured to incorporate underlying input signals or any combination of multiple types of inputs to support an application on its underlying platform, such as electromagnetic signals, optical signals, LiDAR or radio signals for computer vision applications, radio signals for radar-based applications, sonar, polarization / light orientation, etc., according to a desired implementation. Such combinations of inputs can be determined based on the underlying application and platform to be tested.For networked / network-based applications, simulated network data (e.g., packet, traffic data, etc.) can also be used as input.

[0048] 8 illustrates an example user interface for submitting an application and corresponding parameters for incident testing according to an example implementation. The interface may include a selection 1702 of the application to be submitted from a user device or account, any initial settings or environmental conditions 1703 to be specified for the application, a set of input devices allowed for the application 1705, and a type of incident to be tested 1706. Once the information is selected, the user may register the application and parameters with the cloud system as shown at 1707. Predefined incidents such as, but not limited to, freezes in operation, exceptions, error codes, and stack traces may also be included according to the desired implementation.

[0049] Some portions of the detailed description are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the essence of their innovations to others skilled in the art. An algorithm is a sequence of defined steps leading to a desired end state or result. In the illustrated implementations, the steps performed require specific quantities of physical manipulations to achieve specific results.

[0050] Unless expressly stated otherwise, as will be apparent from this discussion, discussions throughout the description utilizing terms such as "processing," "computing," "calculating," "determining," "displaying," and the like, will be appreciated to include the operation and processing of a computer system or other information processing device that manipulates and converts data and memory represented as physical (electronic) quantities in the registers and memory of the computer system into other data similarly represented as physical quantities in the memory or registers of the computer system or other information storage, transmission, or display device.

[0051] Exemplary implementations also relate to apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. Computer-readable storage media may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memory, random access memory, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation that includes instructions to perform the operations of a desired implementation.

[0052] Various general-purpose systems may be used with the programs and modules in accordance with the examples herein, or it may be convenient to construct more specialized apparatus for performing the desired method steps. Moreover, the example implementations are described without reference to any particular programming language. It will be appreciated that a variety of programming languages ​​can be used to implement the teachings of the example implementations described herein. Instructions in the programming language may be executed by one or more processing devices, such as a central processing unit (CPU), processor, or controller.

[0053] As is known in the art, the operations described above may be performed by hardware, software, or some combination of software and hardware. Various aspects of the example implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform methods for implementing the implementations of the present application. Furthermore, some example implementations of the present application may be performed exclusively in hardware, while other example implementations may be performed exclusively in software. Still further, the various functions described may be performed in a single unit or may be distributed across several components in any number of ways. When performed by software, the methods may be executed by a processor, such as a general purpose computer, based on instructions stored on a computer-readable medium. If desired, the instructions may be stored on the medium in a compressed and / or encrypted format.

[0054] Furthermore, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the teachings of the present application. Various aspects and / or components of the described example implementations may be used alone or in any combination. It is intended that the specification and example implementations be considered as merely illustrative, with a true scope and spirit of the present application being indicated by the following claims.

Claims

1. 1. A method for testing an application using a machine learning algorithm, comprising: executing a plurality of instances of an application, each of the plurality of instances configured to receive input from the machine learning algorithm configured to provide input to the application for causing one or more of a plurality of defined incident types of the application, each of the plurality of defined incident types being defined with a win condition or target condition of the machine learning algorithm; executing the machine learning algorithm to provide the input to each of the plurality of instances; and generating an output indicating, for an incident type of the defined plurality of incident types detected in an instance of the one or more instances that caused the application in the instance of the one or more instances to be inoperable, the type of incident that caused the application to be inoperable.

2. The method of claim 1 , wherein the plurality of incident types includes a fault screen.

3. The method of claim 1 , wherein the output includes a video of the instance of the one or more instances.

4. The method of claim 1 , wherein the output comprises audio of the application in the instance of the one or more instances.

5. The method of claim 1 , wherein the output comprises haptic feedback from the application in the instance of the one or more instances.

6. The method of claim 1 , wherein the output comprises an input log for the application.

7. 2. The method of claim 1 , wherein the allowed inputs provided by the machine learning algorithm are limited based on one or more input devices selected from a plurality of input devices to be tested on the application, each of the plurality of input devices being associated with inputs and types of connections of inputs that are excluded to define the allowed inputs.

8. The method of claim 1 , wherein the input comprises network data sent to the application.

9. The method of claim 1 , wherein the plurality of incident types includes a freeze of operation.

10. The method of claim 1 , wherein the type of incident is determined from an application log of the application.

11. The method of claim 1 , wherein the type of incident is determined from a debug hook.

12. The method of claim 1 , wherein the plurality of incident types includes a physical condition affecting an underlying platform or physical hardware of the application.

13. The method of claim 1 , wherein the plurality of incident types comprises a memory consumption level.

14. The method of claim 1 , wherein the plurality of incident types includes an infinite loop.

15. The method of claim 1 , wherein the input comprises sensor data.

16. The method of claim 1 , wherein the input comprises a text message from a mobile device.

17. The method of claim 1 , wherein the incident types of the defined plurality of incident types are detected by a separate detection application configured to provide feedback to the machine learning algorithm.

18. A computer program storing instructions which, when executed by one or more processors, cause the computer to carry out the method of any one of claims 1 to 17.

19. 18. An apparatus comprising a processor configured to perform the method of any one of claims 1 to 17.

Images