Systems and methods for software design management and quality assurance

Abstract

A design management architecture and software quality assurance system for analyzing stimulus-response patterns in user activity data. The design management architecture and software quality assurance system may process the user activity data according to a classifier model to evaluate the impact of incremental design changes on one or more quality measures of the software product. According to an embodiment, the one or more quality measures may include one or more efficacy, safety, and / or performance metrics associated with one or more features of the software product. In an embodiment, the software product may include software as a digital health intervention or a medical device product.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Application No. 17 / 117,050, filed December 9, 2020, and U.S. Provisional Application No. 63 / 003,673, filed April 1, 2020, each of which is incorporated herein by reference in its entirety at least.

[0002] The present disclosure relates to the field of software quality assurance systems, and more particularly to systems and methods for procedural analysis of stimulus input patterns in user test data to evaluate, measure and / or verify one or more performance metrics associated with program code builds. [Background technology]

[0003] Quality assurance is often an integral part of software development. For example, quality assurance personnel may inspect newly developed software, identify defects contained within the software, and determine whether the software is of sufficiently high quality to be released to end users. Software quality assurance can encompass the entire software development process, including requirements definition, software design, coding, code review, source code control, software configuration management, inspection, release management, and product integration. The software quality assurance process can be organized into goals, commitments, capabilities, activities, measurement, and verification.

[0004] Software is an increasingly important area of ​​healthcare product development. A growing area of ​​healthcare product development is that of digital health interventions (i.e., interventions delivered via digital technologies such as smartphones, mobile computing devices, wearable electronics, and the like) that provide effective, cost-effective, safe, and scalable interventions to improve health and healthcare. Digital health interventions (DHIs) and software as a medical device (SaMD) can be used to promote healthy behaviors, improve outcomes, and provide remote access to effective treatments, e.g., computer-controlled cognitive behavioral therapy for mental health and physical problems, in people with long-term conditions such as cardiovascular disease, diabetes, and mental health conditions. Software as a medical device (SaMD) is defined by the International Medical Device Regulators Forum (IMDRF) as "software intended to be used for one or more medical purposes and that performs these purposes without being part of a hardware medical device." DHIs are often complex interventions with multiple components, and many have multiple objectives, including enabling users to become better informed about their health, share experiences with similarly situated others, change health awareness and perceptions, assess and monitor specified health conditions or behaviors, titrate medications, clarify health priorities and arrive at treatment decisions consistent with these, and improve communication between patients and healthcare professionals (HCPs). Active components may include information, psychoeducation, personal stories, formal decision aids, behavior change support, interactions with HCPs and other patients, self-assessment or monitoring tools (questionnaires, wearables, monitors, and valid theory-based psychological interventions developed for in-person delivery, such as cognitive behavioral therapy or mindfulness training). Some DHI and SaMD products may include software that is itself directly therapeutic in treating and / or targeting one or more neural circuits associated with one or more neurological, psychological, and / or physical conditions, diseases, and / or disorders, rather than simply being a component of an overall treatment.

[0005] The unique nature of SaMD and DHI software products, compared to business or consumer software products, poses particular challenges with respect to software development and software quality assurance policies, processes, and standards. Through diligent effort, ingenuity, and innovation, applicants have identified deficiencies in prior art solutions and developed the solutions embodied by the present disclosure, which are described in detail below. Summary of the Invention [Problem to be solved by the invention]

[0006] The following is a simplified summary of certain embodiments of the invention in order to provide a basic understanding of the invention. This summary is not extensive and is not intended to identify key / critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present embodiments of the invention in a simplified form as a prelude to the more detailed description that follows.

[0007] Objects of the present disclosure include a design management architecture and quality assurance system for analyzing stimulus-response patterns in user activity data to evaluate the effectiveness, safety, and / or performance of one or more features of a software product. Further objects include a design management architecture and quality assurance system for evaluating the impact of incremental design changes on the effectiveness, safety, and / or performance of a software product. Certain aspects of the present disclosure provide a design management architecture and quality assurance system / method for monitoring and configuring one or more software development processes for DHI or SaMD, including software design, coding, code review, source code control, software configuration management, inspection, release management, and product integration.

[0008] Objects of the present disclosure include systems and methods for measuring the performance of a DHI or SaMD. Further objects include systems and methods for measuring and / or quantifying the amount of therapeutic activity and / or effectiveness of a DHI or SaMD. Certain embodiments of the present disclosure may include systems and methods for measuring and / or verifying the effectiveness of a DHI or SaMD to treat and / or target one or more neural circuits associated with one or more neurological, psychological, and / or physical conditions, diseases, and / or disorders of a user. Further objects include systems and methods for measuring and / or quantifying the amount of therapeutic activity and / or effectiveness of one or more individual features or aspects of a DHI or SaMD. Further objects include systems and methods for evaluating the impact of incremental design changes on the effectiveness of a DHI or SaMD to treat and / or target one or more neural circuits associated with one or more neurological, psychological, and / or physical conditions, diseases, and / or disorders of a user.

[0009] The present disclosure includes systems and methods for analyzing one or more patterns in user activity data resulting from one or more instances or sessions of a DHI or SaMD product. According to certain embodiments, the one or more patterns may include one or more clinically validated stimulus-response patterns. Certain embodiments of the present disclosure may include a machine learning framework, classifier model, and / or classification algorithm for classifying and / or mapping user input into one or more categories. In certain embodiments, the one or more categories may correspond to one or more safety, efficacy, or performance aspects of a DHI or SaMD product. [Means for solving the problem]

[0010] An aspect of the present disclosure provides a processor-implemented method for software quality assurance, comprising: using a computing device to present a test instance of a software build to a user, the software build including at least one test feature configured to present one or more computer-controlled stimuli or interactions to the user via a graphical user interface; using the computing device to receive one or more user inputs in response to the one or more computer-controlled stimuli or interactions, the one or more user inputs including user activity data; receiving the user activity data using a processor communicatively coupled to the computing device; using the processor to process the user activity data according to at least one classification model for classifying the activity data, the at least one classification model configured to classify one or more variables associated with targeted stimulus-response patterns; analyzing the stimulus input patterns between the user activity data and the one or more computer-controlled stimuli or interactions to determine one or more performance metrics for the software build, the one or more performance metrics including a measure of effectiveness of the at least one test feature; and evaluating the software build according to the one or more performance metrics to determine a pass / fail status of the software build according to a minimum performance threshold.

[0011] According to the aspects of the present disclosure, the method for software quality assurance may further include evaluating the software build according to one or more performance metrics to determine a pass / fail status of at least one test feature according to a minimum performance threshold. The method may further include comparing the one or more performance metrics for the software build with one or more performance metrics from a previous or subsequent software build to determine an amount of change in the one or more performance metrics. In one embodiment, the one or more performance metrics include an amount of effective therapy delivery received by a user in response to a test instance of the software build.

[0012] According to the aspect of the present disclosure, the method for software quality assurance may further include presenting the user activity data and one or more performance metrics to an administrative user via a graphical user interface, the graphical user interface configured to receive one or more user queries for evaluation of at least one test feature. In some embodiments, the one or more performance metrics include a measure of safety of the at least one test feature and / or software build. In some embodiments, the one or more computer-controlled stimuli or interactions may be configured to effectively treat or target one or more neurological, psychological, or physical conditions of the user. According to the embodiment, the measure of effectiveness may correspond to the degree of therapeutic treatment for the one or more neurological, psychological, or physical conditions of the user. Further according to the embodiment, at least one classification model may be configured to classify one or more variables associated with treating or targeting one or more neurological, psychological, and / or physical conditions of the user.

[0013] A further aspect of the present disclosure provides a processor-implemented system for software quality assurance, comprising at least one server including at least one processor, the at least one server communicatively engaged with one or more computing devices for receiving a plurality of user-generated inputs in response to one or more computer-controlled stimuli or interactions being presented within a test instance of a software build, the plurality of user-generated inputs including user activity data, and a non-transitory computer-readable storage medium operably engaged with the at least one processor and encoded with computer-executable instructions, the computer-executable instructions when executed by the at least one processor for: receiving the user activity data; performing one or more operations to: process user activity data according to at least one classification model for classifying the activity data, the at least one classification model configured to classify one or more variables associated with a targeted stimulus-response pattern; analyze at least one stimulus input pattern between the user activity data and one or more computer-controlled stimuli or interactions to determine one or more performance metrics for the software build, the one or more performance metrics including a measure of safety, performance, or effectiveness of at least one test feature; and evaluate the software build according to the one or more performance metrics to determine a pass / fail status of the software build according to a minimum performance threshold.

[0014] According to the above aspects of the present disclosure, the system for software quality assurance may be further configured such that the classification model includes at least one machine learning framework including an ensemble learning model and / or a supervised learning model. According to an embodiment, the system may be further configured such that the classification model includes a random forest algorithm or a random decision forest algorithm. The system may be further configured such that the at least one server is communicatively coupled to at least one external server via an application program interface to receive a plurality of user-generated inputs.

[0015] According to an embodiment, the system for software quality assurance may be further configured so that the computer-executable instructions further include an operation for presenting user activity data and one or more performance metrics to an administrative user via a graphical user interface. The system may be further configured so that the computer-executable instructions further include an operation for comparing the one or more performance metrics for a software build with a previous or subsequent software build to determine an amount of change in the one or more performance metrics. The system may be further configured so that the computer-executable instructions further include an operation for determining an amount of effective therapy delivery received by a user in response to one or more computer-controlled stimuli or interactions.

[0016] According to an embodiment, the system for software quality assurance may be further configured such that the computer-executable instructions further include an operation for comparing one or more performance metrics for the software build with one or more performance metrics from a previous or subsequent software build to analyze a causal relationship between at least one test feature and the one or more performance metrics. The system may be further configured such that the at least one test feature further includes a design change or variation from a previous version of the at least one test feature. According to an embodiment, the computer-executable instructions may further include an operation for analyzing a causal relationship between the design change or variation and the one or more performance metrics.

[0017] A still further aspect of the present disclosure provides a non-transitory computer-readable storage medium encoded with instructions for directing one or more processors to perform operations for software quality assurance, the operations including: receiving a plurality of user-generated inputs in response to one or more computer-controlled stimuli or interactions being presented within a test instance of the software build, the plurality of user-generated inputs including user activity data; processing the user activity data according to at least one classification model for classifying the user activity data, the at least one classification model configured to classify one or more variables associated with targeted stimulus-response patterns; analyzing the at least one stimulus-response pattern between the user activity data and the one or more computer-controlled stimuli or interactions to determine one or more performance metrics for the software build, the one or more performance metrics including measures of safety, performance, or effectiveness of at least one test feature; and evaluating the software build according to the one or more performance metrics to determine a pass / fail status of the software build according to a minimum performance threshold.

[0018] A still further aspect of the present disclosure provides a method for software quality assurance, comprising: presenting an instance of a software build to a user via a graphical user interface using a user computing device, the software build including at least one feature including one or more computer-controlled stimuli or interactions configured to elicit an expected stimulus input pattern from the user in response to the one or more computer-controlled stimuli or interactions; receiving, using at least one sensor communicatively coupled to the user computing device, a plurality of user inputs in response to the presentation of the one or more computer-controlled stimuli or interactions within the instance of the software build, the plurality of user inputs including user activity data for a session of the software build; and receiving, using a processor communicatively coupled to the computing device, a plurality of user inputs in response to the presentation of the one or more computer-controlled stimuli or interactions within the instance of the software build, the plurality of user inputs including user activity data for a session of the software build. receiving user activity data; processing, with a processor, the user activity data in accordance with at least one data model to determine one or more actual stimulus input patterns for each user input in the plurality of user inputs; comparing the one or more actual stimulus input patterns for each user input in the plurality of user inputs with expected stimulus input patterns for the at least one feature to determine a total number of instances in which the one or more actual stimulus input patterns reflected the expected stimulus input pattern within the software build session; calculating, with the processor, at least one output value for the user activity data in accordance with the at least one data model, the at least one output value including a qualitative or quantitative degree of fit of the model for the at least one feature; and determining a pass / fail status for the software build in accordance with the at least one output value.

[0019] According to the aspect of the present disclosure, the method for software quality assurance may further include calculating an amount of net therapy activity within a session of the software build according to a total number of instances in which one or more actual stimulus input patterns reflected the expected stimulus input patterns. The method may further include calculating an amount of effective therapy delivery for at least one feature within the session of the software build according to a total number of instances in which one or more actual stimulus input patterns reflected the expected stimulus input patterns. According to some embodiments, the at least one data model may include a classification model configured to classify one or more variables associated with one or more performance, safety, or efficacy parameters. According to some embodiments, the method may further include determining a pass / fail status for the at least one feature according to at least one output value. The method may further include comparing the at least one output value with at least one previous output value associated with at least one previous version of the software build to determine an amount of change attributable to the at least one feature. The method may further include comparing an amount of net therapy activity within the session of software build to at least one previous amount of net therapy activity associated with at least one previous version of the software build to determine an amount of change attributable to at least one feature. The method may further include determining a pass / fail status for the software build according to the amount of net therapy activity within the session of software build. The method may further include determining a pass / fail status for the software build according to an amount of effective therapy delivery for at least one feature within the session of software build.

[0020] A still further aspect of the present disclosure provides a system for software quality assurance, comprising: a processor; and a non-transitory computer-readable storage medium communicatively engaged with the processor and encoded with processor-executable instructions that, when executed, cause the processor to present a graphical user interface including an instance of a software build, the software build including at least one feature including one or more computer-controlled stimuli or interactions configured to elicit an expected stimulus input pattern from a user in response to the one or more computer-controlled stimuli or interactions; receiving a plurality of user activity data for a session of the software build, the plurality of user activity data including a plurality of user inputs in response to the one or more computer-controlled stimuli or interactions; processing the user activity data in accordance with at least one data model to determine one or more actual stimulus input patterns for each user input in the plurality of user inputs; comparing the one or more actual stimulus input patterns for each user input in the plurality of user inputs with an expected stimulus input pattern for the at least one feature to determine a total number of instances in which the one or more actual stimulus input patterns reflected the expected stimulus input pattern within the software build session; calculating at least one output value for the plurality of user activity data in accordance with the at least one data model, the at least one output value including a qualitative or quantitative degree of fit of the model for the at least one feature; and determining a pass / fail status for the software build in accordance with the at least one output value.

[0021] According to the above aspect of the present disclosure, the system for software quality assurance may be further configured such that the one or more operations further include calculating an amount of net therapy activity within a session of the software build according to a total number of instances in which one or more actual stimulus input patterns reflected the expected stimulus input patterns. In some embodiments, the one or more operations may further include calculating an amount of effective therapy delivery for at least one feature within a session of the software build according to a total number of instances in which one or more actual stimulus input patterns reflected the expected stimulus input patterns. In some embodiments, the at least one data model may include a classification model configured to classify one or more variables associated with one or more performance, safety, or efficacy parameters. In some embodiments, the one or more operations may further include determining a pass / fail status for the at least one feature according to at least one output value. In some embodiments, the one or more operations may further include comparing the at least one output value with at least one previous output value associated with at least one previous version of the software build to determine an amount of change attributable to the at least one feature.

[0022] In some embodiments, the one or more operations may further include comparing an amount of net therapy activity within the session of the software build to at least one previous amount of net therapy activity associated with at least one previous version of the software build to determine an amount of change attributable to at least one feature. In some embodiments, the one or more operations may further include determining a pass / fail status for the software build according to the amount of net therapy activity within the session of the software build. In some embodiments, the one or more operations may further include determining a pass / fail status for the software build according to an amount of effective therapy delivery for at least one feature within the session of the software build. In some embodiments, the one or more operations may further include determining a pass / fail status for the software build according to at least one safety parameter associated with one or more computer-controlled stimuli or interactions.

[0023] A still further aspect of the present disclosure provides a non-transitory computer-readable storage medium encoded with instructions for directing one or more processors to perform operations for software quality assurance, the operations including: presenting a graphical user interface including an instance of a software build, the software build including at least one feature including one or more computer-controlled stimuli or interactions configured to elicit an expected stimulus input pattern from a user in response to the one or more computer-controlled stimuli or interactions; receiving a plurality of user activity data for a session of the software build, the plurality of user activity data including a plurality of user inputs in response to the one or more computer-controlled stimuli or interactions; processing the plurality of user activity data according to the at least one data model to determine one or more actual stimulus input patterns for each user input in the plurality of user inputs; comparing the one or more actual stimulus input patterns for each user input in the plurality of user inputs with expected stimulus input patterns for the at least one feature to determine a total number of instances in which the one or more actual stimulus input patterns reflected the expected stimulus input pattern within the session of the software build; calculating at least one output value for the plurality of user activity data according to the at least one data model, the at least one output value including a qualitative or quantitative degree of fit of the model for the at least one feature; and determining a pass / fail status for the software build according to the at least one output value.

[0024] The foregoing has outlined, rather broadly, the more pertinent and important features of the present invention in order that the detailed description thereof that follows may be better understood, so that the present contribution to the art may be more fully appreciated. Additional features of the invention will be described hereinafter and form the subject of the claims of the invention. It will be appreciated by those skilled in the art that the concepts and specific methods and structures disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. It will also be recognized by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the invention.

[0025] Those skilled in the art will understand that the figures described herein are for illustrative purposes only. It should be understood that in some cases, various aspects of the described embodiments may be shown exaggerated or enlarged to facilitate understanding of the described embodiments. In the drawings, like reference characters generally refer to like features, functionally similar, and / or structurally similar elements throughout the various views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the teachings. The drawings are not intended to limit the scope of the present teachings in any way. The present systems and methods may be better understood from the following exemplary description, with reference to the following drawings: [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a functional block diagram of an exemplary computing system in which certain aspects of the present disclosure may be implemented. [Figure 2] FIG. 1 is an architecture diagram of a software design management system according to certain aspects of the present disclosure. [Figure 3] FIG. 1 is a functional block diagram of a software design management system according to certain aspects of the present disclosure. [Figure 4] FIG. 1 is a functional block diagram of a software design management system according to certain aspects of the present disclosure. [Figure 5] FIG. 1 is a functional block diagram of a software design management system according to certain aspects of the present disclosure. [Figure 6] FIG. 1 is a process flow diagram of a system and method for software design management and quality assurance according to certain aspects of the present disclosure. [Figure 7] FIG. 1 is a functional block diagram of a software design management system according to certain aspects of the present disclosure. [Figure 8] FIG. 2 is a functional block diagram of a routine for classifying user activity data within a software design management system, according to an aspect of the present disclosure. [Figure 9] FIG. 2 is a functional block diagram of a routine for determining the pass / fail status of a software build within a software design management system, according to certain aspects of the present disclosure. [Figure 10] FIG. 2 is a functional block diagram of a routine for determining the impact of a design change in a software design management system, according to certain aspects of the present disclosure. [Figure 11] FIG. 1 is a process flow diagram of a method for software design management and quality assurance according to certain aspects of the present disclosure. [Figure 12] FIG. 1 is a process flow diagram of a method for software design management and quality assurance according to certain aspects of the present disclosure. [Figure 13] FIG. 1 is a process flow diagram of a method for software design management and quality assurance according to certain aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0027] It will be understood that all combinations of concepts described in more detail below (provided such concepts are not inconsistent) are contemplated as part of the inventive subject matter disclosed herein. It will also be understood that terms explicitly employed herein that may also appear in any disclosure incorporated by reference should be given the meaning that most closely matches the specific concepts disclosed herein.

[0028] The following is a more detailed description of various concepts related to inventive methods, apparatus, systems, and non-transitory computer-readable storage media having instructions stored thereon, and embodiments thereof, wherein the instructions stored on the non-transitory computer-readable storage media include instructions for one or more of the methods, apparatus, and systems to: present, using a computing device, a test instance of a software build to a user, the software build including at least one test feature configured to present one or more computer-controlled stimuli or interactions to the user via a graphical user interface; receive, using the computing device, one or more user inputs in response to the one or more computer-controlled stimuli or interactions, the one or more user inputs including user activity data; receiving user activity data using a processor communicatively coupled to the playing device; processing the user activity data using the processor according to at least one classification model for classifying the activity data, the at least one classification model configured to classify one or more variables associated with targeted stimulus-response patterns; analyzing stimulus input patterns between the user activity data and one or more computer-controlled stimuli or interactions to determine one or more performance metrics for the software build, the one or more performance metrics including a measure of effectiveness of at least one test feature; and evaluating the software build according to the one or more performance metrics to determine a pass / fail status of the software build according to a minimum performance threshold.

[0029] It will be appreciated that the various concepts introduced above and described in further detail below can be implemented in any of numerous ways, as the disclosed concepts are not limited to any particular manner of implementation. Examples of specific implementations and applications are provided primarily for illustrative purposes. The present disclosure should in no way be limited to the exemplary implementations and techniques illustrated in the drawings and described below.

[0030] Where a range of values ​​is provided, unless the context clearly dictates otherwise, it is understood that each intermediate value, to the tenth of the unit of the lower limit, between the upper and lower limits of that range, and any other stated or intermediate value within the stated range, is encompassed by the invention. The upper and lower limits of these smaller ranges may be independently included within the smaller ranges and are also encompassed by the invention, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the endpoint boundaries, ranges excluding either or both of those included endpoints are also included within the scope of the invention.

[0031] As used herein, "exemplary" means serving as an example or illustration and does not necessarily represent the ideal or best.

[0032] As used herein, the term "include" means including but not limited to, and the term "including" means including but not limited to. The term "based on" means based at least in part on.

[0033] As used herein, the term "software build" refers to a version, intermediate product, and / or compilation of program code that includes a software program or product and / or one or more component(s) or subcomponent(s) of a software program or product. In one embodiment, "software build" refers to a version, intermediate product, and / or compilation of program code that is the subject of one or more software quality assurance processes in association with a software program. In one embodiment, "software build" refers to a version, intermediate product, and / or compilation of program code that is pending release or publication in association with a software product.

[0034] As used herein, the term "toolchain" refers to any set of programming tools used to perform one or more software development task(s) and / or create a software product, which may include other computer programs and / or sets of related programs. In an embodiment, the programming tools that comprise a toolchain may be executed in series, such that the output or resulting environmental state of each tool becomes the input or starting environmental state of the next tool. In an embodiment, the programming tools that comprise a toolchain may include a set of related programming tools that may or may not be executed in series.

[0035] As used herein, the term "stimulus" refers to a sensory event configured to elicit a specified functional response from an individual. The degree and type of response can be quantified based on the individual's interaction with a measurement component (including the use of a sensor device or other measurement component).

[0036] As used herein, the term "user activity data" refers to data collected from measuring user interactions with a software program, product, and / or platform.

[0037] As used herein, the term "computerized stimuli or interaction" or "CSI" refers to a computerized element presented to a user to facilitate the user's interaction with the stimuli or other interaction. As non-limiting examples, a computing device can be configured to present an auditory stimulus (e.g., presented as an auditory computerized adjustable element or element of a computerized auditory task) or initiate other auditory-based interaction with the user, and / or present a vibration stimulus (e.g., presented as a vibratory computerized adjustable element or element of a computerized vibration task) or initiate other stimulus-based interaction with the user, and / or present a tactile stimulus (e.g., presented as a tactile computerized adjustable element or element of a computerized tactile task) or initiate other tactile-based interaction with the user, and / or present a visual stimulus or initiate other visual-based interaction with the user.

[0038] In examples where the computing device is configured to present visual CSI, the CSI is provided as at least one user interface presented to the user. In some examples, the at least one user interface is configured to measure responses as the user interacts with the CSI computerized element provided in the at least one user interface. In a non-limiting example, the user interface can be configured such that the CSI computerized element(s) are active and may request at least one response from the user, such that the user interface is configured to measure data indicative of the type or extent of the user's interaction with the platform product. In another example, the user interface can be configured such that the CSI computerized element(s) are passive and are presented to the user using the at least one user interface, but may not request a response from the user. In this example, the at least one user interface can be configured to exclude recorded responses of the user's interactions and apply a weighting factor to the data indicative of the responses (e.g., weight responses toward lower or higher values), or to measure the data indicative of the user's response with the platform product as a measure of the user's incorrect responses (e.g., to issue a notification or other feedback to the user of the incorrect response).

[0039] As used in certain examples herein, the term "user" encompasses one or more end users and / or test users of a software program, product, and / or platform, and may further include patients who engage with a software program, product, or platform for targeted medical or personal wellness purposes; participants in clinical trials, research, or evaluations of a software program, product, or platform; and users who engage with a software program, product, or platform for purposes of evaluating or developing one or more technical, clinical, and / or functional aspects of the software as a digital health intervention and / or medical device program, product, or platform.

[0040] As used herein, the terms "digital health intervention (DHI)" and "software as a medical device (SaMD)" may be used interchangeably to encompass any software program, product, or platform, including any software / hardware combination, designed and / or utilized for any general or targeted medical or personal wellness purpose, including but not limited to, treatment, diagnosis, management, prevention, therapy, or generating / providing clinical / health / wellness insights or recommendations to one or more users for one or more medical, health, or personal wellness purposes; promoting healthy behaviors, improving outcomes, and providing effective treatments in people with long-term conditions, such as cardiovascular disease, diabetes, and mental health conditions; for example, computer-controlled cognitive behavioral therapy for mental health and physical problems; It may further encompass one or more software programs, products, or platforms, including any product(s), program(s), and / or platform(s) that integrate any combination of hardware and software that is not merely a component of an overall treatment, but has a direct therapeutic effect in treating and / or targeting one or more neural circuits associated with one or more neurological, psychological, and / or physical conditions, diseases, and / or disorders.

[0041] All definitions should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or the ordinary meaning of the defined terms as defined and used herein. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, exemplary methods and materials are described herein. All publications mentioned herein are incorporated by reference to disclose and describe the methods and / or materials in connection with which the publications are cited.

[0042] Exemplary systems, methods, and non-transitory computer-readable storage media consistent with the principles herein provide a design management architecture and quality assurance system for analyzing stimulus-response patterns in user activity data to evaluate the effectiveness, safety, and / or performance of one or more features of a software product. In some embodiments, the software product is configured as a DHI or SaMD. According to some embodiments, the design management architecture and quality assurance system is configured to process the user activity data according to a classifier model to evaluate the impact of incremental design changes on the effectiveness, safety, and / or performance of the software product. Exemplary systems and methods may be configured to control or inform one or more software development processes associated with a software program, platform, and / or product. The one or more software development processes may include one or more of software design, requirements analysis, feature verification, build verification, code review, source code control, quality control / assurance, software configuration management, verification, release management, and / or product integration.

[0043] Exemplary systems, methods, and non-transitory computer-readable storage media according to the principles herein provide for analyzing one or more patterns in user activity data from one or more instances or sessions of a DHI or SaMD. According to certain embodiments, the one or more patterns may include procedural analysis to determine the degree of agreement with one or more clinically validated stimulus-response patterns. Certain embodiments of the present disclosure may include a machine learning framework, classifier model, and / or classification algorithm for classifying and / or mapping user input into one or more categories. In certain embodiments, the one or more categories may correspond to one or more safety, efficacy, and / or performance aspects of a DHI or SaMD product.

[0044] Exemplary systems, methods, and non-transitory computer-readable storage media according to the principles herein include systems and methods for measuring the performance of a DHI or SaMD. According to certain embodiments, the systems and methods of the present disclosure are configured to process user activity data according to a classifier model to measure and / or quantify the amount of therapeutic activity and / or effectiveness of a DHI or SaMD. In examples herein, the systems, methods, and non-transitory computer-readable storage media provide for measuring and / or verifying the effectiveness of a DHI or SaMD to treat and / or target one or more neural circuits associated with one or more neurological, psychological, and / or physical conditions, diseases, and / or disorders of a user. According to such example(s), measuring and / or verifying the effectiveness of a DHI or SaMD may further include measuring and / or quantifying the amount of therapeutic activity and / or effectiveness of one or more individual features or aspects of the DHI or SaMD. In examples herein, systems, methods, and non-transitory computer-readable storage media are provided for evaluating the impact of incremental design changes on the effectiveness of a DHI or SaMD to treat and / or target one or more neural circuits associated with one or more neurological, psychological, and / or physical conditions, diseases, and / or disorders of a user.

[0045] Systems, methods, and computer platform products according to the principles herein enable a software product manufacturer to provide user activity data from one or more instances or sessions of the software product to a cloud server configured to analyze one or more patterns within the user activity data. The system is configured to process the user activity data to determine the degree of agreement with one or more clinically validated stimulus-response patterns to provide a quantitative analysis of one or more safety, efficacy, or performance aspects of the software product. According to certain embodiments, the software product manufacturer may use the resulting analysis to inform or control one or more software development processes for the software product, including, but not limited to, software design, requirements analysis, feature testing, build testing, code review, source code control, quality control / assurance, software configuration management, testing, release management, and / or product integration.

[0046] Aspects of the present disclosure provide a design management architecture and software quality assurance system for analyzing stimulus-response patterns in user activity data. Exemplary embodiments of the present disclosure provide a design management architecture and software quality assurance system configured to process user activity data received from user devices executing instances of a software build, process the user activity data according to a classifier model, and process a framework to enable a developer user to evaluate the impact of incremental design changes on one or more quality measures of the software product. According to certain embodiments, the one or more quality measures may include one or more efficacy, safety, and / or performance metrics associated with one or more features of the software product. According to various embodiments, the design management architecture and software quality assurance system enable developers of software products, including software as digital health interventions or medical device products, to perform qualitative and / or quantitative analysis of one or more safety, efficacy, and / or performance metrics of the software product and specific features thereof.

[0047] Referring now in detail to the drawings, in which like reference characters denote like elements throughout the several views, FIG. 1 illustrates a computing system in which certain exemplary embodiments of the present disclosure may be implemented.

[0048] Referring now to FIG. 1 , a processor-implementing computing device is illustrated in which one or more aspects of the present disclosure may be implemented. According to one embodiment, processing system 100 generally includes at least one processor 102, memory 104, an input device 106 for receiving input data 118, and an output device 108 for generating output data 120, coupled together via at least one bus 110. In some embodiments, input device 106 and output device 108 may be the same device. An interface 112 may also be provided for coupling processing system 100 to one or more peripheral devices; for example, interface 112 may be a PCI card or PC card. At least one database storage device 114 containing at least one database 116 may also be provided. Memory 104 may be any type of memory device, such as volatile or non-volatile memory, solid-state storage, magnetic devices, etc. Processor 102 may include two or more separate processing units, for example, for processing different functions within processing system 100. The input devices 106 receive input data 118 and may include, for example, a keyboard, a pointer device such as a pen-like device or mouse, an audio receiver for voice control activation such as a microphone, a data receiver or antenna such as a modem or wireless data adapter, a data acquisition card, etc. The input data 118 may come from different sources, for example, keyboard commands along with data received over a network. The output devices 108 create or generate output data 120 and may include, for example, a display device or monitor if the output data 120 is visible, a printer if the output data 120 is printed, a port such as a USB port, a peripheral component adapter, a data transmitter or antenna such as a modem or wireless network adapter, etc. The output data 120 may be separate and originate from a different output device, for example, a visual display on a monitor along with data transmitted over a network. A user may view the data output, or an interpretation of the data output, for example, on a monitor or using a printer. The storage device 114 may be any form of data or information storage means, for example, volatile or non-volatile memory, solid-state storage, magnetic devices, and the like.

[0049] In use, the processing system 100 is adapted to allow data or information to be stored in and / or retrieved from at least one database 116 via wired or wireless communication means. The interface 112 may allow wired and / or wireless communication between the processing unit 102 and peripheral components that may serve special purposes. Generally, the processor 102 may receive instructions as input data 118 via the input device(s) 106 and may display processing results or other output to a user by utilizing the output device(s) 108. More than one input device 106 and / or output device 108 may be provided. It will be understood that the processing system 100 may be any form of terminal, server, dedicated hardware, or the like.

[0050] It will be appreciated that processing system 100 may be part of a networked communication system. Processing system 100 may be connected to a network, for example, the Internet or a WAN. Input data 118 and output data 120 may be communicated to other devices via the network. Transfer of information and / or data via the network may be accomplished using wired or wireless communication means. A server may facilitate the transfer of data between the network and one or more databases. The server and one or more databases provide examples of information sources.

[0051] 1 may operate in a networked environment using logical connections to one or more remote computers, which may be personal computers, servers, routers, network PCs, peer devices or other common network nodes, and typically include many or all of the elements listed above.

[0052] It will be further understood that the logical connections depicted in FIG. 1 include a local area network (LAN) and a wide area network (WAN), but may also include other networks, such as a personal area network (PAN). Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets, and the Internet. For example, when used in a LAN networking environment, computing system environment 100 is connected to the LAN through a network interface or adapter. When used in a WAN networking environment, the computing system environment typically includes a modem or other means for establishing communications over the WAN, such as the Internet. The modem, which may be internal or external, may be connected to the system bus through a user input interface or through another appropriate mechanism. In a networked environment, program modules depicted relative to computing system environment 100, or portions thereof, may be stored in a remote memory storage device. It will be understood that the illustrated network connections in FIG. 1 are exemplary and other means of establishing a communications link between computers may be used.

[0053] FIG. 1 is intended to provide a brief overview of an illustrative and / or suitable representative environment in which various embodiments of the present invention may be implemented. FIG. 1 is one example of a suitable environment and is not intended to suggest any limitations with respect to the structure, scope of use, or functionality of an embodiment of the present invention. A particular environment should not be interpreted as having any dependency or requirement regarding any one component or combination of components illustrated in the exemplary operating environment. For example, in some cases, one or more elements of the environment may not be deemed necessary and may be omitted. In other cases, one or more other elements may be deemed necessary and may be added.

[0054] In the description that follows, certain embodiments may be described with reference to symbolic representations of acts and operations that are performed by one or more computing devices, such as computing system 100 of FIG. 1 . As such, it will be understood that such acts and operations, sometimes referred to as computer-executed, include the manipulation by the computer's processor of electrical signals that represent data in a structured form. This manipulation transforms the data or maintains them in storage locations within the computer's memory system, which reconfigures or otherwise alters the operation of the computer in a manner understood by those skilled in the art. The data structures in which data are maintained are physical locations in memory that have particular characteristics defined by the format of the data. However, while one embodiment is described in the foregoing context, it is not intended to be limiting, as those skilled in the art will understand that the acts and operations described below may also be implemented in hardware.

[0055] Embodiments of the present invention can be implemented with numerous other general-purpose or special-purpose computing devices, systems, or configurations. Examples of well-known computing systems, environments, and configurations suitable for use with embodiments of the present invention include personal computers, handheld or laptop devices, personal digital assistants, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, networks, minicomputers, server computers, game server computers, web server computers, mainframe computers, and distributed computing environments that include any of the foregoing systems or devices.

[0056] Various embodiments of the invention are described herein in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In some embodiments, distributed computing environments may also be employed where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0057] In conjunction with the general-purpose computing system environment 100 of FIG. 1 shown and described above, the following description and remaining figures relate to various exemplary embodiments of the present invention generally relating to a system and method for procedural analysis of stimulus input patterns within user test data to evaluate and validate one or more performance metrics associated with a program code build. In general, methods described herein involve presenting, using a computing device, a test instance of a software build to a user, the software build including at least one test feature configured to present one or more computer-controlled stimuli or interactions to the user via a graphical user interface; receiving, using the computing device, one or more user inputs in response to the one or more computer-controlled stimuli or interactions, the one or more user inputs including user activity data; receiving, using a processor communicatively coupled to the computing device, the user activity data; processing, using the processor, the user activity data according to at least one classification model for classifying the activity data, the at least one classification model configured to classify one or more variables associated with a targeted stimulus-response pattern; analyzing the stimulus input pattern between the user activity data and the one or more computer-controlled stimuli or interactions to determine one or more performance metrics for the software build, the one or more performance metrics including a measure of effectiveness of at least one test feature; and evaluating the software build according to the one or more performance metrics to determine a pass / fail status of the software build according to a minimum performance threshold.

[0058] Referring now to FIG. 2 , an architectural diagram of a software design management system 200 is shown. According to certain aspects of the present disclosure, system 200 includes a software development subsystem 202 and a design management subsystem 204. Software development subsystem 202 and design management subsystem 204 may be communicatively coupled through a communications network 216 (e.g., the Internet). In certain embodiments, software development subsystem 202 and design management subsystem 204 may be configured as independent systems communicatively coupled through an application programming interface or other data transfer protocol. In other embodiments, software development subsystem 202 and design management subsystem 204 may be related subsystems within a larger integrated system. For example, software development subsystem 202 and design management subsystem 204 may be related components (contiguous or non-contiguous) of a development operations (DevOps) toolchain. According to various aspects of the present disclosure, software development subsystem 202 and design management subsystem 204 may be configured as separate software modules executing on the same server; or software development subsystem 202 and design management subsystem 204 may be configured as separate software modules executing on different servers within a local computing environment (i.e., LAN); or software development subsystem 202 and design management subsystem 204 may be configured as separate software modules executing on different servers across a distributed computing environment (i.e., WAN). In an embodiment, software development subsystem 202 and design management subsystem 204 may be operatively coupled via one or more third-party servers 218. For example, third-party server 218 may include a cloud repository where program code associated with a software program may be stored remotely within the software development environment.Cloud repositories may include versioning platforms, such as GITHUB® or BITBUCKET®; and may further include cloud storage and / or computing environments, such as those available from AMAZON WEB SERVICES®. In an embodiment, third-party server 218 may provide one or more services, data, and / or processing capabilities to software development subsystem 202 and / or design management subsystem 204 (e.g., design management module 208 and / or software development module 222 may run, in whole or in part, on third-party server 218).

[0059] According to one embodiment, software development subsystem 202 may be defined as a software development environment for developing at least one software program 228. In one embodiment, software program 228 is configured as a DHI or SaMD product. Software development subsystem 202 may include at least one developer workstation 224 configured to execute one or more workstation applications 228, at least one local and / or remote server 220 including one or more software development modules 222, and at least one database 226 communicatively coupled to server 220 and one or more software development modules 222. Design management subsystem 204 may be comprised of a design management server 206, a design management module 208 executing on design management server 206, a design management database 210 communicatively coupled to design management server 206 and design management module 208, and a design management workstation 214. In one embodiment, design management workstation 214 may be configured to execute an instance of design management application 214; developer workstation 224 may be configured to execute an instance of design management application 214'.

[0060] According to an embodiment, software development subsystem 202 and design management subsystem 204 may be operatively engaged to define one or more deployment environments or tiers associated with software program 228. Some exemplary deployment environments or tiers in which software development subsystem 202 and design management subsystem 204 may be operatively engaged to develop, deploy, and / or execute one or more aspects of software program 228 may include any combination of the environments or tiers shown and described in Table 1. [Table 1]

[0061] According to an embodiment, system 200 may further include a user device 218 communicatively coupled with server 220 to execute an instance 228′ of software program 228. Instance 228′ may include an instance of software program 228 associated with one or more development tiers, as shown and described in Table 1. For example, instance 228′ may be configured as a test instance of software program 228. In an embodiment, user device 218 may be configured as an end-user device (i.e., an external user), while in other embodiments, user device 218 may be configured as a developer-user device (i.e., an internal user). User device 218 may include one or more computing devices operable to execute a software program, including, but not limited to, a smartphone, a tablet computer, a wearable electronic device (e.g., a smartwatch), a laptop computer, and the like. User device 218 may be communicatively coupled with one or more of server 220, third-party server 218, and / or design management server 206 via communication network 216.

[0062] Referring now to FIG. 3 , a functional block diagram of a software design management system 300 is shown. According to certain aspects of the present disclosure, system 300 may include a development subsystem 302 and a design management subsystem 304. In an embodiment, development subsystem 302 and design management subsystem 304 may be configured as software development subsystem 202 and design management subsystem 204, as shown and described in FIG. 2 . According to an embodiment, system 300 is configured to be operable to receive, process, and analyze stimulus-response patterns within user activity data to evaluate the effectiveness, safety, and / or performance of one or more features of the software product. In an embodiment, the software product is configured as a DHI or SaMD.

[0063] According to one embodiment, the development subsystem 302 may be configured to execute instances 308 of the software program 306 within one or more development tiers. In one embodiment, the development tier is a testing or pre-production tier. In other embodiments, the development tier is a production or live tier. In embodiments where the development tier is a testing or pre-production tier, the software program 306 may include builds of a version or sub-version of the software product that has not yet been deployed to the production or live tier. The development subsystem 302 may be configured to receive user activity data associated with one or more instances 308 of the software program 306. In one embodiment, the software program 306 includes at least one engine / algorithm configured according to at least one clinically validated stimulus-response pattern. In one embodiment, the at least one clinically validated stimulus-response pattern may include a therapeutic mechanism of action for treating or targeting one or more neurological, psychological, and / or physical conditions. The software program 306 may be configured to provide stimuli 316 including one or more CSIs within the instance 308 of the software program 306 executing on a user device. According to an embodiment, the stimuli 316 may include one or more test feature(s) or incremental design change(s) embodied in one or more CSIs and / or one or more graphical user interface elements of the instance 308. The instance 308 may be configured to receive one or more user inputs 318 in response to the stimuli 316 and provide the inputs to the software program 306. The software program 306 may provide the one or more stimuli 316 and receive the one or more inputs 318 via the instance 308 to include at least one session of the software program 306 within one or more development layers. In one embodiment, the at least one session may include a testing session. The one or more stimuli 316 and inputs 318 presented and received within the at least one session of the software program 306 may include a plurality of user activity data.

[0064] Continuing with reference to FIG. 3 , user activity data from the design management subsystem 304 may be communicated to the design management subsystem 304 via a data transfer interface 320 in real time or via a batch processing protocol. The data transfer interface 320 may include a network communication protocol for the application programming interface (API) 310. In an embodiment, the data transfer interface 320 may also include a server push for cloud storage 310 of user activity data in one or more databases (e.g., the design management database 210 as shown in FIG. 2 ). The design management subsystem 304 may be configured to communicate the user activity data to the classification module 312 for processing. In an embodiment, the classification module 312 may optionally be communicatively coupled to one or more databases configured to store user activity data via cloud storage 310 through a simple notification service / simple queuing service 324. The simple notification service / simple queuing service 324 may be configured as a publish / subscribe model between the classification module 312 and one or more databases configured to store user activity data via cloud storage 310. The classification module 312 may be configured to execute one or more classification algorithms configured to process and classify user activity data according to at least one classifier model. In an embodiment, the classification module 312 may be configured as a procedural reflection of at least one engine / algorithm of the software program 306. The at least one classifier model of the classification module 312 may be configured to classify one or more variables associated with at least one clinically validated stimulus-response pattern. In an embodiment, the at least one classifier model of the classification module 312 may be configured to classify one or more variables associated with a clinically validated stimulus-response pattern to treat or target one or more neurological, psychological, and / or physical conditions.The classification model may be configured as a machine learning framework, including an ensemble learning model and / or a supervised learning model. The ensemble learning model may include two or more algorithms configured to analyze two or more respective variables, or metrics, within the instance 308. For example, the ensemble learning framework may include a procedural algorithm that analyzes safety metrics within the user data (e.g., whether the user is in a moving vehicle) and a classifier algorithm that analyzes efficacy metrics (e.g., whether one or more stimulus input patterns reflect an expected outcome). In some embodiments, the classification model may include a random forest algorithm and / or a random decision forest algorithm. In such embodiments, one or more features of the software program 306 may be represented as node(s) in the random forest or random decision forest algorithm. The classification module 312 may be configured to process the user activity data according to the classification model to classify (322) one or more variables within the user activity data. According to some embodiments, the one or more variables may include one or more stimulus input patterns between the user activity data and one or more computer-controlled stimuli or interactions. For example, a feature of the software program 306 may include a type or category of CSI, such as a target discriminative stimulus or a visuomotor tracking task. According to such an embodiment, within an instance 308, the target discriminative stimulus (i.e., Feature A) may be configured to evoke a first type of user response (i.e., Input Type A), and the visuomotor tracking task (i.e., Feature B) may be configured to evoke a second type of user response (i.e., Input Type B). The classification module 312 may be configured to enable a developer-user to configure Feature A and Feature B as nodes in a classification model, and Input Type A and Input Type B as variables associated with targeted (i.e., expected) stimulus-response patterns.The classification module 312 may be further configured to receive user activity data from one or more previous instances of the software program 306 and process the data to generate a training data set for the classification model.

[0065] According to an embodiment, the design management subsystem 304 may provide the output of the classification module 312 to a reporting module 314. The reporting / review module 314 may be configured to process the output of the classification module 312 to generate one or more performance metrics for the software program 306. In an embodiment, the reporting / review module 314 may be configured to analyze stimulus input patterns between user activity data and one or more computer-controlled stimuli or interactions to determine one or more performance metrics for the software program 306. The one or more performance metrics may include quantified measures of one or more variables associated with clinically validated stimulus-response patterns for treating or targeting one or more neurological, psychological, and / or physical conditions in a user. In an embodiment, the one or more performance metrics may include qualitative or quantitative measures of the safety, effectiveness, and / or performance of at least one test feature or incremental design modification to the software program 306. The reporting module 314 may be further configured to process the output of the classification module 312 to analyze performance metrics of the software program 306 and / or determine a pass / fail status according to a minimum performance threshold. In an embodiment, the minimum performance threshold is a minimum model match between the actual stimulus-response pattern(s) represented in the user activity data and the expected (e.g., clinically validated) stimulus-response pattern(s) for treating or targeting one or more neurological, psychological, and / or physical conditions. The minimum performance threshold may further include a minimum amount of therapeutic activity delivered to the user in response to the instance 308 of the software program 306 and / or a minimum amount of therapeutic activity delivered to the user on a per-feature (e.g., per CSI) basis. The reporting module 314 may be further configured to provide a graphical user interface including one or more data visualization and / or data query functions configured to enable the developer user to perform a review 324 on the software program 306.The review 324 may be associated with one or more software development processes, including, but not limited to, software / feature design, requirements analysis, feature inspection, build inspection, code review(s), source code control, quality control / assurance, software configuration management, testing, release management, product integration, and the like. According to an embodiment, a developer user may analyze one or more safety, efficacy, and / or performance metrics for the software program 306 to evaluate a pass / fail status for one or more test feature(s) or incremental design change(s) for the instance 308. According to an embodiment, the pass / fail status may include a minimum net therapeutic effect threshold. The minimum net therapeutic effect threshold may define a minimum amount of therapeutic activity delivered to a user within the instance 308. The reporting / review module 314 may be configured to calculate the minimum net therapeutic effect based on the total number of instances in the user activity data where the actual stimulus input pattern reflected the expected stimulus input pattern, on a per-CSI basis and in aggregate for the instance 308.

[0066] Referring now to FIG. 4 , a functional block diagram of a software design management system 400 is shown. According to certain aspects of the present disclosure, the software design management module 402 may include components of a DevOps toolchain. The software design management system 400 may be incorporated within one or more aspects of the system 200 of FIG. 2 and / or the system 300 of FIG. 3 . The system 400 may include a design management module 402 configured to enable a developer user 416 to analyze and verify one or more safety, efficacy, and performance aspects for one or more design changes and / or test features of a software program / product 404. In certain embodiments, the software program / product includes a DHI or SaMD product. The design management module 402 may include a classification engine 406 and / or input from the classification engine 406. The classification engine 406 may include a classification model 408 that represents a procedural reflection of at least one algorithm 412. The classification engine 406 may configure or select the classification model 408 in response to the algorithm 412. The algorithm 412 may correspond to at least one clinically validated stimulus-response pattern. The algorithm 412 may provide the basis for at least one processing engine for the software program 404. The design management module 402 may receive user activity data 410 in response to an instance 414 of the software program 404 having one or more design changes or test features. In an embodiment, the instance 414 includes a version of the software program 404 running in a development tier other than the production / live tier (e.g., a testing tier or a pre-production tier). According to one embodiment, the design management module 402 receives the user activity data 410 and provides it to the classification engine 406. The classification engine 406 processes the user activity data 410 to determine and identify one or more variables that reflect the clinically validated stimulus-response pattern of the algorithm 412. The design management module 402 processes the output of the classification engine 406 to determine one or more performance metrics for the design change(s) and / or test feature(s) 414.The design management module 402 may provide performance metrics and other data visualizations corresponding to the user activity data and / or classification models to the developer user 416. The developer user 416 may review the performance metrics to determine the impact of the design change(s) and / or test feature(s) 414 on one or more safety, effectiveness, and / or performance aspects of the software program 404.

[0067] Referring now to FIG. 5 , a functional block diagram of a software design management process 500 is shown. According to certain aspects of the present disclosure, the software design management process 500 may be incorporated into a software design management system, such as those shown and described in any one of FIGS. 2-4 . The design management process 500 may include steps for evaluating and verifying one or more features and / or design changes in a version of a software product in one or more development environments, such as a testing environment and / or a pre-production environment. According to one embodiment, the design management process 500 may include submitting software version X 502 in the design management system. Software version X 502 may include one or more features and / or design changes, e.g., feature A 504, feature B 506, and feature N 508, to be evaluated and verified. User activity data 526 may be received in response to presenting each of feature A 504, feature B 506, and feature N 508 to a user via an instance (e.g., a testing instance or a pre-production instance) of software version X 502. Design management process 500 may further include running a classification model for each of feature A 504, feature B 506, and feature N 508 to generate classification 514, classification 516, and classification 518. According to one embodiment, classification 514 may include a procedural reflection and a clinically validated stimulus-response pattern between the stimulus input pattern of user activity data 526 and feature A 504. Similarly, classification 516 may include a procedural reflection and a clinically validated stimulus-response pattern between the stimulus input pattern of user activity data 526 and feature B 506, and classification 518 may include a procedural reflection and a clinically validated stimulus-response pattern between the stimulus input pattern of user activity data 526 and feature N 508. According to one embodiment, design management process 500 may be configured to determine the success or failure of a feature and deliver a desired stimulus-response pattern into software version X 502 based on the classification of the user activity data. In the illustrated example, classification 518 indicates a failure of feature N 508 to deliver a desired stimulus-response pattern in user activity data 526 .According to one embodiment, design management process 500 may be further configured to determine, in aggregate, the success or failure of software version X 502 in delivering the desired stimulus-response pattern based on the classification of user activity data. In the illustrated example, classification 518 indicates the success of software version X 502 in delivering the desired stimulus-response pattern in user activity data 526. Design management process 500 may be further configured to process one or more of classifications 514, 516, 518, 520 to perform one or more design controls 524. According to one embodiment, design control 524 may include rejecting one or more non-conforming features and / or design changes (e.g., feature N 508) through one or more software quality assurance processes. According to an embodiment, design control 524 may include calculating one or more performance metrics for software version X 502 based on one or more of classifications 514, 516, 518, 520. Design management 524 may optionally be further configured to compare one or more performance metrics associated with software version X 502 with performance metrics associated with one or more previous versions. Design management process 500 may be further configured to accept or reject one or more of feature A 504, feature B 506, and / or feature N 508 to define software version X' 510.

[0068] Referring now to FIG. 6, a process flow diagram of a software validation process 600 is shown. According to certain aspects of the present disclosure, the software validation process 600 may be incorporated into a software design management system, such as those shown and described in any one of FIGS. 2-4. According to one embodiment, the software validation process 600 includes a software program 602 including at least one new feature or design change 604 being deployed (610) within a development environment 606. One or more users 608 may execute an instance 612 of the software program 602 including the at least one new feature or design change 604. User activity data 622 may be processed according to a classification model 614. The classification model 614 may be configured to classify one or more variables associated with clinically validated stimulus-response patterns. The software validation process 600 may further be configured to evaluate (616) whether the user activity data 622 reflects (618) a desired stimulus-response pattern or does not reflect (620) a desired stimulus-response pattern. If the user activity data 622 does not reflect the desired stimulus-response pattern according to the classification 614 (620), the software validation process 600 may be configured to assign a FAIL status to the feature or design change 604. If the user activity data 622 reflects the desired stimulus-response pattern according to the classification 614 (618), the software validation process 600 may be configured to assign a PASS status to the feature or design change 604.

[0069] Referring now to FIG. 7 , a functional block diagram of a software design management system 700 is shown. According to certain aspects of the present disclosure, a development platform 702 may be communicatively coupled to a development environment 704 and a third-party environment 706 via one or more APIs. According to certain embodiments, the development environment 704 may include source code for application A 716 and application B 718. In certain embodiments, the third-party environment 706 may include source code for application C 720. In certain embodiments, application A 716 may be communicatively coupled to an engine A API 708 to configure one or more CSIs within application A 716. Similarly, application B 718 may be communicatively coupled to an engine B API 710 to configure one or more CSIs within application B 718, and application C 720 may be communicatively coupled to an engine C API 714 to configure one or more CSIs within application C 720. Application A 716, application B 718, and / or application C 720 may be communicatively coupled to a design management API 712 to provide user activity data and application data to a design management subsystem 728. The design management subsystem 728 may be configured to process the user activity data and application data according to one or more classification model(s). According to the illustrated example, the design management subsystem 728 is configured to execute classification model A 722 to classify the user activity data associated with application A 716, execute classification model B 724 to classify the user activity data associated with application B 718, and execute classification model C 726 to classify the user activity data associated with application C 720. According to an embodiment, the design management subsystem 728 includes tools in a DevOps toolkit within the development environment 704 and / or the third party environment 706.

[0070] Referring now to FIG. 8 , a functional block diagram of a routine 800 for classifying user activity data in a software design management system is shown. According to certain aspects of the present disclosure, the routine 800 may be incorporated into a software design management system such as those shown and described in any one of FIGS. 2-4 . The routine 800 may include one or more steps performed between a development subsystem and a design management subsystem, e.g., the development subsystem 202 and / or 302 and the design management subsystem 204 and / or 304 (as shown and described in FIGS. 2-3 , respectively). According to one embodiment, the routine 800 may begin by establishing (802) a communication interface between the development subsystem and the design management subsystem. The design management subsystem may be configured to configure (804) one or more classification model(s) in response to one or more application data from the development subsystem. Configuring the one or more classification model(s) 804 may further include determining (824) one or more model variables including one or more performance metrics. The routine 800 may continue by compiling (806) updated program code including one or more new features and / or incremental design changes. The routine 800 may continue by executing (808) a test or pre-production instance of the updated program code. The test or pre-production instance 808 may include providing one or more stimuli 812 to one or more user devices 810 and receiving one or more inputs 814 in response to the one or more stimuli 812. According to an embodiment, the one or more stimuli may include one or more CSIs. The routine 800 may continue by processing (816) user activity data and communicating the user activity data to the design management subsystem. The routine 800 may continue in the design management subsystem by receiving and optionally storing (818) the user activity data. The routine 800 may continue by executing (820) a classification model (as configured in step 804) to classify (822) the user activity data.The routine 800 may conclude by processing the categorized user activity data and calculating 824 one or more performance metrics associated with the updated program code. According to an embodiment, the one or more performance metrics may include one or more quantified measures of at least one safety, efficacy, or performance variable associated with the updated program code.

[0071] Referring now to FIG. 9 , a functional block diagram of a routine 900 for determining the pass / fail status of a software build within a software design management system is shown. According to certain aspects of the present disclosure, the routine 900 may be incorporated within a software design management system such as those shown and described in any one of FIGS. 2-4 . The routine 900 may include one or more steps executed between a development subsystem and a design management subsystem, e.g., the development subsystems 202 and / or 302 and the design management subsystems 204 and / or 304 (as shown and described in FIGS. 2-3 , respectively). In certain embodiments, the routine 900 may include a continuation of one or more steps of the routine 800 (as shown and described in FIG. 8 ).

[0072] Routine 900 may include analyzing 902 data classification(s) corresponding to the execution of one or more data models on user activity data associated with an instance of a software program. According to an embodiment, step 902 may include one or more sub-steps 904-910 and / or 912-918. In an embodiment, sub-steps 904-910 may include evaluating a model fit to the classification model (on a per-feature basis) (904); evaluating one or more performance metrics for versions of the software program (on a per-feature basis) (906); quantifying the one or more performance metrics for versions of the software program (on a per-feature basis) (908); and determining at least one pass / fail status for versions of the software program (on a per-feature basis) (910). In one embodiment, sub-steps 912-918 may include the steps of evaluating a model fit for the classification model (for the current build / version) (912); evaluating one or more performance metrics for the version of the software program (for the current build / version) (914); quantifying the one or more performance metrics for the version of the software program (for the current build / version) (916); and determining at least one pass / fail status for the version of the software program (for the current build / version) (918).

[0073] Referring now to FIG. 10 , a functional block diagram of a routine 1000 for determining the impact of a design change within a software design management system is shown. According to certain aspects of the present disclosure, the routine 1000 may be incorporated within a software design management system such as those shown and described in any one of FIGS. 2-4 . The routine 1000 may include one or more steps executed between a development subsystem and a design management subsystem, e.g., the development subsystems 202 and / or 302 and the design management subsystems 204 and / or 304 (shown and described in FIGS. 2-3 , respectively). In certain embodiments, the routine 1000 may include a continuation of one or more steps of the routines 800 and / or 900 (shown and described in FIGS. 8-9 ).

[0074] Routine 1000 may include comparing 1002 at least one performance metric between a current version of the software program and at least one previous version of the software program. According to an embodiment, step 1002 may include one or more sub-steps 1004-1008 and / or 1010-1014. In an embodiment, sub-steps 1004-1008 may include comparing (on a per-feature basis) 1004 a model fit to a classification model between the current version of the software program and at least one previous version of the software program; comparing (on a per-feature basis) 1006 one or more performance metrics between the current version of the software program and at least one previous version of the software program; and determining (on a per-feature basis) 1008 one or more changes in performance between the current version of the software program and at least one previous version of the software program. In one embodiment, substeps 1010-1014 may include the steps of comparing (1010) a model fit to the classification model between the current version of the software program and at least one previous version of the software program; comparing (1012) one or more performance metrics between the current version of the software program and at least one previous version of the software program; and determining (1014) one or more changes in performance between the current version of the software program and at least one previous version of the software program. According to one embodiment, routine 1000 may conclude by determining (1016) qualitative and / or quantitative measures of the impact of new features and / or design changes on the software program, compared between the current version of the software program and one or more previous versions of the software program.

[0075] Referring now to FIG. 11 , a process flow diagram of a method 1100 for software design management and quality assurance is shown. According to an embodiment, the software program is configured as a DHI or SaMD product. According to certain aspects of the present disclosure, one or more steps of the method 1100 may be incorporated within and / or embodied by one or more systems, processes, and / or routines shown and described in FIGS. 1-10 . The method 1100 may include one or more steps executed between a development subsystem and a design management subsystem, e.g., the development subsystems 202 and / or 302 and the design management subsystems 204 and / or 304 (shown and described in FIGS. 2-3 , respectively). According to one embodiment, the method 1100 may include presenting 1102 a test instance of a software build including at least one test feature or design change to a user via a graphical user interface. The method 1100 may further include receiving 1104 user activity data from one or more users in response to presenting the test instance via the graphical user interface. The method 1100 may further include processing the user activity data according to at least one classification model to classify the user activity data (1106). The method 1100 may further include analyzing the user activity data to determine (1108) one or more performance metrics for the software build. In an embodiment, the method 1100 may further include analyzing (1112) the user activity data to determine performance metrics for one or more feature(s). The method 1100 may further include analyzing (1110) the performance metrics to determine a pass / fail status of the software build. In an embodiment, the method 1100 may further include analyzing (1114) the performance metrics to determine a pass / fail status of the one or more feature(s).

[0076] Referring now to FIG. 12 , a process flow diagram of a method 1200 for software design management and quality assurance for a software program is shown. According to an embodiment, the software program is configured as a DHI or SaMD product. According to certain aspects of the present disclosure, one or more steps of method 1200 may be incorporated within and / or embodied by one or more systems, processes, or routines shown and described in FIGS. 1-10 . Method 1200 may include one or more steps executed between a development subsystem and a design management subsystem, e.g., development subsystem 202 and / or 302 and design management subsystem 204 and / or 304 (shown and described in FIGS. 2-3 , respectively). According to certain embodiments, method 1200 may include a continuation of, or otherwise be incorporated within, one or more steps or sub-steps of method 1100 (shown and described in FIG. 11 ). According to certain aspects of the present disclosure, method 1200 may include one or more steps or operations for testing and / or software quality assurance for a DHI or SaMD product. Method 1200 may include one or more steps for presenting an instance of a software build to a user via a graphical user interface using a user computing device (step 1202). In an embodiment, the software build may include a test or production instance of a software product that includes at least one feature configured to elicit an expected stimulus input pattern from a user in response to presenting one or more computer-controlled stimuli or interactions in the graphical user interface. Method 1200 may proceed by performing or executing one or more steps for receiving a plurality of user inputs via an input sensor of the user computing device in response to presenting one or more computer-controlled stimuli or interactions within the instance of the software build (step 1204).The user activity data may be transmitted and received by a local or remote processor communicatively coupled to an input sensor of the user computing device (step 1206) and stored in at least one non-transitory computer-readable medium communicatively coupled to the local or remote processor. Method 1200 may proceed by using the local or remote processor to process the user activity data according to at least one data model to perform or accomplish one or more steps or substeps to determine one or more actual stimulus input patterns for each user input in the plurality of user inputs (step 1208). According to an embodiment, the at least one data model may include a classification model configured to classify one or more variables associated with one or more performance, safety, or efficacy parameters for the software product (e.g., a DHI or SaMD product). Method 1200 may proceed by performing one or more steps or substeps to compare the one or more actual stimulus input patterns for each user input in the plurality of user inputs with an expected stimulus input pattern for at least one feature to determine a total number of instances in which the one or more actual stimulus input patterns reflected the expected stimulus input pattern within the software build session (step 1210). According to an embodiment, method 1200 may include one or more steps for calculating an amount of net therapy activity within a session of the software build according to a total number of instances in which one or more actual stimulus input patterns reflected the expected stimulus input patterns. The one or more steps may further include one or more substeps for calculating an amount of effective therapy delivery for at least one feature within the session of the software build according to a total number of instances in which one or more actual stimulus input patterns reflected the expected stimulus input patterns. Method 1200 may proceed by performing or executing, with a local or remote processor, one or more of the steps or substeps for calculating at least one output for user activity data according to at least one data model (step 1212).According to a typical embodiment, the at least one output value is configured to provide a qualitative or quantitative measure of the model fit for the at least one feature. Method 1200 may proceed by performing one or more steps or substeps for determining a pass / fail status for the software build according to the at least one output value (step 1214) and / or determining a pass / fail status for the at least one feature according to the at least one output value (step 1216). In an embodiment, method 1200 may include one or more steps or substeps for comparing the at least one output value with at least one previous output value associated with at least one previous version of the software build to determine an amount of change attributable to the at least one feature.

[0077] In an embodiment in which one or more amounts of therapy activity / delivery are measured within a session of the software build, method 1200 may also include one or more steps for comparing the amount of net therapy activity within the session of the software build to at least one previous amount of net therapy activity associated with at least one previous session of the software build to determine an amount of change attributable to at least one feature. In such an embodiment, a pass / fail status for the software build may be determined according to the amount of net therapy activity within the session of the software build and / or the amount of effective therapy delivery for at least one feature within the session of the software build.

[0078] Referring now to FIG. 13 , a process flow diagram of a method 1300 for software design management and quality assurance for a software program is shown. According to an embodiment, the software program is configured as a DHI or SaMD product. According to certain aspects of the present disclosure, method 1300 may be incorporated within a software design management system such as those shown and described in any one of FIGS. 2-4 . Method 1300 may include one or more steps executed between a development subsystem and a design management subsystem, e.g., development subsystem 202 and / or 302 and design management subsystem 204 and / or 304 (shown and described in FIGS. 2-3 , respectively). In an embodiment, method 1300 may include a continuation of, or otherwise be incorporated within, one or more steps or sub-steps of method 1100 (shown and described in FIG. 11 ) and / or method 1200 (shown and described in FIG. 12 ). According to one embodiment, method 1300 may include configuring 1302 a digital intervention to effectively treat or target a neurological, psychological, or physical condition of a user. Method 1300 may further include configuring 1304 a classification model according to clinically validated stimulus-response patterns. Method 1300 may further include presenting 1306 an instance of the digital intervention to the user. Method 1300 may further include collecting 1308 stimulus input data in response to the instance of the digital intervention. Method 1300 may further include processing 1310 the stimulus input data according to the classification model. Method 1300 may further include determining 1312 a model appropriate for the stimulus input data in response to processing the stimulus input data according to the classification model to generate one or more performance metrics. According to an embodiment, the one or more performance metrics may include one or more quantified measures of safety, efficacy, and / or performance for the digital intervention. The method 1300 may further include analyzing 1314 performance metrics for one or more characteristics of the digital intervention.The method 1300 may further include determining and / or quantifying 1316 a degree of therapeutic treatment for a neurological, psychological, or physical condition associated with each of the one or more features of the digital intervention and / or the digital intervention as a whole. The method 1300 may further include determining 1318 a pass / fail status for each of the one or more features of the digital intervention and / or the digital intervention as a whole.

[0079] As will be appreciated by those skilled in the art, the present invention may be embodied as a method (including, for example, a computer-implemented process, a business process, and / or any other process), an apparatus (including, for example, a system, a machine, a device, a computer program product, and / or the like), or a combination of the foregoing. Accordingly, embodiments of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, which may be generally referred to herein as a "system." Moreover, embodiments of the present invention may take the form of a computer program product on a computer-executable medium having computer-executable program code embodied in the medium.

[0080] Any suitable transitory or non-transitory computer-readable medium may be utilized. The computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples of computer-readable media include, but are not limited to, the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a tangible storage medium such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a read-only compact disc (CD-ROM), or other optical or magnetic storage device.

[0081] In the context of this document, a computer-readable medium may be any medium that can contain, store, transmit, or carry a program for use by or in connection with an instruction execution system, apparatus, or device. The computer-usable program code may be transmitted using any suitable medium, including but not limited to the Internet, wire, fiber optic cable, radio frequency (RF) signal, or other medium.

[0082] Computer-executable program code for carrying out operations of embodiments of the present invention may be written in an object-oriented, scripting or non-scripting programming language, such as Java, Perl, Smalltalk, C++, or the like. However, computer program code for carrying out operations of embodiments of the present invention may also be written in conventional procedural programming languages, such as the "C" programming language or a similar programming language.

[0083] Embodiments of the present invention are described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products. It will be understood that each block of the flowchart illustrations and / or block diagrams, and / or combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable program code portions that are provided to a general-purpose computer processor, a special-purpose computer, or other programmable data processing apparatus to create specific machines, whereby the code portions, executing via the computer processor or other programmable data processing apparatus, create mechanisms for implementing the function(s) / act(s) specified in the flowchart and / or block diagram block or blocks.

[0084] These computer-executable program code portions (i.e., computer-executable instructions), which can direct a computer or other programmable data processing apparatus to function in a particular manner, may also be stored in a computer-readable memory, such that the code portions stored in the computer-readable memory create an article of manufacture including an instruction mechanism that implements the functions / acts specified in the flowchart and / or block diagram block(s). Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0085] The computer-executable program code may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational phases to be executed on the computer or other programmable apparatus to generate a computer-implemented process such that the code portions executing on the computer or other programmable apparatus provide phases for implementing the functions / acts specified in the flowchart and / or block diagram block(s). Alternatively, the computer-program-implemented phases or acts may be combined with operator or human-implemented phases or acts to perform one embodiment of the present invention.

[0086] As the phrase is used herein, a processor may be "operable" or "configured" to perform a function in various ways, including, for example, by causing one or more general-purpose circuits to perform the function by executing specific computer-executable program code embodied in a computer-readable medium, and / or by causing one or more application-specific circuits to perform the function.

[0087] The terms "program" or "software" are used generically herein to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of the present technology as described above. Additionally, in accordance with one aspect of the present embodiments, it will be appreciated that one or more computer programs that, when executed, perform the methods of the present technology need not reside on a single computer or processor, but may be distributed in a modular manner among several different computers or processors to implement various aspects of the present technology.

[0088] All definitions should be understood to be governed by dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms as defined and used herein.

[0089] The indefinite articles "a" and "an," as used in the specification and claims, unless expressly indicated to the contrary, should be understood to mean "at least one." As used herein, the terms "right," "left," "upper," "lower," "upper," "lower," "inner," and "outer" designate directions in the drawings to which reference is made.

[0090] The phrase "and / or," as used in the specification and claims, should be understood to mean "either or both" of the elements so conjoined, i.e., elements presented conjoined in some instances and presented separately in other instances. Multiple elements listed with "and / or" should be construed in the same manner, i.e., "one or more" of the elements so conjoined. Other elements, whether related or unrelated to those elements specifically identified, may optionally refer to elements other than those specifically identified by the "and / or" clause. Thus, as a non-limiting example, a reference to "A and / or B," when used with open-ended language such as "comprising," may refer, in one embodiment, to A only (optionally including elements other than B); in another embodiment, to B only (optionally including elements other than A); or in yet another embodiment, to both A and B (optionally including other elements).

[0091] As used herein and in the claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" should be interpreted as inclusive, i.e., including at least one, but also including two or more, and optionally additional unlisted items, of a number of elements or list of elements. Only one item expressly indicated to the contrary, e.g., "only one of" or "exactly one of," or when used in the claims, "consisting of" refers to the inclusion of exactly one element of a number of elements or list of elements. In general, the term "or" as used herein shall only be interpreted as indicating exclusive alternatives (i.e., "one or the other, but not both") when accompanied by terms of exclusivity, such as "either," "one of," "only one of," or "exactly one of." "Consisting essentially of," when used in the claims, shall have its ordinary meaning as used in the field of patent law.

[0092] As used in the specification and claims, the phrase "at least one," in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed in the list of elements, and not excluding any combinations of elements in the list of elements. This definition also allows for elements, optionally, to be present other than those specifically identified in the list of elements to which the phrase "at least one" refers, whether related or unrelated to those specifically identified elements. Thus, as a non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B" or, equivalently, "at least one of A and / or B") can refer to, in one embodiment, at least one, optionally more than one, A, with no B present (and optionally including elements other than B); in another embodiment, at least one, optionally more than one, B, with no A present (and optionally including elements other than A); in yet another embodiment, at least one, optionally more than one, A, and at least one, optionally more than one, B (and optionally including other elements); etc.

[0093] In the claims, and in the foregoing specification, all transitional phrases, such as "comprising," "including," "carrying," "having," "containing," "involving," "holding," "composed of," and the like, are to be understood as open-ended, i.e., meaning inclusive but not limited to. Only the transitional phrases "consisting of" and "consisting essentially of" shall be closed or semi-closed transitional phrases, respectively, as set forth in the United States Patent Office Manual of Examining Procedure, Section 2111.03.

[0094] The present disclosure includes not only the foregoing description but also what is contained in the appended claims. While the present invention has been described in its exemplary form, with a certain degree of particularity, it will be understood that the disclosure is made by way of example only, and that many changes in details of construction and in the combination and arrangement of parts may be adopted without departing from the spirit and scope of the invention.

Claims

1. presenting an instance of a software build to a user via a graphical user interface using a user computing device, said software build including at least one feature including one or more computer-controlled stimuli or interactions configured to elicit an expected stimulus input pattern from said user in response to said one or more computer-controlled stimuli or interactions; receiving, with at least one sensor communicatively coupled to the user computing device, a plurality of user inputs in response to presentation of the one or more computer-controlled stimuli or interactions within the instance of the software build, the plurality of user inputs comprising user activity data for a session of the software build; receiving, with a processor communicatively coupled to the user computing device, the user activity data; using the processor to process the user activity data according to at least one data model to determine one or more actual stimulus input patterns for each user input in the plurality of user inputs; comparing the one or more actual stimulus input patterns for each user input in the plurality of user inputs to the expected stimulus input pattern for the at least one feature to determine a total number of instances in which the one or more actual stimulus input patterns reflected the expected stimulus input pattern within the session of the software build; calculating, with the processor, at least one output value for the user activity data in accordance with the at least one data model, the at least one output value including a qualitative or quantitative measure of model fit for the at least one characteristic; determining a pass / fail status for the software build according to the at least one output value; Methods for software quality assurance, including:

2. 2. The method of claim 1, further comprising: using the processor to calculate an amount of net therapeutic activity within the session of the software build according to the total number of instances in which the one or more actual stimulation input patterns reflected the expected stimulation input patterns.

3. 3. The method of claim 2, further comprising: using the processor to calculate an amount of effective therapy delivery for the at least one feature within the session of the software build according to the total number of instances in which the one or more actual stimulation input patterns reflected the expected stimulation input patterns.

4. 10. The method of claim 1, wherein the at least one data model comprises a classification model configured to classify one or more variables associated with one or more performance, safety, or efficacy parameters.

5. The method of claim 1 , further comprising: using the processor to determine a pass / fail status for the at least one feature according to the at least one output value.

6. 10. The method of claim 1, further comprising: using the processor to compare the at least one output value to at least one previous output value associated with at least one previous version of the software build to determine an amount of change attributable to the at least one characteristic.

7. 4. The method of claim 3, further comprising: using the processor to compare the amount of net therapy activity within the session of the software build to at least one previous amount of net therapy activity associated with at least one previous version of the software build to determine an amount of change attributable to the at least one characteristic.

8. The method of claim 2 , further comprising: using the processor to determine the pass / fail status for the software build according to the amount of net therapy activity within the session of the software build.

9. 4. The method of claim 3, further comprising: using the processor to determine the pass / fail status for the software build according to the amount of effective therapy delivery for the at least one feature within the session of the software build.

10. a processor; a non-transitory computer-readable storage medium communicatively coupled to the processor and encoded with processor-executable instructions that, when executed, cause the processor to: presenting a graphical user interface including an instance of a software build, the software build including at least one feature including one or more computer-controlled stimuli or interactions configured to elicit an expected stimulus input pattern from a user in response to the one or more computer-controlled stimuli or interactions; receiving a plurality of user activity data for the software build session, the plurality of user activity data including a plurality of user inputs in response to the one or more computer-controlled stimuli or interactions; processing the plurality of user activity data according to at least one data model to determine one or more actual stimulus input patterns for each user input in the plurality of user inputs; comparing the one or more actual stimulus input patterns for each user input in the plurality of user inputs to the expected stimulus input pattern for the at least one feature to determine a total number of instances in which the one or more actual stimulus input patterns reflected the expected stimulus input pattern within the session of the software build; calculating at least one output value for the plurality of user activity data in accordance with the at least one data model, the at least one output value comprising a qualitative or quantitative measure of a model fit for the at least one feature; determining a pass / fail status for the software build according to the at least one output value; a non-transitory computer-readable storage medium for causing the computer to perform one or more operations including A system for software quality assurance comprising:

11. 11. The system of claim 10, wherein the one or more operations further include calculating an amount of net therapeutic activity within the session of the software build according to the total number of instances in which the one or more actual stimulation input patterns reflected the expected stimulation input patterns.

12. 12. The system of claim 11, wherein the one or more operations further include calculating an amount of effective therapy delivery for the at least one feature within the session of the software build according to the total number of instances in which the one or more actual stimulation input patterns reflected the expected stimulation input patterns.

13. 11. The system of claim 10, wherein the at least one data model comprises a classification model configured to classify one or more variables associated with one or more performance, safety, or efficacy parameters.

14. The system of claim 10 , wherein the one or more actions further comprise determining a pass / fail status for the at least one feature according to the at least one output value.

15. 11. The system of claim 10, wherein the one or more actions further comprise comparing the at least one output value to at least one previous output value associated with at least one previous version of the software build to determine an amount of change attributable to the at least one characteristic.

16. 13. The system of claim 12, wherein the one or more operations further include comparing the amount of net therapy activity within the session of the software build to at least one previous amount of net therapy activity associated with at least one previous version of the software build to determine an amount of change attributable to the at least one feature.

17. The system of claim 11 , wherein the one or more actions further comprise determining the pass / fail status for the software build according to the amount of net therapy activity within the session of the software build.

18. 13. The system of claim 12, wherein the one or more actions further comprise determining the pass / fail status for the software build according to the amount of effective therapy delivery for the at least one feature within the session of the software build.

19. 11. The system of claim 10, wherein the one or more actions further comprise determining the pass / fail status for the software build according to at least one safety parameter associated with the one or more computer-controlled stimuli or interactions.

20. 1. A non-transitory computer-readable storage medium encoded with instructions for directing one or more processors to perform operations for software quality assurance, the operations comprising: presenting a graphical user interface including an instance of a software build, the software build including at least one feature including one or more computer-controlled stimuli or interactions configured to elicit an expected stimulus input pattern from a user in response to the one or more computer-controlled stimuli or interactions; receiving a plurality of user activity data for the software build session, the plurality of user activity data including a plurality of user inputs in response to the one or more computer-controlled stimuli or interactions; processing the plurality of user activity data according to at least one data model to determine one or more actual stimulus input patterns for each user input in the plurality of user inputs; comparing the one or more actual stimulus input patterns for each user input in the plurality of user inputs to the expected stimulus input pattern for the at least one feature to determine a total number of instances in which the one or more actual stimulus input patterns reflected the expected stimulus input pattern within the session of the software build; calculating at least one output value for the plurality of user activity data in accordance with the at least one data model, the at least one output value comprising a qualitative or quantitative measure of a model fit for the at least one feature; determining a pass / fail status for the software build according to the at least one output value; Including, A non-transitory computer-readable storage medium.

21. 21. The non-transitory computer-readable storage medium of claim 20, wherein the operations further include calculating an amount of net therapeutic activity within the session of the software build according to the total number of instances in which the one or more actual stimulation input patterns reflected the expected stimulation input patterns.

22. 22. The non-transitory computer-readable storage medium of claim 21, wherein the operations further include calculating an amount of effective therapy delivery for the at least one feature within the session of the software build according to the total number of instances in which the one or more actual stimulation input patterns reflected the expected stimulation input patterns.

23. 21. The non-transitory computer-readable storage medium of claim 20, wherein the at least one data model comprises a classification model configured to classify one or more variables associated with one or more performance, safety, or efficacy parameters.

24. 21. The non-transitory computer-readable storage medium of claim 20, wherein the operations further comprise determining a pass / fail status for the at least one characteristic according to the at least one output value.

25. 21. The non-transitory computer-readable storage medium of claim 20, wherein the operations further include comparing the at least one output value to at least one previous output value associated with at least one previous version of the software build to determine an amount of change attributable to the at least one characteristic.

26. 23. The non-transitory computer-readable storage medium of claim 22, wherein the operations further include comparing the amount of net therapy activity within the session of the software build to at least one previous amount of net therapy activity associated with at least one previous version of the software build to determine an amount of change attributable to the at least one feature.

27. 22. The non-transitory computer-readable storage medium of claim 21, wherein the operations further comprise determining the pass / fail status for the software build according to the amount of net therapy activity within the session of the software build.

28. 23. The non-transitory computer-readable storage medium of claim 22, wherein the operations further comprise determining the pass / fail status for the software build according to the amount of effective therapy delivery for the at least one feature within the session of the software build.

29. 21. The non-transitory computer-readable storage medium of claim 20, wherein the one or more actions further comprise determining the pass / fail status for the software build according to at least one safety parameter associated with the one or more computer-controlled stimuli or interactions.

30. A non-transitory computer-readable storage medium encoded with instructions for directing one or more processors to perform operations to implement the method of any one of claims 1 to 9.

31. 1. A system for performing software quality assurance, comprising: a processor; a non-transitory computer-readable storage medium communicatively coupled to the processor and encoded with processor-executable instructions that, when executed, cause the processor to perform one or more operations for implementing a method according to any one of claims 1 to 9; A system comprising:

32. 1. An apparatus for performing software quality assurance, comprising: a processor; a non-transitory computer-readable storage medium communicatively coupled to the processor and encoded with processor-executable instructions that, when executed, cause the processor to perform one or more operations for implementing a method according to any one of claims 1 to 9; An apparatus comprising:

33. 10. The method of claim 1, further comprising determining the pass / fail status for the software build according to at least one safety parameter associated with the one or more computer-controlled stimuli or interactions.

34. 4. The method of claim 1, wherein the at least one data model comprises a classification model configured to classify one or more variables associated with one or more performance, safety, or efficacy parameters.

35. The method of any one of claims 1 to 4, further comprising using the processor to determine a pass / fail status for the at least one feature according to the at least one output value.

36. 6. The method of claim 1, further comprising: using the processor to compare the at least one output value to at least one previous output value associated with at least one previous version of the software build to determine an amount of change attributable to the at least one characteristic.

37. 10. The method of claim 1, further comprising: using the processor to compare an amount of net therapy activity within the session of the software build to at least one previous amount of net therapy activity associated with at least one previous version of the software build to determine an amount of change attributable to the at least one feature.

38. The method of claim 1 , further comprising: using the processor to determine the pass / fail status for the software build according to an amount of net therapy activity within the session of the software build.

39. 10. The method of claim 1, further comprising: using the processor to determine the pass / fail status for the software build according to an amount of effective therapy delivery for the at least one feature within the session of the software build.

40. A non-transitory computer readable storage medium encoded with instructions for directing one or more processors to perform operations to implement the method of any one of claims 33 to 39.

41. 1. A system for performing software quality assurance, comprising: a processor; a non-transitory computer-readable storage medium communicatively coupled to the processor and encoded with processor-executable instructions that, when executed, cause the processor to perform one or more operations to implement a method according to any one of claims 33 to 39; and A system comprising:

42. 1. An apparatus for performing software quality assurance, comprising: a processor; a non-transitory computer-readable storage medium communicatively coupled to the processor and encoded with processor-executable instructions that, when executed, cause the processor to perform one or more operations to implement a method according to any one of claims 33 to 39; and An apparatus comprising:

Images