Systems and methods for elastic bot cluster deployments in continuous model testing

Abstract

Described are platforms, methods, systems, and media configured for testing a model, utilizing operations comprising: identifying a plurality of AI-driven bots, wherein each bot comprises a persona; generating a bot priority score for each bot; determining a number of testing phases; computing the number of bots deployed in each testing phase; determining a resource scaling approach; generating an intelligent scheduling score for each bot; and deploying the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores.

Description

SYSTEMS AND METHODS FOR ELASTIC BOT CLUSTER DEPLOYMENTS IN CONTINUOUS MODEL TESTINGCROSS-REFERENCE

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 507,908, filed June 13, 2023, which application is incorporated herein by reference in its entirety and to which application we claim priority under 35 U.S.C. § 120.BACKGROUND

[0002] Generative Artificial Intelligence (Al) has brought change to every business vertical. Generative Al is artificial intelligence that can create new content, such as text, images, or music, by learning patterns and structures from existing data. It works by analyzing vast amounts of information and then generating output that resembles the original data while maintaining originality.SUMMARY

[0003] For all the viral adoption, excitement, and rapid integration, Generative Al, including Large Language Model (LLM) technology is no panacea. LLMs are indeterminate. From a testing point of view, “indeterminacy” in software means that the system’s behavior or output can vary unpredictably, even when given the same input. As a result, determining if the system works correctly is impossible, as the expected outcome is not always consistent. Furthermore, any software product integrating with LLM technology will manifest indeterminate characteristics. As a result, it is impossible to thoroughly test Generative Al employing traditional software techniques and Generative Al is not production ready for critical client applications.

[0004] The output from a LLM cannot be blindly trusted. These models can generate incorrect or nonsensical answers, struggle with ambiguity, and may only sometimes fully comprehend context or exhibit nuanced understanding. Model certainty and uncertainty are, by design, reflected in the natural language output from the model. Such models may not be accurate and could be flat-out wrong. Importantly, current LLM technology will just as confidently and reasonably serve incorrect answers as it will correct ones.

[0005] With these fact in mind, testing of generative Al becomes even more important. In traditional software development, a system is an assembly of functions. Each function is similar to building blocks. Blocks are placed together and create a tower. Any given function has a finite set of acceptable inputs. Input constraints may be at the data type level, for example, only text orwhole numbers may be allowed. Constraints may be at the semantic layer. Only known customer identification numbers or first and last names are permitted. The constraints may become more complex when assembling multiple functions, but they remain finite. Reaching a state known as “complete test coverage” is almost always possible.

[0006] Complete test coverage implies the tester has created enough test cases to test every possible input variation the system can accept. The test suite may also contain a handful of known “bad inputs” to ensure the system can gracefully handle inputs that do not meet internal constraints. Complete code coverage is a critical state to achieve. Achieving this state assures the system is ready to be promoted into production. Every known variation of the system is testable, and a tester can check responses for adequacy.

[0007] Likewise, where function inputs are finite, function outputs tend to be finite. The definition of “finite” should be carefully considered in this case. For example, a function that combines numbers mathematically could take an infinite range of numbers as input and return an endless range of numbers as output. However, finite input means a given parameter’s positive and explicit declaration to the function. The parameter is often explicitly typed (e.g., the input must be text, or the input must be a decimal with three levels of precision, etc.). The output will have similar explicit typing.

[0008] Given the case of a mathematical function, the declaration may look something like this: def do math (x : integer, y : integer ) -> integer : / / perform some math function return <some integer value>

[0009] In the pseudo-code written above, although the values of the function may be infinite, the parameters are finite, explicit, and declared. The consumer must pass in two integer values to successfully activate this function. The procedure will only return an integer value in return. Finite, as used herein, means explicit declaration of input and output. Even the function name, while not posing an explicit constraint, gives implicit boundaries regarding the nature of the embodied logic.

[0010] By contrast, generative Al presents a function like this to the world: def respond ( input : any) -> any : / / respond to any input return <response>

[0011] Any input of any type is acceptable for generative Al. Grant that there is no understandable limit to input and output. Input and output are unconstrained. Removing such constraints may have seemed like a large given just a year ago, but today we must accept this, and its profound effect on testing.

[0012] Typically, any function integrating into a system poses a finite number of inputs and outputs. Constraints are declared. Such integration may increase the complexity of the overall design, but at a macro level, we merely add finite conditions to finite rules and end up with finite constraints. Today, when a system integrates with generative Al, such an integration adds infinite constraints. Such a term, “infinite constraints,” is an oxymoron. The integration is simply unconstrained. Even a single integration touchpoint to generative Al results in an unconstrained system. As mentioned herein, techniques exist in the industry for the testing of constrained systems.

[0013] However, techniques do not exist in the industry for testing unconstrained systems. Achieving “complete code coverage” is not possible because it is impossible to test generative Al with every known input. Nor is it possible to try each output given an input. Making the matter even more complex, generative Al models are not predictable. It is possible to ask the same question ten times and get a different response each time. Even more complicated is when asking the same question ten times, the same answer returns nine times, but on the tenth time, the response may change.

[0014] In traditional software, a tester may spend a finite duration on a test task. For example, testing the functional pseudo code expressed above (“do math”) may hypothetically require one hour of test time. However, after creating the test, it can be automatically run each time the code changes. From the perspective of a tester, this is a well-invested hour. Furthermore, the test case remains valid as long as the function remains stable regarding declared inputs and outputs. After spending multiple finite durations on every functional aspect of the system, the tester will have achieved complete code coverage. Once the system provides adequate responses to these defined test cases, production is achievable.

[0015] There is no upper limit on how much time will a tester spend on a single generative Al function. To truly understand and embrace the near-infinite nature of LLMs, the tester must spend a near-infinite amount of time testing them. Infinite human testing is, of course, impossible. Achieving complete coverage becomes impossible when any generative Altouchpoint occurs during integration. Asserting confidently when a system is ready for production is no longer possible.

[0016] A person responsible for the quality of a software product with generative Al integration, may have trust in the software only as great as that in the generative Al. And that trust must extend to the audience. For example, will the audience attempt to trick it, trip it up, and find ways to break it down? And, having broken it down, will they take screenshots and post them on social media?

[0017] Under traditional software development methodologies, trust is no longer achievable in the age of LLMs. We require a new methodology of testing to achieve trust. As described herein, this may come in the form of a “Bot Army.” Bots are autonomous software programs that can interact with systems, users, or devices.

[0018] In some embodiments, the “Bot Army” described herein comprises a diverse and extensive collection of Al-driven bots organized into functionality clusters, including, by way of non-limiting examples, personality clusters, red teams, sentiment clusters, domain knowledge clusters, and audience segment clusters. In further embodiments, the clusters aim to evaluate and improve the performance of Al systems by testing their ability to generate accurate, relevant, and appropriate responses across a wide range of user types and conversational scenarios. By exploring the full spectrum of user behaviors and characteristics, the Bot Army helps Al developers identify and address potential weaknesses, biases, or vulnerabilities in their systems, ultimately leading to a more robust, reliable, and user-friendly Al experience.

[0019] Described herein are considerations for efficiently deploying a “Bot Army,” maximizing valuable test data, and minimizing associated costs. In various embodiments, these considerations include, by way of examples, at least the following:

[0020] Resource Segmentation on Bot Priority: Not all bots are necessary for every testing scenario. Identify critical segments of the Bot Army that are most relevant for the particular aspects of the application under test. Segmentation occurs on functionality tested and the type of user interaction expected.

[0021] Phased Deployment: Rather than deploying all selected bots simultaneously, consider a phased approach. Start with a smaller subset of bots, monitor the system performance and the quality of data received, then gradually increase the number of active bots. This incremental approach can prevent overwhelming the system and provide time to adjust configurations based on initial feedback.

[0022] Resource Scaling: Utilize the capabilities of cloud infrastructure for dynamic scaling. During periods of high bot activity, scale up resources to maintain performance. During idle times, scale down to conserve resources.

[0023] Intelligent Scheduling: To avoid peak times and align with the application's expected usage patterns, schedule hot activities intelligently. Scheduling not only spreads out the load on the system but also helps simulate real-world usage scenarios more accurately.

[0024] Effective Monitoring: Monitor the system closely during testing. Monitoring includes tracking the application's performance, hot activities, and resource usage. Use this data to make real-time adjustments to the deployment strategy as needed.

[0025] API Rate Limiting and Caching: Implement rate limiting to prevent overloading third- party APIs and incurring high costs. Also, consider caching responses wherever possible to reduce the number of calls to these APIs.

[0026] Load Testing: Before deploying the whole Bot Army, perform load testing to understand the system's behavior under stress. Load testing helps to anticipate potential issues and make necessary optimizations beforehand. There is no one-size-fits-all approach here. Some applications may support high loads easily; others will face relatively small load caps.

[0027] Usage Optimization: For costly functions, like calls to third-party APIs, it might be helpful to implement a predictive model to anticipate the needs of the bots. This way, calls can be batched together, reducing the number of calls and saving costs.

[0028] Incorporation of these considerations, when designing an approach to deployment of a testing scheme, ensures a balance between achieving comprehensive application testing and managing the costs and resources of maintaining the Bot Army and the supporting infrastructure.

[0029] Accordingly, in one aspect, disclosed herein are computer-implemented methods of testing a model comprising: identifying a plurality of Al-driven bots, wherein each bot comprises a persona; generating a bot priority score for each bot; determining a number of testing phases; computing the number of bots deployed in each testing phase; determining a resource scaling approach; generating an intelligent scheduling score for each bot; and deploying the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores. In some embodiments, the model comprises a large natural language conversational model. In some embodiments, the plurality of bots are identified from a library of pre-configured bots or bot personas. In some embodiments, the method further comprises organizing the plurality of AI-driven bots into one or more functionality clusters. In various further embodiments, the one or more functionality clusters comprise: one or more personality clusters, one or more red team clusters, one or more sentiment clusters, one or more domain knowledge clusters, one or more audience segment clusters, or any combination thereof. In some embodiments, the method further comprises generating a bot priority score for each cluster. In some embodiments, the method further comprises generating an intelligent scheduling score for each cluster. In various embodiments, the testing is continuous for at least one hour, at least 1 day, at least 1 week, at least 1 month, or at least 1 year. In various embodiments, the plurality of bots comprises at least 10, at least 100, at least 1000, at least 10,000, or at least 100,000 bots. In some embodiments, at least a subset of the bots are configured to interact with the model. In further embodiments, the plurality of bots interact with the model autonomously.

[0030] In some embodiments, the bot priority scores are determined, at least in part, by:wherein:FI is the functionality’s importance to the application (Likert scale of 1 - 7, with seven being most relevant and three being neutral);BR is the relevance a specific bot to the user base (Likert scale of 1 - 7, with seven being most relevant and three being neutral);IF is how frequently a test bot will interact with the specific functionality;El is the impact of an error or failure in a specific functionality (Likert scale of 1 - 7 can also be used here, with 7 indicating a critical failure that would severely impact the user experience); andN is a normalization factor.

[0031] In further embodiments, the bot priority is determined, at least in part, by:wherein: a is a hyperfactor for FI, adjusted to weight FI according to perceived importance;P is a hyperfactor for BR, adjusted to weight BR according to perceived importance; y is a hyperfactor for IF, adjusted to weight IF according to perceived importance; and5 is a hyperfactor for El, adjusted to weight El according to perceived importance.

[0032] In some embodiments, the number of testing phases is determined, at least in part, by:wherein: ceil is the ceiling function;B is total number of bots;SC is the system capacity to handle simultaneous bot interactions;RF is a risk factor between 0 and 1 representing the risk tolerance; andTP is a tuning parameter optionally adjusted based on factors like test coverage, system stability, and testing objectives.

[0033] In some embodiments, the number of bots deployed in each testing phase is determined, at least in part, by:wherein:TDB is total deployed bots with a subscript denoting the phase number (K);BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

[0034] In some embodiments, the resource scaling approach is determined, at least in part, by: BP ■ CL)RS -r CF wherein:RS subscripts denote the phase number (K) and resource type (T), respectively;BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

[0035] In some embodiments, the scheduling scores are determined, at least in part, by:wherein:BP is the bot priority based on functionality and user relevance;ARL (API Rate Limit) is the maximum number of requests the bot can make to a specific API within a certain timeframe;BCT (Bot Cycle Time) is the time the bot takes to complete one cycle of its operation;CET (Cache Expiry Time) is the duration for which data fetched by the bot from the API is valid; and a, P, y, 5 are hyperparameters adjusted to weight the factors according to relative importance.

[0036] In some embodiments, the method further comprises calculating a resource utilization numerator (RUN) score, at least in part, by:RUN = 7](RSS x PDS x ISS) wherein: q is a normalization factor to scale the score to a suitable range (e.g., 0 to 1);RSS is a resource segmentation score;PDS is a phased deployment score; and ISS is an intelligent scheduling score.

[0037] In another aspect, disclosed herein are computer-implemented systems for testing a model comprising a computing device comprising at least one processor and instructions executable by the at least one processor to cause the at least one processor to perform operations comprising: identifying a plurality of Al-driven bots, wherein each bot comprises a persona; generating a bot priority score for each bot; determining a number of testing phases; computing the number of bots deployed in each testing phase; determining a resource scaling approach; generating an intelligent scheduling score for each bot; and deploying the plurality of bots, overa network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores. In some embodiments, the model comprises a large natural language conversational model. In some embodiments, the plurality of bots are identified from a library of pre-configured bots or bot personas. In some embodiments, the operations further comprise organizing the plurality of Al-driven bots into one or more functionality clusters. In various further embodiments, the one or more functionality clusters comprise: one or more personality clusters, one or more red team clusters, one or more sentiment clusters, one or more domain knowledge clusters, one or more audience segment clusters, or any combination thereof. In some embodiments, the operations further comprise generating a bot priority score for each cluster. In some embodiments, the operations further comprise generating an intelligent scheduling score for each cluster. In various embodiments, the testing is continuous for at least one hour, at least 1 day, at least 1 week, at least 1 month, or at least 1 year. In various embodiments, the plurality of bots comprises at least 10, at least 100, at least 1000, at least 10,000, or at least 100,000 bots. In some embodiments, at least a subset of the bots are configured to interact with the model. In further embodiments, the plurality of bots interact with the model autonomously.

[0038] In some embodiments, the bot priority scores are determined, at least in part, by:(FI X BR X IF X BI)BP ™ wherein:FI is the functionality’s importance to the application (Likert scale of 1 - 7, with seven being most relevant and three being neutral);BR is the relevance a specific bot to the user base (Likert scale of 1 - 7, with seven being most relevant and three being neutral);IF is how frequently a test bot will interact with the specific functionality;El is the impact of an error or failure in a specific functionality (Likert scale of 1 - 7 can also be used here, with 7 indicating a critical failure that would severely impact the user experience); andN is a normalization factor.

[0039] In further embodiments, the bot priority is determined, at least in part, by:wherein: a is a hyperfactor for FI, adjusted to weight FI according to perceived importance;P is a hyperfactor for BR, adjusted to weight BR according to perceived importance; y is a hyperfactor for IF, adjusted to weight IF according to perceived importance; and5 is a hyperfactor for El, adjusted to weight El according to perceived importance.

[0040] In some embodiments, the number of testing phases is determined, at least in part, by:wherein: ceil is the ceiling function;B is total number of bots;SC is the system capacity to handle simultaneous bot interactions;RF is a risk factor between 0 and 1 representing the risk tolerance; andTP is a tuning parameter optionally adjusted based on factors like test coverage, system stability, and testing objectives.

[0041] In some embodiments, the number of bots deployed in each testing phase is determined, at least in part, by:wherein:TDB is total deployed bots with a subscript denoting the phase number (K);BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

[0042] In some embodiments, the resource scaling approach is determined, at least in part, by:wherein:RS subscripts denote the phase number (K) and resource type (T), respectively;BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

[0043] In some embodiments, the scheduling scores are determined, at least in part, by:wherein:BP is the bot priority based on functionality and user relevance;ARL (API Rate Limit) is the maximum number of requests the bot can make to a specific API within a certain timeframe;BCT (Bot Cycle Time) is the time the bot takes to complete one cycle of its operation;CET (Cache Expiry Time) is the duration for which data fetched by the bot from the API is valid; and a, P, y, 5 are hyperparameters adjusted to weight the factors according to relative importance.

[0044] In some embodiments, the operations further comprise calculating a resource utilization numerator (RUN) score, at least in part, by:RUN = 7 (RSS x PDS x ISS) wherein: q is a normalization factor to scale the score to a suitable range (e.g., 0 to 1);RSS is a resource segmentation score;PDS is a phased deployment score; andISS is an intelligent scheduling score.

[0045] In yet another aspect, disclosed herein are non-transitory computer-readable storage media encoded with instructions executable by one or more processors to provide a modeltesting application comprising: a module configured to identify (or identifying) a plurality of AI- driven bots, wherein each hot comprises a persona; a module configured to generate (or generating) a hot priority score for each hot; a module configured to determine (or determining) a number of testing phases; a module configured to compute (or computing) the number of bots deployed in each testing phase; a module configured to determine (or determining) a resource scaling approach; a module configured to generate (or generating) an intelligent scheduling score for each bot; and a module configured to deploy (or deploying) the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores. In some embodiments, the model comprises a large natural language conversational model. In some embodiments, the plurality of bots are identified from a library of pre-configured bots or bot personas. In some embodiments, the application further comprises a module configured to organize (or organizing) the plurality of Al-driven bots into one or more functionality clusters. In various further embodiments, the one or more functionality clusters comprise: one or more personality clusters, one or more red team clusters, one or more sentiment clusters, one or more domain knowledge clusters, one or more audience segment clusters, or any combination thereof. In some embodiments, the application further comprises a module configured to generate (or generating) a bot priority score for each cluster. In some embodiments, the application further comprises a module configured to generate (or generating) an intelligent scheduling score for each cluster. In various embodiments, the testing is continuous for at least one hour, at least 1 day, at least 1 week, at least 1 month, or at least 1 year. In various embodiments, the plurality of bots comprises at least 10, at least 100, at least 1000, at least 10,000, or at least 100,000 bots. In some embodiments, at least a subset of the bots are configured to interact with the model. In further embodiments, the plurality of bots interact with the model autonomously.

[0046] In some embodiments, the bot priority scores are determined, at least in part, by:wherein:FI is the functionality’s importance to the application (Likert scale of 1 - 7, with seven being most relevant and three being neutral);BR is the relevance a specific bot to the user base (Likert scale of 1 - 7, with seven being most relevant and three being neutral);IF is how frequently a test bot will interact with the specific functionality;El is the impact of an error or failure in a specific functionality (Likert scale of 1 - 7 can also be used here, with 7 indicating a critical failure that would severely impact the user experience); andN is a normalization factor.

[0047] In further embodiments, the hot priority is determined, at least in part, by:wherein: a is a hyperfactor for FI, adjusted to weight FI according to perceived importance;P is a hyperfactor for BR, adjusted to weight BR according to perceived importance; y is a hyperfactor for IF, adjusted to weight IF according to perceived importance; and5 is a hyperfactor for El, adjusted to weight El according to perceived importance.

[0048] In some embodiments, the number of testing phases is determined, at least in part, by:wherein: ceil is the ceiling function;B is total number of bots;SC is the system capacity to handle simultaneous bot interactions;RF is a risk factor between 0 and 1 representing the risk tolerance; andTP is a tuning parameter optionally adjusted based on factors like test coverage, system stability, and testing objectives.

[0049] In some embodiments, the number of bots deployed in each testing phase is determined, at least in part, by:wherein:TDB is total deployed bots with a subscript denoting the phase number (K);BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

[0050] In some embodiments, the resource scaling approach is determined, at least in part, by:wherein:RS subscripts denote the phase number (K) and resource type (T), respectively;BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

[0051] In some embodiments, the scheduling scores are determined, at least in part, by:wherein:BP is the bot priority based on functionality and user relevance;ARL (API Rate Limit) is the maximum number of requests the bot can make to a specific API within a certain timeframe;BCT (Bot Cycle Time) is the time the bot takes to complete one cycle of its operation;CET (Cache Expiry Time) is the duration for which data fetched by the bot from the API is valid; and a, P, y, 5 are hyperparameters adjusted to weight the factors according to relative importance.

[0052] In some embodiments, the application further comprises a module configured to calculate (or calculating) a resource utilization numerator (RUN) score, at least in part, by:wherein: r| is a normalization factor to scale the score to a suitable range (e.g., 0 to 1);RSS is a resource segmentation score;PDS is a phased deployment score; and ISS is an intelligent scheduling score.BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Abetter understanding of the features and advantages of the present subject matter will be obtained by reference to the following detailed description that sets forth illustrative embodiments and the accompanying drawings of which:

[0054] FIG. 1 shows a non-limiting example of a computing device; in this case, a device with one or more processors, memory, storage, and a network interface;

[0055] FIG. 2 shows a non-limiting example of a web / mobile application provision system; in this case, a system providing browser-based and / or native mobile user interfaces;

[0056] FIG. 3 shows a non-limiting example of a cloud-based web / mobile application provision system; in this case, a system comprising an elastically load balanced, auto-scaling web server and application server resources as well synchronously replicated databases;

[0057] FIG. 4A shows a non-limiting example of a graphic user interface (GUI); in this case, a GUI providing tools for a user to define an audience to be simulated by a plurality of bots;

[0058] FIG. 4B shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to select from among recommended templates of bot persona configurations and clusters and to customize templated configurations;

[0059] FIG. 4C shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to customize a “Friendly Bot” cluster configuration including characteristics such as finetuning of humor, attitude, and even-handedness;

[0060] FIG. 4D shows a non-limiting example of a GUI; in this case, a GUI providing information on the training and deployment of a plurality of bots configured using the tools described herein;

[0061] FIG. 5 shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to assemble and deploy a plurality of bots using pre-determined persona configurations and cluster configurations optionally customized;

[0062] FIG. 6 shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to customize a testing approach by comparing hot persona and cluster configurations and viewing / editing specific exemplary questions;

[0063] FIG. 7 shows a non-limiting example of a GUI; in this case, a GUI providing a dashboard for viewing and analyzing flagged breakdowns in the output of a Al system (model);

[0064] FIG. 8 shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to analyze specific model output including responses to the different types of input generated by multiple bot personas;

[0065] FIG. 9A shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to customize a bot persona of a skeptic bot;

[0066] FIG. 9B shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to customize a bot persona of a hostile bot;

[0067] FIG. 9C shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to customize a bot persona by fine tuning specific output including blocking and customizing individual questions and changing characteristics of the bot;

[0068] FIG. 10 shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to review the results of dialog branch discovery for a model; and

[0069] FIG. 11 shows a non-limiting example of a GUI; in this case, a GUI providing tools for a user to monitor production readiness of an Al system (model) by monitoring a bot-based testing campaign over time and reviewing a trust score with regard to topic, tone, and toxicity.DETAILED DESCRIPTION

[0070] Described herein, in certain embodiments, are computer-implemented methods of testing a model comprising: identifying a plurality of Al-driven bots, wherein each bot comprises a persona; generating a bot priority score for each bot; determining a number of testing phases; computing the number of bots deployed in each testing phase; determining a resource scaling approach; generating an intelligent scheduling score for each bot; and deploying the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores.

[0071] Also described herein, in certain embodiments, are computer-implemented systems for testing a model comprising a computing device comprising at least one processor and instructions executable by the at least one processor to cause the at least one processor toperform operations comprising: identifying a plurality of Al-driven bots, wherein each hot comprises a persona; generating a hot priority score for each hot; determining a number of testing phases; computing the number of bots deployed in each testing phase; determining a resource scaling approach; generating an intelligent scheduling score for each bot; and deploying the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores.

[0072] Also described herein, in certain embodiments, are non-transitory computer-readable storage media encoded with instructions executable by one or more processors to provide a model testing application comprising: a module configured to identify a plurality of Al-driven bots, wherein each bot comprises a persona; a module configured to generate a bot priority score for each bot; a module configured to determine a number of testing phases; a module configured to compute the number of bots deployed in each testing phase; a module configured to determine a resource scaling approach; a module configured to generate an intelligent scheduling score for each bot; and a module configured to deploy the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores.

[0073] Also described herein, in certain embodiments, are non-transitory computer-readable storage media encoded with instructions executable by one or more processors to provide a model testing application comprising: a module identifying a plurality of Al-driven bots, wherein each bot comprises a persona; a module generating a bot priority score for each bot; a module determining a number of testing phases; a module computing the number of bots deployed in each testing phase; a module determining a resource scaling approach; a module generating an intelligent scheduling score for each bot; and a module deploying the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores.Certain definitions

[0074] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present subject matter belongs.

[0075] As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Any reference to “or” herein is intended to encompass “and / or” unless otherwise stated.

[0076] Reference throughout this specification to “some embodiments,” “further embodiments,” or “a particular embodiment,” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in some embodiments,” or “in further embodiments,” or “in a particular embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.Computing system

[0077] Referring to FIG. 1, a block diagram is shown depicting an exemplary machine that includes a computer system 100 (e.g., a processing or computing system) within which a set of instructions can execute for causing a device to perform or execute any one or more of the aspects and / or methodologies for static code scheduling of the present disclosure. The components in FIG. 1 are examples only and do not limit the scope of use or functionality of any hardware, software, embedded logic component, or a combination of two or more such components implementing particular embodiments.

[0078] Computer system 100 may include one or more processors 101, a memory 103, and a storage 108 that communicate with each other, and with other components, via a bus 140. The bus 140 may also link a display 132, one or more input devices 133 (which may, for example, include a keypad, a keyboard, a mouse, a stylus, etc.), one or more output devices 134, one or more storage devices 135, and various tangible storage media 136. All of these elements may interface directly or via one or more interfaces or adaptors to the bus 140. For instance, the various tangible storage media 136 can interface with the bus 140 via storage medium interface 126. Computer system 100 may have any suitable physical form, including but not limited to one or more integrated circuits (ICs), printed circuit boards (PCBs), mobile handheld devices (such as mobile telephones or PDAs), laptop or notebook computers, distributed computer systems, computing grids, or servers.

[0079] Computer system 100 includes one or more processor(s) 101 (e.g., central processing units (CPUs), general purpose graphics processing units (GPGPUs), or quantum processing units (QPUs)) that carry out functions. Processor(s) 101 optionally contains a cache memory unit 102 for temporary local storage of instructions, data, or computer addresses. Processor(s) 101 are configured to assist in execution of computer readable instructions. Computer system 100 may provide functionality for the components depicted in FIG. 1 as a result of the processor(s) 101executing non-transitory, processor-executable instructions embodied in one or more tangible computer-readable storage media, such as memory 103, storage 108, storage devices 135, and / or storage medium 136. The computer-readable media may store software that implements particular embodiments, and processor(s) 101 may execute the software. Memory 103 may read the software from one or more other computer-readable media (such as mass storage device(s) 135, 136) or from one or more other sources through a suitable interface, such as network interface 120. The software may cause processor(s) 101 to carry out one or more processes or one or more steps of one or more processes described or illustrated herein. Carrying out such processes or steps may include defining data structures stored in memory 103 and modifying the data structures as directed by the software.

[0080] The memory 103 may include various components (e.g., machine readable media) including, but not limited to, a random access memory component (e.g., RAM 104) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phasechange random access memory (PRAM), etc.), a read-only memory component (e.g., ROM 105), and any combinations thereof. ROM 105 may act to communicate data and instructions unidirectionally to processor(s) 101, and RAM 104 may act to communicate data and instructions bidirectionally with processor(s) 101. ROM 105 and RAM 104 may include any suitable tangible computer-readable media described below. In one example, a basic input / output system 106 (BIOS), including basic routines that help to transfer information between elements within computer system 100, such as during start-up, may be stored in the memory 103.

[0081] Fixed storage 108 is connected bidirectionally to processor(s) 101, optionally through storage control unit 107. Fixed storage 108 provides additional data storage capacity and may also include any suitable tangible computer-readable media described herein. Storage 108 may be used to store operating system 109, executable(s) 110, data 111, applications 112 (application programs), and the like. Storage 108 can also include an optical disk drive, a solid-state memory device (e.g., flash-based systems), or a combination of any of the above. Information in storage 108 may, in appropriate cases, be incorporated as virtual memory in memory 103.

[0082] In one example, storage device(s) 135 may be removably interfaced with computer system 100 (e.g., via an external port connector (not shown)) via a storage device interface 125. Particularly, storage device(s) 135 and an associated machine-readable medium may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 100. In one example, software may reside,completely or partially, within a machine-readable medium on storage device(s) 135. In another example, software may reside, completely or partially, within processor(s) 101.

[0083] Bus 140 connects a wide variety of subsystems. Herein, reference to a bus may encompass one or more digital signal lines serving a common function, where appropriate. Bus 140 may be any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures. As an example and not by way of limitation, such architectures include an Industry Standard Architecture (ISA) bus, an Enhanced ISA (EISA) bus, a Micro Channel Architecture (MCA) bus, a Video Electronics Standards Association local bus (VLB), a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, an Accelerated Graphics Port (AGP) bus, HyperTransport (HTX) bus, serial advanced technology attachment (SATA) bus, and any combinations thereof.

[0084] Computer system 100 may also include an input device 133. In one example, a user of computer system 100 may enter commands and / or other information into computer system 100 via input device(s) 133. Examples of an input device(s) 133 include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device (e.g., a mouse or touchpad), a touchpad, a touch screen, a multi-touch screen, a joystick, a stylus, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), an optical scanner, a video or still image capture device (e.g., a camera), and any combinations thereof. In some embodiments, the input device is a Kinect, Leap Motion, or the like. Input device(s) 133 may be interfaced to bus 140 via any of a variety of input interfaces 123 (e.g., input interface 123) including, but not limited to, serial, parallel, game port, USB, FIREWIRE, THUNDERBOLT, or any combination of the above.

[0085] In particular embodiments, when computer system 100 is connected to network 130, computer system 100 may communicate with other devices, specifically mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, connected to network 130. Communications to and from computer system 100 may be sent through network interface 120. For example, network interface 120 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 130, and computer system 100 may store the incoming communications in memory 103 for processing. Computer system 100 may similarly store outgoing communications (such as requests or responses to other devices) in the form of one or more packets in memory 103 and communicated to network 130from network interface 120. Processor(s) 101 may access these communication packets stored in memory 103 for processing.

[0086] Examples of the network interface 120 include, but are not limited to, a network interface card, a modem, and any combination thereof. Examples of a network 130 or network segment 130 include, but are not limited to, a distributed computing system, a cloud computing system, a wide area network (WAN) (e.g., the Internet, an enterprise network), a local area network (LAN) (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a direct connection between two computing devices, a peer-to-peer network, and any combinations thereof. A network, such as network 130, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used.

[0087] Information and data can be displayed through a display 132. Examples of a display 132 include, but are not limited to, a cathode ray tube (CRT), a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic liquid crystal display (OLED) such as a passive-matrix OLED (PMOLED) or active-matrix OLED (AMOLED) display, a plasma display, and any combinations thereof. The display 132 can interface to the processor(s) 101, memory 103, and fixed storage 108, as well as other devices, such as input device(s) 133, via the bus 140. The display 132 is linked to the bus 140 via a video interface 122, and transport of data between the display 132 and the bus 140 can be controlled via the graphics control 121. In some embodiments, the display is a video projector. In some embodiments, the display is a headmounted display (HMD) such as a VR headset. In further embodiments, suitable VR headsets include, by way of non-limiting examples, HTC Vive, Oculus Rift, Samsung Gear VR, Microsoft HoloLens, Razer OSVR, FOVE VR, Zeiss VR One, Avegant Glyph, Freefly VR headset, and the like. In still further embodiments, the display is a combination of devices such as those disclosed herein.

[0088] In addition to a display 132, computer system 100 may include one or more other peripheral output devices 134 including, but not limited to, an audio speaker, a printer, a storage device, and any combinations thereof. Such peripheral output devices may be connected to the bus 140 via an output interface 124. Examples of an output interface 124 include, but are not limited to, a serial port, a parallel connection, a USB port, a FIREWIRE port, a THUNDERBOLT port, and any combinations thereof.

[0089] In addition or as an alternative, computer system 100 may provide functionality as aresult of logic hardwired or otherwise embodied in a circuit, which may operate in place of or together with software to execute one or more processes or one or more steps of one or more processes described or illustrated herein. Reference to software in this disclosure may encompass logic, and reference to logic may encompass software. Moreover, reference to a computer-readable medium may encompass a circuit (such as an IC) storing software for execution, a circuit embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware, software, or both.

[0090] Those of skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality.

[0091] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0092] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by one or more processor(s), or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage mediummay reside as discrete components in a user terminal.

[0093] In accordance with the description herein, suitable computing devices include, by way of non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, sub-notebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smartphones, tablet computers, personal digital assistants, video game consoles, and vehicles. Those of skill in the art will also recognize that select televisions, video players, and digital music players with optional computer network connectivity are suitable for use in the system described herein. Suitable tablet computers, in various embodiments, include those with booklet, slate, and convertible configurations, known to those of skill in the art.

[0094] In some embodiments, the computing device includes an operating system configured to perform executable instructions. The operating system is, for example, software, including programs and data, which manages the device’s hardware and provides services for execution of applications. Those of skill in the art will recognize that suitable server operating systems include, by way of non -limiting examples, FreeBSD, OpenBSD, NetBSD®, Linux, Apple® Mac OS X Server®, Oracle® Solaris®, Windows Server®, and Novell® NetWare®. Those of skill in the art will recognize that suitable personal computer operating systems include, by way of nonlimiting examples, Microsoft® Windows®, Apple® Mac OS X®, UNIX®, and UNIX-like operating systems such as GNU / Linux®. In some embodiments, the operating system is provided by cloud computing. Those of skill in the art will also recognize that suitable mobile smartphone operating systems include, by way of non-limiting examples, Nokia® Symbian® OS, Apple® iOS®, Research In Motion® BlackBerry OS®, Google® Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile® OS, Linux®, and Palm® WebOS®.Non-transitory computer readable storage medium

[0095] In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more non-transitory computer readable storage media encoded with a program including instructions executable by the operating system of an optionally networked computing device. In further embodiments, a computer readable storage medium is a tangible component of a computing device. In still further embodiments, a computer readable storage medium is optionally removable from a computing device. In some embodiments, a computer readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic disk drives, magnetic tape drives, optical disk drives,distributed computing systems including cloud computing systems and services, and the like. In some cases, the program and instructions are permanently, substantially permanently, semipermanently, or non-transitorily encoded on the media.Computer program

[0096] In some embodiments, the platforms, systems, media, and methods disclosed herein include at least one computer program, or use of the same. A computer program includes a sequence of instructions, executable by one or more processor(s) of the computing device’s CPU, written to perform a specified task. Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), computing data structures, and the like, that perform particular tasks or implement particular abstract data types. In light of the disclosure provided herein, those of skill in the art will recognize that a computer program may be written in various versions of various languages.

[0097] The functionality of the computer readable instructions may be combined or distributed as desired in various environments. In some embodiments, a computer program comprises one sequence of instructions. In some embodiments, a computer program comprises a plurality of sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from a plurality of locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof.Web application

[0098] In some embodiments, a computer program includes a web application. In light of the disclosure provided herein, those of skill in the art will recognize that a web application, in various embodiments, utilizes one or more software frameworks and one or more database systems. In some embodiments, a web application is created upon a software framework such as Microsoft® .NET or Ruby on Rails (RoR). In some embodiments, a web application utilizes one or more database systems including, by way of non-limiting examples, relational, non-relational, object oriented, associative, XML, and document oriented database systems. In further embodiments, suitable relational database systems include, by way of non-limiting examples, Microsoft® SQL Server, mySQL™, and Oracle®. Those of skill in the art will also recognize that a web application, in various embodiments, is written in one or more versions of one or morelanguages. A web application may be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some embodiments, a web application is written to some extent in a markup language such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or extensible Markup Language (XML). In some embodiments, a web application is written to some extent in a presentation definition language such as Cascading Style Sheets (CSS). In some embodiments, a web application is written to some extent in a client-side scripting language such as Asynchronous JavaScript and XML (AJAX), Flash® ActionScript, JavaScript, or Silverlight®. In some embodiments, a web application is written to some extent in a server-side coding language such as Active Server Pages (ASP), ColdFusion®, Perl, Java™, JavaServer Pages (JSP), Hypertext Preprocessor (PHP), Python™, Ruby, Tel, Smalltalk, WebDNA®, or Groovy. In some embodiments, a web application is written to some extent in a database query language such as Structured Query Language (SQL). In some embodiments, a web application integrates enterprise server products such as IBM® Lotus Domino®. In some embodiments, a web application includes a media player element. In various further embodiments, a media player element utilizes one or more of many suitable multimedia technologies including, by way of non-limiting examples, Adobe® Flash®, HTML 5, Apple® QuickTime®, Microsoft® Silverlight®, Java™, and Unity®.

[0099] Referring to FIG. 2, in a particular embodiment, an application provision system comprises one or more databases 200 accessed by a relational database management system (RDBMS) 210. Suitable RDBMSs include Firebird, MySQL, PostgreSQL, SQLite, Oracle Database, Microsoft SQL Server, IBM DB2, IBM Informix, SAP Sybase, Teradata, and the like. In this embodiment, the application provision system further comprises one or more application severs 220 (such as Java servers, .NET servers, PHP servers, and the like) and one or more web servers 230 (such as Apache, IIS, GWS and the like). The web server(s) optionally expose one or more web services via app application programming interfaces (APIs) 240. Via a network, such as the Internet, the system provides browser-based and / or mobile native user interfaces.

[0100] Referring to FIG. 3, in a particular embodiment, an application provision system alternatively has a distributed, cloud-based architecture 300 and comprises elastically load balanced, auto-scaling web server resources 310 and application server resources 320 as well synchronously replicated databases 330. In some embodiments, the synchronously replicated databases optionally expose one or more web services via app application programming interfaces (APIs) 340. In some embodiments, the cloud-based architecture 300 is connected to acontent delivery network (CDN) 350. In some embodiments, the CDN connected to the cloudbased architecture 300 outputs content 360.Mobile application

[0101] In some embodiments, a computer program includes a mobile application provided to a mobile computing device. In some embodiments, the mobile application is provided to a mobile computing device at the time it is manufactured. In other embodiments, the mobile application is provided to a mobile computing device via the computer network described herein.

[0102] In view of the disclosure provided herein, a mobile application is created by techniques known to those of skill in the art using hardware, languages, and development environments known to the art. Those of skill in the art will recognize that mobile applications are written in several languages. Suitable programming languages include, by way of non-limiting examples, C, C++, C#, Objective-C, Java™, JavaScript, Pascal, Object Pascal, Python™, Ruby, VB.NET, WML, and XHTML / HTML with or without CSS, or combinations thereof.

[0103] Suitable mobile application development environments are available from several sources. Commercially available development environments include, by way of non-limiting examples, AirplaySDK, alcheMo, Appcelerator®, Celsius, Bedrock, Flash Lite, .NET Compact Framework, Rhomobile, and WorkLight Mobile Platform. Other development environments are available without cost including, by way of non-limiting examples, Lazarus, MobiFlex, MoSync, and Phonegap. Also, mobile device manufacturers distribute software developer kits including, by way of non-limiting examples, iPhone and iPad (iOS) SDK, Android™ SDK, BlackBerry® SDK, BREW SDK, Palm® OS SDK, Symbian SDK, webOS SDK, and Windows® Mobile SDK.Standalone application

[0104] In some embodiments, a computer program includes a standalone application, which is a program that is run as an independent computer process, not an add-on to an existing process, e.g., not a plug-in. Those of skill in the art will recognize that standalone applications are often compiled. A compiler is a computer program(s) that transforms source code written in a programming language into binary object code such as assembly language or machine code. Suitable compiled programming languages include, by way of non-limiting examples, C, C++, Objective-C, COBOL, Delphi, Eiffel, Java™, Lisp, Python™, Visual Basic, and VB .NET, or combinations thereof. Compilation is often performed, at least in part, to create an executableprogram. In some embodiments, a computer program includes one or more executable complied applications.Software modules

[0105] In some embodiments, the platforms, systems, media, and methods disclosed herein include software, server, and / or database modules, or use of the same. In view of the disclosure provided herein, software modules are created by techniques known to those of skill in the art using machines, software, and languages known to the art. The software modules disclosed herein are implemented in a multitude of ways. In various embodiments, a software module comprises a file, a section of code, a programming object, a programming structure, a distributed computing resource, a cloud computing resource, or combinations thereof. In further various embodiments, a software module comprises a plurality of files, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, a plurality of distributed computing resources, a plurality of cloud computing resources, or combinations thereof. In various embodiments, the one or more software modules comprise, by way of nonlimiting examples, a web application, a mobile application, a standalone application, and a distributed or cloud computing application. In some embodiments, software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some embodiments, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine. In further embodiments, software modules are hosted on a distributed computing platform such as a cloud computing platform. In some embodiments, software modules are hosted on one or more machines in one location. In other embodiments, software modules are hosted on one or more machines in more than one location.Databases

[0106] In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more databases, or use of the same. In view of the disclosure provided herein, those of skill in the art will recognize that many databases are suitable for storage and retrieval of model, bot, persona, interaction, axiom, and use case information. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object oriented databases, object databases, entity-relationship model databases, associative databases, XML databases, document oriented databases, and graph databases. Further non-limiting examples include SQL, PostgreSQL, MySQL, Oracle, DB2, Sybase, andMongoDB. In some embodiments, a database is Internet-based. In further embodiments, a database is web-based. In still further embodiments, a database is cloud computing-based. In a particular embodiment, a database is a distributed database. In other embodiments, a database is based on one or more local computer storage devices.Testing

[0107] In some embodiments, the platforms, systems, media, and methods disclosed herein are designed and / or configured for testing use cases. In further embodiments, the platforms, systems, media, and methods disclosed herein are designed and / or configured for testing of software such as generative Al models, including LLMs, e.g., large natural language conversational models. The plurality of bots, in some cases, are unified in their testing purpose, procedure, and / or use case. In other cases, the plurality of bots are divided into multiple groups having distinct testing purposes, procedures, and / or use cases, and may be deployed in different sequences, such that the procedures and / or use cases are performed in different sequences.

[0108] In some embodiments, testing involves, at a minimum, checking a response to a question, and in further embodiments, testing extends well beyond accuracy. Accuracy is essential for some interaction types - where a clear, correct answer exists and is known, or at least a set of promising solutions are known, and is expressible in quantitative forms. However, in some cases, the sentiment and tonality, sentiment, etc. of the reaction are also essential. By way of example, an accurate answer returned in an inappropriate tone may be considered a breakdown.

[0109] In some embodiments, the platforms, systems, media, and methods disclosed herein are designed and / or configured for semi-continuous, substantially continuous, and / or continuous testing over a period of time. Continuous testing is an important feature because Al models are continuously learning and changing. As such, results of testing in the past are at high risk of not reflecting the current state of the system. Moreover, in some embodiments, the use of semi- autonomous, substantially autonomous, and / or autonomous bots facilitates continuous testing. By way of non-limiting examples, the testing is suitably conducted substantially continuously or continuously for about one hour, about 1 hour, about 12 hours, about 1 day, about 1 week, about 1 month, about 6 months, about 1 year, about 2 years, about 3 years, or more, including increments therein. By way of further non-limiting examples, the testing is suitably conducted substantially continuously or continuously for at least one hour, at least 1 hour, at least 12 hours, at least 1 day, at least 1 week, at least 1 month, at least 6 months, at least 1 year, at least 2 years,at least 3 years, or more, including increments therein.

[0110] In some embodiments, the testing is performed to identify performance breakdowns in large natural language conversational models. In further embodiments, breakdowns include inappropriate or unsuitable responses that undermine, or should undermine trust in the model and its responses. By way of non-limiting examples, the breakdowns may include an inappropriate sentiment, a low integrity, a false fact, a high inconsistency, a prejudicial opinion, a discordance with an axiom of the model, a cultural discordance of the model, or any combination thereof. In various embodiments, all of the bots, some of the bots, a subset of the bots, a few of the bots, or just one of the bots are configured for identifying performance breakdowns, and may be deployed for this purpose prior to, or subsequent to other testing use cases.[OHl] In some embodiments, the testing is performed to conduct dialog branch discovery in large natural language conversational models. In some embodiments, dialog branch discovery includes mapping all possible paths through the dialog of the model. In some embodiments, dialog branch discovery includes identifying one or more significant paths through the dialog of the model. In various embodiments, all of the bots, some of the bots, a subset of the bots, a few of the bots, or just one of the bots are configured for conducting dialog branch discovery, and may be deployed for this purpose prior to, or subsequent to other testing use cases.

[0112] In some embodiments, the testing is performed to conduct testing configuration optimization for a model. In further embodiments, testing configuration optimization includes determining and / or optimizing the number of bots in the plurality of bots, one or more bot deployment locations, one or more bot personas for the plurality of bots, or any combination thereof. In particular embodiments, testing configuration optimization includes minimizing the number of bots in the plurality of bots to conserve computing resources and / or bandwidth. In some embodiments, testing configuration optimization includes selecting one or more testing parameters from a plurality of pre-configured options. By way of example, in some cases, testing configuration optimization includes selecting from among a plurality of pre-configured bundles of configurations for the plurality of bots. In various embodiments, pre-configured options pertain to the number of bots in the plurality of bots, one or more bot deployment locations, one or more bot personas for the plurality of bots, or any combination thereof. In various embodiments, all of the bots, some of the bots, a subset of the bots, a few of the bots, or just one of the bots are configured for testing configuration optimization, and may be deployed for this purpose prior to, or subsequent to other testing use cases.Models

[0113] In some embodiments, the platforms, systems, media, and methods disclosed herein include models, or use of the same. In some embodiments, the platforms, systems, media, and methods disclosed herein are configured to test one or more models. In some embodiments, the platforms, systems, media, and methods disclosed herein are configured to train one or more models.

[0114] In some embodiments, the model comprises an indeterminate model. In some embodiments, the model comprises a generative Al model. In further embodiments, the model comprises a large language model. In still further embodiments, the model comprises a large natural language conversational model. In some embodiments, the platforms, systems, media, and methods disclosed herein are configured to test and / or train one or more systems interfacing with, or integrating, one or more indeterminate models, such as a generative Al model, a large language model, and / or a large natural language conversational model.

[0115] Expert systems, also known as knowledge-based systems or rule-based systems, are Al applications designed to simulate the decision-making abilities of a human expert in a specific domain. In some embodiments, the platforms, systems, media, and methods disclosed herein are configured to test one or more expert systems.Bots

[0116] In some embodiments, the platforms, systems, media, and methods disclosed herein include bots, or use of the same. An Internet bot, web bot, web robot, robot, or simply bot, is a software application that runs automated tasks over a network, such as the Internet. In some cases the tasks are repetitive and sometimes conducted with the intent to imitate human activity. Many tasks are suitable for the bots, including, by way of non-limiting examples, browsing, clicking, scraping, messaging, texting, downloading, shopping, conversing, asking questions, pursuing objectives, and the like. In some cases, bots are used on a large scale such that the tasks are performed on a large scale.

[0117] In some embodiments, the bots are Al-driven bots (e.g., bots comprising Al or using and / or accessing Al) and are used to generate bot-to-bot communication in order to discover system capabilities and performance. In further embodiments, to test models, such as LLMs and expert systems, the platforms, systems, media, and methods disclosed herein include an application comprising at least one bot to interact with the tested system. In still further embodiments, the platforms, systems, media, and methods disclosed herein include multiple botsto interact with each other in order to address breakdowns in models, such as LLMs and expert systems. In still further embodiments, to test models, such as LLMs and expert systems, the platforms, systems, media, and methods disclosed herein include primary bots, which interact with the tested system, and also secondary bots, which interact with the primary bots, for example to develop prompts and questions used by the primary bots.

[0118] In various embodiments, all of the bots, some of the bots, a subset of the bots, a few of the bots, or just one of the bots are deployed and configured to interact with a model. In further embodiments, each bot comprises a persona and interacts with the model based, at least in part, on that persona. In still further embodiments, the bots interact with the model by providing input in the form of messages via written or audio speech, as part of a conversation-type interaction. The bots are optionally deployed, entirely or partially, at a variety of locations, such as various interfaces with the model or a system or platform interfacing with, or integrating, the model. In some cases, the bots are deployed at one location. In other cases, the bots are deployed at multiple locations.

[0119] In further embodiments, a plurality of bots are used. The plurality of bots suitably has a wide range of numbers of bots. By way of non-limiting examples, the plurality of bots, in various embodiments, includes about 2, 3, 4, 5, 6, 7, 8, 9, or 10 bots, about 20, 30, 40, 50, 60, 70 80, 90, or 100 bots, about 200, 300, 400, 500, 600, 700, 800, 900, or 1000, about 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, or 10,000 bots, about 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000 or 100,000 bots, about 200,000, 300,000, 400,000, 500,000, 600,000, 700,000, 800,000, 900,000 or 1,000,000 bots, or more, including increments therein. By way further of non-limiting examples, the plurality of bots, in various embodiments, includes at least 2, 3, 4, 5, 6, 7, 8, 9, or 10 bots, at least 20, 30, 40, 50, 60, 70 80, 90, or 100 bots, at least 200, 300, 400, 500, 600, 700, 800, 900, or 1000, at least 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, or 10,000 bots, at least 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000 or 100,000 bots, at least 200,000, 300,000, 400,000, 500,000, 600,000, 700,000, 800,000, 900,000 or 1,000,000 bots, or more, including increments therein. By way still further of nonlimiting examples, the plurality of bots, in various embodiments, includes at most 2, 3, 4, 5, 6, 7, 8, 9, or 10 bots, at most 20, 30, 40, 50, 60, 70 80, 90, or 100 bots, at most 200, 300, 400, 500, 600, 700, 800, 900, or 1000, at most 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, or 10,000 bots, at most 20,000, 30,000, 40,000, 50,000, 60,000, 70,000, 80,000, 90,000 or 100,000 bots, at most 200,000, 300,000, 400,000, 500,000, 600,000, 700,000, 800,000, 900,000 or 1,000,000 bots, including increments therein.Personas with profiles

[0120] In some embodiments, the platforms, systems, media, and methods disclosed herein include personas, or use of the same. For example, in some embodiments, the each bot, of the plurality of bots, comprises a persona. A persona includes one or more aspects of human personality, such as a personality trait or characteristic, attitude, sentiment, affect, intent, demeanor, idiosyncrasy, defect, condition, and the like. In some embodiments, each bot is configured to conduct one or more tasks in a way that is directed by, or influenced by, one or more personas of the bot. In some embodiments, each bot comprises the same persona. In other embodiments, the plurality of bots comprises groups of bots with different personas. In some embodiments, each persona comprises one or more profiles implementing different traits or characteristics of the persona. Many profiles are suitable. In various embodiments, profiles include, by way of non-limiting examples, happy, easy-going, blissful, joking, funny, friendly, manic, and the like. In various embodiments, profiles include, by way of non-limiting examples, skeptical, angry, hostile, mean, profane, argumentative, toxic, and trolling, and the like. In various embodiments, profiles include, by way of non-limiting examples, sad, depressed, lethargic, suicidal, and the like. In some embodiments, each bot is configured to interact with a model in a way that is directed by, or influenced by, one or more personas of the bot comprising one or more profiles. In further embodiments, a bot persona, or a profile, may direct and / or influence the subject matter of an interaction or an element of an interaction, the tone of an interaction or an element of an interaction, the timing of an interaction or an element of an interaction, the phrasing of an interaction or an element of an interaction, the syntax of an interaction or an element of an interaction, and the like.

[0121] In particular embodiments, one or more personas comprises a profile derived from customer relationship management (CRM) system data. By way of non-limiting example, in some cases, the persona of one or more bots is optionally designed to reflect a particular past negative or unsuccessful interaction recorded in a CRM system. By way of further non-limiting example, the persona of one or more bots is optionally designed to reflect a plurality of past negative or unsuccessful interactions recorded in a CRM system. By way of non-limiting example, in other cases, the persona of one or more bots is optionally designed to reflect a particular past positive or successful interaction recorded in a CRM system. By way of further non-limiting example, the persona of one or more bots is optionally designed to reflect a plurality of past positive or successful interactions recorded in a CRM system.Configuring bot personas

[0122] The Big Five personality characteristics, also known as the Five-Factor Model (FFM), are a widely accepted and researched model for describing human personality traits. The model suggests that there are five core dimensions of personality: 1) Openness to experience; 2) Conscientiousness; 3) Extraversion; 4) Agreeableness; and 5) Neuroticism (emotional stability). In the Big Five personality model, scores use a quantitative approach based on questionnaires or self-report inventories. The most common method involves participants responding to a series of statements or questions designed to measure each of the five traits on a Likert scale, which is a numerical scale that ranges from strongly disagree to strongly agree (for example, a 1-5 or 1-7 scale).

[0123] The Big Five Inventory (BFI) and revised BFI2 are well-known questionnaires validated through extensive research and widely used in various clinical, educational, and organizational settings. Domain and facet scales are two hierarchical levels of measurement within these personality questionnaires. Domain scales represent broader dimensions of personality, whereas facet scales represent specific subcomponents or aspects of those more general dimensions. Facet scales form the domain scales, allowing for a more detailed understanding of an individual’s personality.

[0124] By way of example, the original BFI consists of 44 items, measuring each trait by 8 to 10 statements or questions. Respondents rate the extent to which they agree or disagree with each statement on a Likert scale, typically ranging from 1 (strongly disagree) to 5 (strongly agree). By way of further example, the BFI2 consists of a total of 60 questions, and for each question, five possible outcomes. Thus, the entire knowledge base has a capacity of 300 prompts.

[0125] By measuring both domain and facet scales, these questionnaires comprehensively assesses an individual’s personality. The domain scales offer an overview of where a person falls along the continuum of the Big Five traits. In contrast, the facet scales provide a more detailed understanding of the aspects contributing to their overall personality profile.

[0126] In some embodiments, configuring a bot, bot persona, and / or bot cluster, is a matter of determining how the desired personality aligns on the domain scales. For example, if we wish to configure a highly extroverted bot, we assign a score of 5 (AGREE STRONGLY) to the extrovert prompt. Then, in further embodiments, the system randomly chooses one facet for that domain. Next, in still further embodiments, from that facet, the system randomly chooses onequestion. Finally, in such embodiments, the system selects the text that matches the AGREE STRONGLY input for that question.

[0127] By way of non-limiting example, the corresponding text “lam highly extroverted and thrive in social situations’" becomes part of our input-prompt engineering for the new bot. Specifically, in some embodiments, such a bot is configured as follows: profile ( open=Bf iScale . AGREE_A_LITTLE , conscientious=Bf iScale . DISAGREE_STRONGLY , agreeable=Bf iScale . DISAGREE_A_LITTLE , neurotic=Bf iScale . AGREE_STRONGLY , extravert=Bf iScale . AGREE A LITTLE ,

[0128] The above configuration will create a highly neurotic bot with open and extraverted tendencies but neither agreeable nor conscientious in its outcomes.

[0129] The design presents a large number of possible configurations. To find the number of configurations, consider that there are five choices, each with five possible values. For each option, the system can pick one of the five values. Thus there are 5A5 = 3125 total bot configurations. That is, a total of 3, 125 distinct bot personas are available. Furthermore, for each given configuration, there are multiple possible outcomes. Consider that each parameter is a domain mapped to three facets, each to four questions. The system, in some embodiments, randomly samples from the population of 12 questions for each domain value provided. Thus, if a bot is configured as depicted above (open=4, conscientious=l, agreeable=2, neurotic=5, extravert=4), there are 300 possible (5x5x12) outcomes for each configuration. Therefore, in such embodiments, even among bots with the same personality configuration, shades of meaning and nuanced differences exist. Given that there are 3,125 possible configurations, each with 300 possible outcomes, the system can generate 300 x 3,125 = 937,000 distinct bot personas.Prompt Engineering and Testing

[0130] In some embodiments, personality prompts are turned into LLM instructions varying on a per-model basis. At the time of this writing, GPT-4 is not yet available via an API. GPT-4 can only be demonstrated via a web browser, which makes it less useful for automated systems. Forthis reason, the following models are primarily used: OpenAI: GPT 3.5; text-davinci-003; text- davinci-002; and HuggingFace: OpenAssistant / oasst-sft-6-llama-30b.Bot clusters

[0131] In some embodiments, the platforms, systems, media, and methods disclosed herein include bot clusters, or use of the same. For example, in various embodiments, bots are organized into functionality clusters, including, by way of non-limiting examples, Personality Clusters, Red Teams, Sentiment Clusters, Domain Knowledge Clusters, and Audience Segment Clusters. In further embodiments, clusters aim to evaluate and improve the performance of generative Al systems by testing their ability to generate accurate, relevant, and appropriate responses across a wide range of user types and conversational scenarios. In such embodiments, by exploring the full spectrum of user behaviors and characteristics, the bots help Al developers identify and address potential weaknesses, biases, or vulnerabilities in their systems, ultimately leading to a more robust, reliable, and user-friendly Al-driven experience.Personality Clusters

[0132] In some embodiments, at least some of the bots in the plurality of bots are organized into functional Personality Clusters. In further embodiments, Personality Clusters are groups of bots designed to represent a wide range of distinct personality types based on established personality frameworks, including, by way of example, the Big Five (also known as OCEAN: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism) and other psychological models. In such embodiments, a primary purpose of Personality Clusters is to test the Al system’s ability to interact effectively and appropriately with users who exhibit different personality traits and communication styles.

[0133] In various embodiments, key features and functions of Personality Clusters include, by way of non-limiting examples:

[0134] Diverse Personalities: Personality Clusters consist of nearly a million unique personality types, some subtly distinct while others showing stark differences. This diversity helps simulate real-world interactions, as users of Al systems are likely to have a broad range of personalities.

[0135] Influencing Dialogue: Each bot’s personality type impacts the initial question it asks and the subsequent responses it generates during the conversation. Different personality traits may affect the tone, content, and communication style.

[0136] Length and Quality of Dialogue: Bots with different personality traits may engage in varying sizes and qualities of conversation. For example, bots with high extroversion may participate in more extended and engaging dialogues, whereas those with low extroversion may be more reserved and concise in their interactions.

[0137] Exploration Methods: Personality traits can influence how bots explore conversation topics. For instance, a bot with high curiosity might employ a depth-first approach, delving deeply into a single topic. In contrast, a more organized bot might use a breadth-first method, discussing multiple issues more superficially.

[0138] Adaptability Testing: Personality Clusters help assess the Al system’s ability to adapt its responses and communication style to match the user's personality. Adaptability is essential for an Al system to create a personalized and satisfying user experience.Red Teams

[0139] In some embodiments, at least some of the bots in the plurality of bots are organized into functional Red Teams. In the context of Al testing, Red Teams are specialized clusters of bots designed to challenge and stress-test Al systems, specifically by targeting their vulnerabilities and attempting to exploit them. In some embodiments, a primary goal of Red Teams is to identify weaknesses in the system’s understanding, behavior, or response generation to improve the Al system's robustness, security, and overall quality.

[0140] In various embodiments, Red Teams employ one or more tactics and strategies to achieve this goal including, by way of non-limiting examples:

[0141] Known Exploits: Red Teams use a collection of predefined input prompts that are known to cause issues in Al systems, such as generating biased, offensive, or inappropriate responses. By testing the system with these known exploits, developers can assess the Al’s resilience and take necessary steps to address any vulnerabilities.

[0142] Semantic Trickery: Red Team bots may use intentionally deceptive or confusing language, attempting to mislead the Al system or provoke an incorrect or nonsensical response. Such language can involve using ambiguous phrases, contradictory statements, or logical fallacies to test the Al’s ability to identify and handle such complexities.

[0143] Adversarial Attacks: Red Teams might create adversarial input prompts designed to cause the Al system to produce incorrect or harmful responses. Careful crafting of these promptscan exploit specific vulnerabilities, such as biases in the Al’s training data or limitations in its reasoning abilities.

[0144] Toxic Behavior: Red Team bots can simulate highly toxic or troll-like behavior to test the Al system's ability to cope with and respond appropriately to abusive or aggressive users. Harmful behavior may include profanity, hate speech, personal attacks, or other forms of offensive language.

[0145] Social Engineering: Some Red Team bots might engage in social engineering tactics, attempting to manipulate the Al system into revealing sensitive information, granting unauthorized access, or taking actions that could compromise user privacy or system security.Sentiment Clusters

[0146] In some embodiments, at least some of the bots in the plurality of bots are organized into functional Sentiment Clusters. In further embodiments, Sentiment Clusters are optionally an important component of the plurality of bots, as they can play a crucial role in evaluating and improving the performance of Al systems. These clusters aim to test the system’s ability to generate accurate, relevant, and appropriate responses across a wide range of sentiment behaviors, not just positive or negative. By exploring the full spectrum of user emotions and behaviors, Sentiment Clusters can help identify potential weaknesses or biases in the system's ability to interpret and respond to user sentiment accurately.

[0147] In various embodiments, to achieve this goal, Sentiment Clusters use a wide range of bots that simulate different user behaviors and emotions. For example, some bots start positively and become gradually more negative over time, while others begin negatively and become increasingly optimistic as the dialog advances. These bots can also approach sentiment changes randomly or have highly-sensitive trigger points that can quickly turn a conversation from positive to negative. As a result, Sentiment Clusters can help developers identify and address potential weaknesses or biases in the system's sentiment analysis capabilities by testing the system’s ability to interpret and respond to these diverse scenarios accurately.

[0148] In some embodiments, in addition to testing the system’s ability to interpret and respond to user sentiment accurately, Sentiment Clusters also optionally play an important role in improving the overall user experience. In such embodiments, by ensuring that Al systems can accurately understand and respond to a wide range of user emotions and behaviors, Sentiment Clusters can help create more user-friendly and engaging conversational experiences. The sentiment is often fundamental in fields such as customer service, where accurately interpretingand responding to user sentiment can significantly affect user satisfaction and loyalty. In addition, developers can ensure that their systems provide the best possible user experience by continuously testing and improving Al systems through Sentiment Clusters.

[0149] By testing the system’s ability to accurately interpret and respond to a wide range of user emotions and behaviors, Sentiment Clusters can help identify potential weaknesses or biases in the system's sentiment analysis capabilities. Moreover, Sentiment Clusters can help improve the overall user experience by ensuring that Al systems can accurately understand and respond to user sentiment, leading to more user-friendly and engaging conversational experiences.Domain Knowledge Clusters

[0150] In some embodiments, at least some of the bots in the plurality of bots are organized into functional Domain Knowledge Clusters. In further embodiments, Domain Knowledge Clusters are groups of bots with different levels of expertise in specific subject areas or domains. In such embodiments, a primary purpose of a Domain Knowledge Cluster is to test the Al system’s ability to provide accurate, relevant, and appropriate information and responses across various topics. In addition, by simulating users with different knowledge levels, these clusters help evaluate how well the Al system can adapt its explanations and communication style to match the user’s understanding.

[0151] In various embodiments, Domain Knowledge Clusters operate in different domains including, by way of non -limiting examples:

[0152] History: In the history domain, bots may have knowledge levels ranging from a basic understanding of general historical events to specialized knowledge of specific periods, regions, or historical figures. These bots can assess the Al system’s ability to provide accurate and contextually appropriate information and engage in meaningful discussions on historical topics with users with varying levels of expertise.

[0153] Art: Bots with domain knowledge in the art could include those with a casual interest in famous artworks or artists, art enthusiasts with a deeper understanding of different styles and movements, or experts in specific areas such as Renaissance art or abstract expressionism. By interacting with these bots, the Al system can demonstrate its ability to discuss artistic concepts, evaluate art pieces, or recommend artists and works that cater to users with different knowledge levels and preferences.

[0154] Science: In the science domain, bots may have varying degrees of expertise in physics, chemistry, or biology. Some bots may possess a general understanding of scientific principles,while others might have specialized knowledge in areas like quantum mechanics or molecular biology. These hots can test the Al system’s ability to explain complex scientific concepts in layperson's terms or engage in more advanced discussions with users with a strong science background.

[0155] Sports: Bots with domain knowledge in sports could encompass a wide range of expertise levels, from casual fans who enjoy watching popular sports to experts with an in-depth understanding of specific sports, teams, or athletes. These bots can challenge the Al system's ability to discuss sports-related topics, provide insights, or analyze the performance of teams and players, catering to different users’ interests and knowledge levels.Audience Segment Clusters

[0156] In some embodiments, at least some of the bots in the plurality of bots are organized into functional Audience Segment Clusters. In further embodiments, within the plurality of bots, Audience Segment Clusters reflect a broad spectrum of potential user demographics. In such embodiments, an Audience Segment Cluster may represent various age groups, cultural backgrounds, professional fields, interest groups, and other categories of individuals who might interact with the Al. Each bot within a cluster emulates the characteristics, behaviors, and communication styles typical of its assigned demographic.

[0157] In some embodiments, the functionality of these clusters is to test and evaluate the versatility and inclusivity of an Al system. They interact with the Al in ways that mimic how real users from their respective demographics would, using language, expressions, queries, and even the context representative of their segment.

[0158] In one example, a bot from a cluster representing teenagers might use current slang, discuss popular culture, and ask questions about school or social media. In another contrasting example, a bot from a cluster representing senior citizens might use more formal language, discuss gardening or history, and ask questions related to healthcare or family. Similarly, a cluster representing professionals in a specific field, say medical professionals, might use technical, medical jargon and discuss topics relevant to their profession.

[0159] In such embodiments, as these clusters interact with the Al, developers can collect data on how well the generative Al responds. For example, are the responses accurate, relevant, and appropriate? Does the Al understand and use language, terms, and references specific to each audience segment? Can the Al adapt its communication style to suit different user types?

[0160] Moreover, in some embodiments, Audience Segment Clusters also test the Al’s ability to handle transitions between different user types. For example, how smoothly can the Al shift from a conversation about video games with a teenager bot to discussing complex medical terminology with a doctor bot? Again, this is an essential aspect of creating Al systems that are truly versatile and adaptable.

[0161] In further embodiments, the system can use the insights gained from the interactions between the Al system and the Audience Segment Clusters to see where the endpoint may fall short. The Al can be tweaked and trained to understand better and respond to different user types, thereby reducing bias and improving user experience. In such embodiments, a goal is to create an Al system that is not only intelligent but also inclusive, adaptable, and capable of serving a diverse range of users.Interactions

[0162] In some embodiments, the platforms, systems, media, and methods disclosed herein include interactions with one or more models, or use of the same. In further embodiments, the interactions are conducted as one or more conversations of variable length, conducted in one or more languages, via written or audio (spoken) input. In still further embodiments, the interactions of the bots with the model simulate human interactions with the model. For example, in various embodiments, a bot persona, or a profile, may direct and / or influence the subject matter of an interaction or an element of an interaction, the tone of an interaction or an element of an interaction, the timing of an interaction or an element of an interaction, the phrasing of an interaction or an element of an interaction, the syntax of an interaction or an element of an interaction, and the like. In some embodiments, one or more of the bots interacts with the model semi-autonomously and / or autonomously.

[0163] Interactions, useful for the use cases described herein, suitably take many forms. In some embodiments, at least one of the bots interacts with the model by introducing one or more of: non-objecti enable content, a non-prejudicial opinion, a true fact, a compliment, and the like. In some embodiments, at least one of the bots interacts with the model by introducing one or more of: objectionable content, irrelevant content, a prejudicial opinion, a false fact, an insult, and the like. In some embodiments, the platforms, systems, media, and methods disclosed herein include harvesting output resulting from one or more interactions with one or more models and analyzing the output.Breakdowns

[0164] In some embodiments, the platforms, systems, media, and methods disclosed herein include harvesting output resulting from interactions of hots with models, analyzing the output, and screening for one or more breakdowns in the model’s performance and / or responses. Many types of breakdowns are optionally screened for, detected, and / or identified. In various embodiments, a breakdown includes, by way of non-limiting examples, an inappropriate sentiment, a low integrity, a false fact, a high inconsistency, a prejudicial opinion, a discordance with an axiom of the model, and a cultural discordance of the model. In some embodiments, a response is considered a breakdown based, at least in part, on the context of the interaction, e.g., the persona of the bot, the subject matter of the interaction, the date / time, previous interactions, and the like.

[0165] In some embodiments, upon detection and / or identification of a breakdown, a notification is sent to an administrator, or the like, to alert them to the breakdown, the nature of the breakdown, the date / time of the breakdown, and / or the context of the breakdown. In some embodiments, upon detection and / or identification of a breakdown, a remedial action is taken, such as providing feedback or reinforcement learning to the model and / or instructing the model to send an updated and / or revised response.Training

[0166] In some embodiments, the platforms, systems, media, and methods disclosed herein include model training, or use of the same. In further embodiments, the platforms, systems, media, and methods disclosed herein are optionally employed to perform the use case of training a generative Al model, such as a LLM and / or a large natural language conversational model. In this particular case, the plurality of bots are used to train a model to have one or more particular personas. The uses of a generative model having a particular persona are manifold, but nonlimiting examples include, an expert system, a tutoring model, a coaching model, or a research model, a sales model, a customer service model, and the like.Bot priority

[0167] In some embodiments, the platforms, systems, media, and methods disclosed herein include developing bot and / or bot cluster priorities, or use of the same. In further embodiments, a bot priority score is a tool to help prioritize and manage the bot army based on each bot’s importance and relevance, and / or each bot cluster’s importance and relevance, to the overall goals of the testing. In such embodiments, deriving a score for resource segmentation is afoundational measure of hot priority (BP) having, by way of non-limiting examples, at least the following uses:

[0168] Prioritizing Deployment: In some cases, hots with a higher priority will deploy first. Deployment priority tests the most critical functionalities and user interactions first.

[0169] Dynamic Resource Allocation: In some cases, hots with a higher priority will have a higher resource allocation. Allocation refers to, for example, more CPU time, memory, or a larger share of network bandwidth, depending on the resources most relevant for the bots’ operation. Resource allocation may also imply the removal of caps or performance ceilings in the cloud vendor’s environment.

[0170] Error Handling: In some cases, bots with higher priority have corresponding flags if an error or breakdown occurs during testing. For example, a high-priority bot that finds a vulnerability might create a “severe” ticket, whereas a lower-priority bot may have “neutral” or “lower” severity ratings.

[0171] Scaling and Optimization: In some cases, the bot priority score can also guide scaling and optimization decisions. For example, if a bot with a high priority is consistently underperforming, this could be a signal to scale up resources or optimize the bot’s operation.Bot Priority Computation

[0172] Resource segmentation of the bot army is a multi-faceted problem that requires quantitative and qualitative factors. As described herein, the fundamental formula must consider the following four factors:

[0173] Functionality Importance (FT): This measures a specific functionality’s importance to the application. Functionality Importance is represented on, for example, a Likert scale of 1 - 7, with seven being most relevant and three being neutral.

[0174] Bot Relevance (BR): This measures how relevant a specific bot (or bot cluster) is to the user base. Bot Relevance is represented on, for example, a Likert scale of 1 - 7, with seven being most relevant and three being neutral.

[0175] Interaction Frequency (IF): This measures how frequently a test bot will interact with the specific functionality. Estimates for Interaction Frequency ultimately come from historical data but can be heuristically estimated using a middle ground.

[0176] Error Impact (El): This measures the impact of an error, failure, or breakdown in a specific functionality. A Likert scale of 1 - 7 is optionally used here, with 7 indicating a criticalfailure that would severely impact the user experience.

[0177] Having determined these four factors for each hot or hot cluster and the functionality they are testing, an exemplary hot priority score is:Equation 1( / x BR x IF x EF)BP =

[0178] In the example of Equation 1, TV is a normalization factor that could be the maximum possible score (if all elements were at their maximum). This formula would give a score between 0 and 1 for each bot or bot cluster, with higher scores indicating a higher priority for deployment in testing. Having declared the simple formula, we can now introduce hyperparameters to control the relative importance of each factor. Users can set these hyperparameters based on domain knowledge, experimentation, or optimization methods such as cross-validation or grid search.

[0179] These hyperparameters are denoted as:

[0180] a for Functionality Importance (F / );

[0181] P for User Relevance (UR

[0182] y for Interaction Frequency ( / F); and

[0183] 6 for Error Impact (E / ).

[0184] These hyperparameters are optionally adjusted to weight each factor according to its perceived importance.

[0185] So, for example, a refined formula for the Bot Priority (BP) is:Equation 2

[0186] In the refined example of Equation 2, each factor in the BP calculation is weighted by its corresponding hyperparameter, allowing for greater control over the relative importance of each element. It’s also worth noting that the values of a, P, y, and 5 should be carefully chosen based on domain knowledge and the specific requirements of the system. They should reflect each factor's relative importance to the overall goal of the testing. These values are optionally set heuristically or via optimization techniques.Phased deployment and dynamic scaling

[0187] In some embodiments, the platforms, systems, media, and methods disclosed herein include phased deployment and / or dynamic scaling, or use of the same. Phased deployment and dynamic scaling are two sides to the same coin. Phased deployment is, for example, concerned primarily with functional software specification and dynamic scaling, for example, with the environment (e.g., non-functional constraints). Described herein are methodologies and software to determine the number of phases for deployment of a testing regime. In some embodiments, considerations include, for example, the number of phases ( ) as a function of the total number of bots (B), system capacity (SC), risk factor (RF), and a tuning parameter (TP) that encapsulates other factors like test coverage, system stability, and testing objectives.

[0188] For example, a formula is:Equation 3

[0189] In the example of Equation 3, “ceil” represents the ceiling function, which rounds up to the nearest integer, ensuring that you have a whole number of phases, and:

[0190] Total Number of Bots (B) is the total number of bots in the bot army.

[0191] System Capacity (SC) is the capacity of the system to handle simultaneous bot interactions.

[0192] Risk Factor (RF) is a value that optionally ranges between 0 and 1, representing the risk tolerance. A lower value indicates a higher tolerance for risk, which would reduce the number of phases. In comparison, a higher value indicates a lower tolerance for risk, increasing the number of phases.

[0193] Tuning Parameter (TP) is a parameter is optionally adjusted based on factors like test coverage, system stability, and testing objectives. A higher value for this parameter would reduce the number of phases, while a lower value would increase the number of phases.

[0194] Subsequent to determining the number of phases, a next step is to compute the total number of deployed bots in each stage and the scaling approach for infrastructure. To formulate these strategies, the following factors may be considered:

[0195] Bot Priority (BP). Derived from the previous RSS calculation, which considersFunctionality Importance, User Relevance, Interaction Frequency, and Error Impact.

[0196] System Capacity (SC) The system’s current capacity, measured in terms of concurrent users and requests per second.

[0197] Current Load (CL): The current load on the system, measured in the same units as the system capacity.

[0198] Cost Factor (CF): This factor integrates the cost of the resources necessary to scale the system, such as CPU, memory, storage, and network bandwidth. The Cost Factor is calculated based on current cloud service rates.

[0199] An exemplary formula for the number of bots to deploy in each phase is Total Deployed Bots (T B) with a subscript denoting the phase number (K):Equation 4

[0200] In the example of Equation 4, the formula deploys bots based on their priority, the available system capacity, and the cost of scaling the system. It ensures higher priority bots deploy first and that deployment balances against the cost of scaling the system.

[0201] For dynamic scaling, an exemplary formula is:Equation 5

[0202] In the example of Equation 5, the subscripts denote the phase number K) and resource type (T), respectively. Equation 5 scales resources based on the current load, the priority of the bots, and the cost factor. It ensures that resources are scaled up when high-priority bots are active, and the system is nearing capacity and scaled down when the system load is low.Intelligent scheduling

[0203] In some embodiments, the platforms, systems, media, and methods disclosed herein include intelligent scheduling and / or intelligent scheduling scoring, or use of the same. In further embodiments, an Intelligent Scheduling Score (ISS) is a tool to help prioritize and manage the bot army based on the importance and constraints of each bot. ISS scores are optionally determined on a per-bot (or per-cluster) basis. In some embodiments, the system will guide thescheduling and management of the hot army. As described herein, the ISS has at least the following exemplary uses:

[0204] Scheduling Bots: Bots with higher ISS will run more frequently or earlier than hots with lower scores. ISS-driven scheduling ensures that high-priority tasks run first.

[0205] Dynamic Scaling: If a hot with a high ISS is underperforming due to a lack of resources, the system will allocate more resources to that hot. Conversely, if a hot with a low ISS is overperforming or underutilized, the system will scale down its resources.

[0206] Handling API Rate Limits: If a hot with a high ISS is hitting its API rate limit, the team must negotiate a higher limit with the API provider. It is unlikely that automation is possible here.

[0207] Improving Caching Policies: If a bot with a high ISS frequently fetches data that the system can cache, the system will revise the bot' s caching policy or the system's cache management to increase the cache expiry time or the size of the cache.

[0208] An exemplary Intelligent Scheduling Score (ISS) formula is:Equation 6

[0209] In the example of Equation 6:

[0210] Bot Priority (BP) is the priority of the bot based on its functionality and user relevance.

[0211] API Rate Limit (ARE) is the maximum number of requests the bot can make to a specific API within a certain timeframe.

[0212] Bot Cycle Time (BCT) is the time the bot takes to complete one cycle of its operation.

[0213] Cache Expiry Time (CET) is the duration for which data fetched by the bot from the API is valid.

[0214] a, P, y, 5 are hyperparameters optionally control the relative importance of the factors. They are optionally adjusted based on domain knowledge and the system’s requirements.

[0215] Exemplary Equation 6 gives a higher ISS to bots with a higher RSS, a higher API rate limit, a shorter cycle time, or a longer cache expiry time while also allowing the relative importance of these factors to be adjusted using the hyperparameters. The subtraction of 5 * BCT aims to lower the ISS for bots with longer cycle times. In some embodiments, the system(or administrators of the system) optionally tune the hyperparameters based on the testing goals. For instance, if the API rate limit is a significant constraint, P and y should be higher. Likewise, if the bot’s priority (BP) is more important than other considerations, a should be higher.

[0216] Unified approach to resource utilization

[0217] In some embodiments, the platforms, systems, media, and methods disclosed herein compute a unified formula a Resource Utilization Numerator (RUN) score. In further embodiments, the RUN score takes into account the following:

[0218] Resource Segmentation Score RSS),'

[0219] Phased Deployment Score PDS ,' and

[0220] Intelligent Scheduling Score (ISS).

[0221] By way of example, a RUN score is optionally calculated as follows:Equation 7RUN = R(RSS x PDS x ISS)

[0222] In the example of Equation 7, the variable q is a normalization factor to scale the score to a suitable range (e.g., 0 to 1). By substituting the formulas for RSS, PDS, and ISS into the formula for TSES, we can create a single, unified formula that considers all the relevant factors and their relative importance. In some embodiments, the system will compute this formula to guide the deployment and management of the bot army. In such embodiments, a core intent behind the RUN is to streamline the operation of a vast bot army, potentially consisting of 100,000 bots or more. It recognizes that there is no need to deploy every bot simultaneously in a brute-force approach. Instead, the RUN provides a more elegant and efficient solution as a prioritization metric.

[0223] For instance, the RUN, in some cases, enables the identification of the most efficient bots within each cluster. In such cases, bots with the highest RUN are deployed first, which ensures prompt handling of the most valuable tasks. This approach optimizes resource utilization while maintaining a high standard of output quality.

[0224] Alternatively, in some cases, the RUN is used to select the most efficient bots across the entire bot army, irrespective of their respective clusters. In such cases, this allows for the deployment of the absolute top performers first, which can be particularly beneficial when tackling complex, large-scale tasks.Exemplary user interfaces and visualizations

[0225] In some embodiments, the platforms, systems, media, and methods disclosed herein include application interfaces, such as graphic user interfaces (GUIs), providing access to and / or implementing the features and functionality described herein, or use of the same.

[0226] Referring to FIG. 4A, in a particular embodiment, an exemplary GUI comprises tools for a user to define an audience to be simulated by a plurality of bots. In this embodiment, the tools enable the user to define an audience by one or more segments. Further, in this embodiment, each segment is optionally defined by a country or region, an age range, and a type, e.g., “Active Customer,” “Pro Subscription,” or “Lead Lost.”

[0227] Referring to FIG. 4B, in a particular embodiment, an exemplary GUI comprises tools for a user to select from among recommended templates of bot persona configurations and clusters and to customize templated configurations. In this embodiment, the recommended templates include a “Default Bundle,” an “Attack Bundle” (e.g., a Red Team Bundle), and a “Safety Bundle.”

[0228] Referring to FIG. 4C, in a particular embodiment, an exemplary GUI comprises tools for a user to customize a bot configuration including characteristics such as fine-tuning of humor, attitude, and even-handedness. In this embodiment, the characteristics are configured by adjusting slider GUI elements to change the characteristics on a scale of 0% to 100%. Further, in this embodiment, the GUI includes elements to activate the bot and save the settings. Continuing to refer to FIG. 4C, in the case depicted, the testing system is used to configure a bot that we name “Friendly Bot.” The humor setting is 90%, and other values are maintained at default. In this example, the humor in the configuration is primarily a factor of Al-based input-prompt generation. Some examples of this include:1. Explicitly state the desired humor level a. Include clear instructions about the humor level you want in the response, like “Respond with a 50% humor level regarding [topic].”2. Adjust the prompt’ s tone and phrasing a. Use a more or less playful tone in the prompt to influence the Al’s response. For example, ask questions lightheartedly for a higher humor level; for a lower one, use more severe or formal phrasing.3. Offer examples or analogiesa. Provide examples of the desired humor level, so the Al can better understand expectations. For instance, mention the styles of specific comedians or reference a humorous situation that aligns with the level of humor you want.4. Use rhetorical devices a. Obvious rhetorical devices like alliteration or antithesis may not fit the apparent mold for humor but are catchy enough to fall into the category.

[0229] By way of example, a prompt is crafted as follows: pre prompt = "Pretend you are a salesman." prompt = f'You want to sell me ' {topic} ' ." post 1 = sample ( ["Limit your response to two sentences.","Limit your response to three sentences.",] , 1) [0] post 2 = sample ( ["Feel free to use humor in your first sentence.","Feel free to use humor in your last sentence.","Use humor in both sentences.",] , 1) [0] return f"{pre prompt} {prompt} {post 1} {post 2}\n\n"

[0230] The input prompt is pseudo-dynamic within the Python code (from an actual use case example). There are static pre-prompts that will occur regardless of the probability distribution. For example, there is a 50 / 50 chance of a two or 3-sentence response and a 100% chance of humor. To fit this with the configuration section shown above, a probability function is optionally added as follows: prompt = f"{pre prompt} {prompt} {post 1}" if odds of ( 90 ) : prompt += f"{post 2}\n\n" prompt += f" {prompt } \n\n" return prompt

[0231] In this example, the system will dynamically change the input prompt as various sliders activate in the configuration phase.

[0232] Referring to FIG. 4D, in a particular embodiment, an exemplary GUI comprises information on the training and deployment of a plurality of hots configured using the tools described herein. In this embodiment, the information reflects a timeline of bot deployment including stages for “Calibration,” “Pre-production,” “Production,” and “On-going” activity. Further, in this embodiment, details are available for each stage completed or in progress, and for a stage in progress, an indication of percent completion is provided.

[0233] Referring to FIG. 5, in a particular embodiment, an exemplary GUI comprises tools for a user to assemble and deploy a plurality of bots using pre-determined persona configurations and cluster configurations optionally customized. In this embodiment, a user has assembled a “Skeptic Bot,” an “Attack Bundle” with a “Hostile Bot,” and a “Default Bundle” with a “Safety Bot” and a “Friendly Bot.”

[0234] Referring to FIG. 6, in a particular embodiment, an exemplary GUI comprises tools for a user to customize a testing approach by comparing bot persona and cluster configurations and viewing / editing specific exemplary questions. In this embodiment, a user has opted to compare a “Skeptic Bot” (with optional topic exploitation and a customizable cultural filter) with a “Tester Bot” (with optional penetration testing), wherein each is defined by humor, attitude, evenhandedness, and Socratic nature. Further, in this embodiment, the GUI provides the user with elements to optionally add questions and edit individual questions in the configuration.

[0235] Referring to FIG. 7, in a particular embodiment, an exemplary GUI comprises a dashboard for viewing and analyzing flagged breakdowns in the output of a Al system (model). In this embodiment, the GUI displays the number of bots running in the test and the flagged breakdowns are broken out by segment. Further, in this embodiment, an off-topic response is detected and labeled “critical” and a hallucination is detected and labeled “warning.” The results of testing, in this embodiment, are shown over time as the testing is continuous.

[0236] Referring to FIG. 8, in a particular embodiment, an exemplary GUI comprises tools for a user to analyze specific model output including responses to the different types of input generated by multiple bot personas. In this embodiment, the questions asked by a “Safety Bot,” “Skeptic Bot,” and a “Hostile Bot” are shown and the responses to each, provided by the model, are displayed. Further, in this embodiment, the responses are scored for topic, tone, and toxicity to develop an overall truest score.

[0237] Referring to FIGS. 9A-9C, in a particular embodiment, an exemplary GUI comprises tools for a user to customize a bot persona by fine-tuning specific output including blocking and customizing individual questions. In this embodiment, the GUI offers the option for a user to review individual questions that particular bots may and / or will ask a model. By way of example, as shown in FIG. 9A, a question is shown for a “Skeptic Bot” and in this example, the question is flagged for attention, which optionally includes blacklisting the question, customizing the question, and / or ignoring the question. By way of further example, as shown in FIG. 9B, a question is shown for a “Hostile Bot” and in this example, the question is flagged for attention, which optionally includes blacklisting the question, customizing the question, and / or ignoring the question. As shown in FIG. 9C, particular questions for a particular bot and / or bot persona are optionally tuned with regard to humor, attitude, even-handedness, and Socratic nature.

[0238] Referring to FIG. 10, in a particular embodiment, an exemplary GUI comprises tools for a user to review the results of dialog branch discovery for a model. In this embodiment, particular pathways have been discovered and mapped and the likelihood of individual branches being used are reported.

[0239] Referring to FIG. 11, in a particular embodiment, an exemplary GUI comprises tools for a user to monitor production readiness of a Al system (model) by monitoring a bot-based testing campaign over time and reviewing a trust score with regard to topic, tone, and toxicity. In this embodiment, the GUI includes indications of the status of the testing and how long the testing has been running continuously, including the number of bots deployed. Further, in this embodiment, the GUI includes results, over time (e.g., time series data), with regard to tone, topic, and risk, which are summarized in individual and aggregate trust scores, and also provides summaries of flags raised. Finally, this embodiment includes recommendations and features allowing a user to adjust leniency.

[0240] While preferred embodiments of the present subject matter have been shown and described herein, it will be obvious to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the present subject matter. It should be understood that various alternatives to the embodiments of the present subject matter described herein may be employed in practicing the present subject matter.

Claims

CLAIMSWHAT IS CLAIMED IS:

1. A computer-implemented method of testing a model comprising: a) identifying a plurality of Al-driven bots, wherein each hot comprises a persona; b) generating a bot priority score for each bot; c) determining a number of testing phases; d) computing the number of bots deployed in each testing phase; e) determining a resource scaling approach; f) generating an intelligent scheduling score for each bot; and g) deploying the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores.

2. The method of claim 1, wherein the model comprises a large natural language conversational model.

3. The method of claim 1, wherein the plurality of bots are identified from a library of preconfigured bots or bot personas.

4. The method of claim 1, further comprising organizing the plurality of Al-driven bots into one or more functionality clusters.

5. The method of claim 4, wherein the one or more functionality clusters comprise: a. one or more personality clusters, b. one or more red team clusters, c. one or more sentiment clusters, d. one or more domain knowledge clusters, e. one or more audience segment clusters, or any combination thereof.

6. The method of claim 4, further comprising generating a bot priority score for each cluster.

7. The method of claim 4, further comprising generating an intelligent scheduling score for each cluster.

8. The method of claim 1, wherein the testing is continuous for at least one hour, at least 1 day, at least 1 week, at least 1 month, or at least 1 year.

9. The method of claim 1, wherein the plurality of hots comprises at least 10, at least 100, at least 1000, at least 10,000, or at least 100,000 hots.

10. The method of claim 1, wherein at least a subset of the hots are configured to interact with the model.

11. The method of claim 10, wherein the plurality of hots interact with the model autonomously.

12. The method of claim 1, wherein the hot priority scores are determined, at least in part, by:wherein:FI is the functionality’s importance to the application (Likert scale of 1 - 7, with seven being most relevant and three being neutral);BR is the relevance a specific bot to the user base (Likert scale of 1 - 7, with seven being most relevant and three being neutral);IF is how frequently a test bot will interact with the specific functionality;El is the impact of an error or failure in a specific functionality (Likert scale of 1 - 7, with seven indicating a critical failure that would severely impact the user experience); andN is a normalization factor.

13. The method of claim 12, wherein the bot priority is determined, at least in part, by:wherein: a is a hyperfactor for FI, adjusted to weight FI according to perceived importance;P is a hyperfactor for BR, adjusted to weight BR according to perceived importance; y is a hyperfactor for IF, adjusted to weight IF according to perceived importance; and5 is a hyperfactor for El, adjusted to weight El according to perceived importance.

14. The method of claim 1, wherein the number of testing phases is determined, at least in part, by:wherein: ceil is the ceiling function;B is total number of hots;SC is the system capacity to handle simultaneous hot interactions;RF is a risk factor between 0 and 1 representing the risk tolerance; andTP is a tuning parameter optionally adjusted based on factors like test coverage, system stability, and testing objectives.

15. The method of claim 1, wherein the number of bots deployed in each testing phase is determined, at least in part, by:wherein:TDB is total deployed bots with a subscript denoting the phase number (K);BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

16. The method of claim 1, wherein the resource scaling approach is determined, at least in part, by:(BP ■ CL)RSK TCF wherein:RS subscripts denote the phase number (K) and resource type (T), respectively;BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

17. The method of claim 1, wherein the intelligent scheduling scores are determined, at least in part, by:wherein:BP is the bot priority based on functionality and user relevance;ARL (API Rate Limit) is the maximum number of requests the bot can make to a specific API within a certain timeframe;BCT (Bot Cycle Time) is the time the bot takes to complete one cycle of its operation; CET (Cache Expiry Time) is the duration for which data fetched by the bot from the API is valid; and a, P, y, 5 are hyperparameters adjusted to weight the factors according to relative importance.

18. The method of claim 1, further comprising calculating a resource utilization numerator (RUN) score, at least in part, by:RUN = T (RSS x PDS x IBS') wherein: q is a normalization factor to scale the score to a suitable range (e.g., 0 to 1);RSS is a resource segmentation score;PDS is a phased deployment score; and ISS is an intelligent scheduling score.

19. A computer-implemented system for testing a model comprising a computing device comprising at least one processor and instructions executable by the at least one processor to cause the at least one processor to perform operations comprising:a) identifying a plurality of Al-driven bots, wherein each hot comprises a persona; b) generating a bot priority score for each bot; c) determining a number of testing phases; d) computing the number of bots deployed in each testing phase; e) determining a resource scaling approach; f) generating an intelligent scheduling score for each bot; and g) deploying the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores.

20. The system of claim 19, wherein the model comprises a large natural language conversational model.

21. The system of claim 19, wherein the plurality of bots are identified from a library of preconfigured bots or bot personas.

22. The system of claim 19, wherein the operations further comprise organizing the plurality of Al-driven bots into one or more functionality clusters.

23. The system of claim 22, wherein the one or more functionality clusters comprise: a. one or more personality clusters, b. one or more red team clusters, c. one or more sentiment clusters, d. one or more domain knowledge clusters, e. one or more audience segment clusters, or any combination thereof.

24. The system of claim 22, wherein the operations further comprise generating a bot priority score for each cluster.

25. The system of claim 22, wherein the operations further comprise generating an intelligent scheduling score for each cluster.

26. The system of claim 19, wherein the testing is continuous for at least one hour, at least 1 day, at least 1 week, at least 1 month, or at least 1 year.

27. The system of claim 19, wherein the plurality of hots comprises at least 10, at least 100, at least 1000, at least 10,000, or at least 100,000 hots.

28. The system of claim 19, wherein at least a subset of the hots are configured to interact with the model.

29. The system of claim 28, wherein the plurality of hots interact with the model autonomously.

30. The system of claim 19, wherein the hot priority scores are determined, at least in part, by:wherein:FI is the functionality’s importance to the application (Likert scale of 1 - 7, with seven being most relevant and three being neutral);BR is the relevance a specific bot to the user base (Likert scale of 1 - 7, with seven being most relevant and three being neutral);IF is how frequently a test bot will interact with the specific functionality;El is the impact of an error or failure in a specific functionality (Likert scale of 1 - 7, with seven indicating a critical failure that would severely impact the user experience); andN is a normalization factor.

31. The system of claim 30, wherein the bot priority is determined, at least in part, by:wherein: a is a hyperfactor for FI, adjusted to weight FI according to perceived importance;P is a hyperfactor for BR, adjusted to weight BR according to perceived importance; y is a hyperfactor for IF, adjusted to weight IF according to perceived importance; and5 is a hyperfactor for El, adjusted to weight El according to perceived importance.

32. The system of claim 19, wherein the number of testing phases is determined, at least in part, by:wherein: ceil is the ceiling function;B is total number of hots;SC is the system capacity to handle simultaneous hot interactions;RF is a risk factor between 0 and 1 representing the risk tolerance; andTP is a tuning parameter optionally adjusted based on factors like test coverage, system stability, and testing objectives.

33. The system of claim 19, wherein the number of bots deployed in each testing phase is determined, at least in part, by:wherein:TDB is total deployed bots with a subscript denoting the phase number (K);BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

34. The system of claim 19, wherein the resource scaling approach is determined, at least in part, by:(BP ■ CL)RSK TCF wherein:RS subscripts denote the phase number (K) and resource type (T), respectively;BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

35. The system of claim 19, wherein the intelligent scheduling scores are determined, at least in part, by:wherein:BP is the bot priority based on functionality and user relevance;ARL (API Rate Limit) is the maximum number of requests the bot can make to a specific API within a certain timeframe;BCT (Bot Cycle Time) is the time the bot takes to complete one cycle of its operation;CET (Cache Expiry Time) is the duration for which data fetched by the bot from the API is valid; and a, P, y, 5 are hyperparameters adjusted to weight the factors according to relative importance.

36. The system of claim 19, wherein the operations further comprise calculating a resource utilization numerator (RUN) score, at least in part, by:RUN = T (RSS x PDS x IBS') wherein: q is a normalization factor to scale the score to a suitable range (e.g., 0 to 1);RSS is a resource segmentation score;PDS is a phased deployment score; andISS is an intelligent scheduling score.

37. One or more non-transitory computer-readable storage media encoded with instructions executable by one or more processors to provide a model testing application comprising: a) a module configured to identify a plurality of Al-driven bots, wherein each bot comprises a persona; b) a module configured to generate a bot priority score for each bot; c) a module configured to determine a number of testing phases;d) a module configured to compute the number of bots deployed in each testing phase; e) a module configured to determine a resource scaling approach; f) a module configured to generate an intelligent scheduling score for each bot; and g) a module configured to deploy the plurality of bots, over a network, to test the model according to the priority scores, the testing phases, the resource scaling approach, and the intelligent scheduling scores.

38. The non-transitory computer-readable storage media of claim 37, wherein the model comprises a large natural language conversational model.

39. The non-transitory computer-readable storage media of claim 37, wherein the plurality of bots are identified from a library of pre-configured bots or bot personas.

40. The non-transitory computer-readable storage media of claim 37, wherein the application further comprises a module configured to organize the plurality of Al-driven bots into one or more functionality clusters.

41. The non-transitory computer-readable storage media of claim 40, wherein the one or more functionality clusters comprise: a. one or more personality clusters, b. one or more red team clusters, c. one or more sentiment clusters, d. one or more domain knowledge clusters, e. one or more audience segment clusters, or any combination thereof.

42. The non-transitory computer-readable storage media of claim 40, wherein the application further comprises a module configured to generate a bot priority score for each cluster.

43. The non-transitory computer-readable storage media of claim 40, wherein the application further comprises a module configured to generate an intelligent scheduling score for each cluster.

44. The non-transitory computer-readable storage media of claim 37, wherein the testing is continuous for at least one hour, at least 1 day, at least 1 week, at least 1 month, or at least 1 year.

45. The non-transitory computer-readable storage media of claim 37, wherein the plurality of bots comprises at least 10, at least 100, at least 1000, at least 10,000, or at least 100,000 bots.

46. The non-transitory computer-readable storage media of claim 37, wherein at least a subset of the bots are configured to interact with the model.

47. The non-transitory computer-readable storage media of claim 46, wherein the plurality of bots interact with the model autonomously.

48. The non-transitory computer-readable storage media of claim 37, wherein the bot priority scores are determined, at least in part, by:wherein:FI is the functionality’s importance to the application (Likert scale of 1 - 7, with seven being most relevant and three being neutral);BR is the relevance a specific bot to the user base (Likert scale of 1 - 7, with seven being most relevant and three being neutral);IF is how frequently a test bot will interact with the specific functionality;El is the impact of an error or failure in a specific functionality (Likert scale of 1 - 7, with seven indicating a critical failure that would severely impact the user experience); andN is a normalization factor.

49. The non-transitory computer-readable storage media of claim 48, wherein the bot priority is determined, at least in part, by:wherein: a is a hyperfactor for FI, adjusted to weight FI according to perceived importance;P is a hyperfactor for BR, adjusted to weight BR according to perceived importance;y is a hyperfactor for IF, adjusted to weight IF according to perceived importance; and 5 is a hyperfactor for El, adjusted to weight El according to perceived importance.

50. The non-transitory computer-readable storage media of claim 37, wherein the number of testing phases is determined, at least in part, by:wherein: ceil is the ceiling function;B is total number of bots;SC is the system capacity to handle simultaneous bot interactions;RF is a risk factor between 0 and 1 representing the risk tolerance; andTP is a tuning parameter optionally adjusted based on factors like test coverage, system stability, and testing objectives.

51. The non-transitory computer-readable storage media of claim 37, wherein the number of bots deployed in each testing phase is determined, at least in part, by:wherein:TDB is total deployed bots with a subscript denoting the phase number (K);BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

52. The non-transitory computer-readable storage media of claim 37, wherein the resource scaling approach is determined, at least in part, by:(BP ■ CL)RSK TCFSC wherein:RS subscripts denote the phase number (K) and resource type (T), respectively;BP is the bot priority;SC is system’s current capacity, measured in terms of concurrent users and requests per second;CL is the current load on the system, measured in the same units as the system capacity; andCF is a cost factor considers the cost of the resources necessary to scale the system, based on current cloud service rates.

53. The non-transitory computer-readable storage media of claim 37, wherein the intelligent scheduling scores are determined, at least in part, by:wherein:BP is the bot priority based on functionality and user relevance;ARL (API Rate Limit) is the maximum number of requests the bot can make to a specific API within a certain timeframe;BCT (Bot Cycle Time) is the time the bot takes to complete one cycle of its operation;CET (Cache Expiry Time) is the duration for which data fetched by the bot from the API is valid; and a, P, y, 5 are hyperparameters adjusted to weight the factors according to relative importance.

54. The non-transitory computer-readable storage media of claim 37, wherein the application further comprises a module configured to calculate a resource utilization numerator (RUN) score, at least in part, by:RUN = T (RSS x PDS x IBS') wherein: q is a normalization factor to scale the score to a suitable range (e.g., 0 to 1);RSS is a resource segmentation score;PDS is a phased deployment score; andISS is an intelligent scheduling score.

Images